kern/141104: 64 bit core freezes at high load

[Tikhon Roschupkin]

>Number:         141104
>Category:       kern
>Synopsis:       64 bit core freezes at high load
>Confidential:   no
>Severity:       critical
>Priority:       high
>Responsible:    freebsd-bugs
>State:          open
>Quarter:        
>Keywords:       
>Date-Required:
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Wed Dec 02 09:40:01 UTC 2009
>Closed-Date:
>Last-Modified:
>Originator:     Tikhon Roschupkin
>Release:        FreeBSD 8.0-RELEASE #3 amd64
>Organization:
Hosting Center
>Environment:
FreeBSD fe90.hc.ru 8.0-RELEASE FreeBSD 8.0-RELEASE #3: Mon Nov 30 16:27:53 MSK 2009     iefremov at fe90.hc.ru:/www/obj/usr/src/sys/HC8-1  amd64
>Description:
Server for mass hosting with 1000 sites on it freezes every 8 hours at the average. No kritical error occurs, no messages. If the server remains unloaded, it's works fine. We've excluded all possible problems with hardware as the problem persists on different servers from different vendors. We've examined FreeBSD 6.5 (32 bit), FreeBSD 7.2 (64 bit) and FreeBSD 8.0 (64 bit) and only 64-bit versions are believed to make the server freezed.

Hosting Center is the second largest hosting provider in Russia. We run over a hundred servers for mass hosting under FreeBSD and until now there was no other problem like this one. We would be very grateful for any help from FreeBSD developers team. 

Additional details as follows :

Core options :

options         QUOTA

options         IPFIREWALL
options         IPFIREWALL_VERBOSE
options         IPFIREWALL_VERBOSE_LIMIT=100
options         IPFIREWALL_FORWARD
options         IPFIREWALL_DEFAULT_TO_ACCEPT


options         IPSTEALTH
device          pf
device          pflog
device          pfsync
#options         ALTQ

options         DEVICE_POLLING
options         HZ=1000

options         GEOM_JOURNAL
options         FULL_PREEMPTION
options         IPI_PREEMPTION


Hardware :

hostb0 at pci0:0:0:0:      class=0x060000 card=0xa48015d9 chip=0x65c08086 rev=0x90 hdr=0x00
pcib1 at pci0:0:2:0:       class=0x060400 card=0x00000000 chip=0x65f78086 rev=0x90 hdr=0x01
pcib2 at pci0:0:4:0:       class=0x060400 card=0x00000000 chip=0x65f88086 rev=0x90 hdr=0x01
pcib3 at pci0:0:6:0:       class=0x060400 card=0x00000000 chip=0x65f98086 rev=0x90 hdr=0x01
hostb1 at pci0:0:16:0:     class=0x060000 card=0xa48015d9 chip=0x65f08086 rev=0x90 hdr=0x00
hostb2 at pci0:0:16:1:     class=0x060000 card=0xa48015d9 chip=0x65f08086 rev=0x90 hdr=0x00
hostb3 at pci0:0:16:2:     class=0x060000 card=0xa48015d9 chip=0x65f08086 rev=0x90 hdr=0x00
hostb4 at pci0:0:17:0:     class=0x060000 card=0xa48015d9 chip=0x65f18086 rev=0x90 hdr=0x00
hostb5 at pci0:0:19:0:     class=0x060000 card=0xa48015d9 chip=0x65f38086 rev=0x90 hdr=0x00
hostb6 at pci0:0:21:0:     class=0x060000 card=0xa48015d9 chip=0x65f58086 rev=0x90 hdr=0x00
hostb7 at pci0:0:22:0:     class=0x060000 card=0xa48015d9 chip=0x65f68086 rev=0x90 hdr=0x00
pcib4 at pci0:0:28:0:      class=0x060400 card=0xa48015d9 chip=0x29408086 rev=0x02 hdr=0x01
pcib5 at pci0:0:28:4:      class=0x060400 card=0xa48015d9 chip=0x29488086 rev=0x02 hdr=0x01
pcib6 at pci0:0:28:5:      class=0x060400 card=0xa48015d9 chip=0x294a8086 rev=0x02 hdr=0x01
pcib7 at pci0:0:30:0:      class=0x060401 card=0xa48015d9 chip=0x244e8086 rev=0x92 hdr=0x01
isab0 at pci0:0:31:0:      class=0x060100 card=0xa48015d9 chip=0x29168086 rev=0x02 hdr=0x00
none0 at pci0:0:31:3:      class=0x0c0500 card=0xa48015d9 chip=0x29308086 rev=0x02 hdr=0x00
mfi0 at pci0:2:0:0:        class=0x010400 card=0x10121000 chip=0x00601000 rev=0x04 hdr=0x00
^^^ Controller
em0 at pci0:5:0:0: class=0x020000 card=0x108c15d9 chip=0x108c8086 rev=0x03 hdr=0x00
em1 at pci0:6:0:0: class=0x020000 card=0x109a15d9 chip=0x109a8086 rev=0x00 hdr=0x00
atapci0 at pci0:7:0:0:     class=0x010185 card=0xa48015d9 chip=0x82131283 rev=0x00 hdr=0x00
vgapci0 at pci0:7:1:0:     class=0x030000 card=0xa48015d9 chip=0x002118ca rev=0x00 hdr=0x00

Firewall : pf


>How-To-Repeat:

>Fix:
If pf is completelly turned off, everything is OK.

>Release-Note:
>Audit-Trail:
>Unformatted: