6.3-RELEASE-p3 recurring panics on multiple SM PDSMi+

[Royce Williams]

We have 10 SuperMicro PDSMi+ 5015M-MTs that are panic'ing every few
days.  This started shortly after upgrade from 6.2-RELEASE to
6.3-RELEASE with freebsd-update.

Other than switching to a debugging kernel, a little sysctl tuning,
and patching with freebsd-update, they are stock.  The debugging
kernel was built from source that is also being patched with
freebsd-update.

These systems are running postfix and Courier imapd for an ISP with a
userbase on the order of 10^4 users.  They use gmirror, but the
mailstore is over NFS.  That NFS server is under pretty high load.
All of the servers with this app and load pattern are panic'ing.

I have little experience with kernel debugging, but the box in
question is out of our farm and available for testing, and I am
motivated to cooperate. :-)

The full debugging kernel options I used are:

include SMP
options KDB
options KDB_TRACE
options DDB
options BREAK_TO_DEBUGGER
options WITNESS
options WITNESS_SKIPSPIN


db> trace
Tracing pid 71182 tid 100325 td 0xcc08b180
kdb_enter(c095f294) at kdb_enter+0x2b
panic(c09768ad,1000,14000000,c145bc88,1000,...) at panic+0x127
kmem_malloc(c14680c0,1000,102,eba6a8cc,c07e3fa5,...) at kmem_malloc+0x89
page_alloc(c1453780,1000,eba6a8bf,102,c06b8a84,...) at page_alloc+0x1a
slab_zalloc(c1453780,102,c14537e0,c1453780,c1460d5c,...) at
slab_zalloc+0xa1
uma_zone_slab(c1453780,2) at uma_zone_slab+0xf0
uma_zalloc_bucket(c1453780,2) at uma_zalloc_bucket+0x11c
uma_zalloc_arg(c1453780,0,2) at uma_zalloc_arg+0x24c
cache_enter(cd02c220,c9e62880,eba6a9fc) at cache_enter+0xa6
nfs_readdirplusrpc(cd02c220,eba6aa60,cc0ab880) at nfs_readdirplusrpc+0x6a6
nfs_doio(cd02c220,dce59668,cc0ab880,cc08b180,dce59668,...) at
nfs_doio+0x20f
nfs_bioread(cd02c220,eba6acb0,0,cc0ab880) at nfs_bioread+0xa64
nfs_readdir(eba6ac90) at nfs_readdir+0xe6
VOP_READDIR_APV(c09ebbc0,eba6ac90) at VOP_READDIR_APV+0x38
getdirentries(cc08b180,eba6ad04) at getdirentries+0x146
syscall(3b,3b,3b,9085f00,9085f00,...) at syscall+0x22f
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (196, FreeBSD ELF32, getdirentries), eip = 0xb825a79b, esp
= 0xbfbfa1fc, ebp = 0xbfbfa228 ---



Royce

-- 
Royce D. Williams                                   - http://royce.ws/
  I don't like that man. I must get to know him better. - A. Lincoln