panic during nfs operations in 4.8S on Dell 2650

[Mark Powell]

Hi,
  We've recently got a couple of Dell Poweredge 2650's with 2x2.8GHz
Xeons, 4GB RAM, PERC 3/Di (aac) RAID controller. They are mounting a 700GB
fs over NFS from a NetAPP. They are connected to a Cisco 3550-12T gigabit
over copper switch. I tried them first on the intel em cards and they
panicked and also the internal bge adapters with the same result.
  Thought everything was fine until I was rsyncing the POP3 mail stores
from the old machines onto these. Rsync runs for about an hour or so and
get's large. In the 300M-600M region the system will always panic. This
happens on both systems, so doesn't seem a hardware fault.

# gdb -k kernel.debug.3 vmcore.3
GNU gdb 4.18 (FreeBSD)
Copyright 1998 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you
are
welcome to change it and/or distribute copies of it under certain
conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for
details.
This GDB was configured as "i386-unknown-freebsd"...Deprecated bfd_read
called at
/usr/src/gnu/usr.bin/binutils/gdb/../../../../contrib/gdb/gdb/dbxread.c
line 2627 in elfstab_build_psymtabs
Deprecated bfd_read called at
/usr/src/gnu/usr.bin/binutils/gdb/../../../../contrib/gdb/gdb/dbxread.c
line 933 in fill_symbuf

SMP 2 cpus
IdlePTD at phsyical address 0x004ad000
initial pcb at physical address 0x003f8d00
panicstr: page fault
panic messages:
---
Fatal trap 12: page fault while in kernel mode
mp_lock = 01001186; cpuid = 1; lapic.id = 02000000
fault virtual address   = 0x0
fault code              = supervisor write, page not present
instruction pointer     = 0x8:0xc0312ea3
stack pointer           = 0x10:0xf946cc60
frame pointer           = 0x10:0xf946cc94
code segment            = base 0x0, limit 0xfffff, type 0x1b
                        = DPL 0, pres 1, def32 1, gran 1
processor eflags        = interrupt enabled, resume, IOPL = 0
current process         = 330 (rsync)
interrupt mask          = bio  <- SMP: XXX
trap number             = 12
panic: page fault
mp_lock = 01001186; cpuid = 1; lapic.id = 02000000
boot() called on cpu#1

syncing disks... 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2
giving up on 2 buffers
Uptime: 1h35m9s

...

(kgdb) where
#0
dumpsys ()
at
../../kern/kern_shutdown.c:487
#1  0xc01c45e7 in boot (howto=256) at ../../kern/kern_shutdown.c:316
#2  0xc01c4a59 in panic (fmt=0xc03781f9 "%s") at
../../kern/kern_shutdown.c:595
#3  0xc031458d in trap_fatal (frame=0xf946cc20, eva=0) at
../../i386/i386/trap.c:974
#4  0xc03141f9 in trap_pfault (frame=0xf946cc20, usermode=0, eva=0) at
../../i386/i386/trap.c:867
#5  0xc0313d53 in trap (frame={tf_fs = -869007336, tf_es = -1069547504,
tf_ds = 16, tf_edi = 0,
      tf_esi = -8400896, tf_ebp = -112800620, tf_isp = -112800692, tf_ebx
= 0,
      tf_edx = -1745141825, tf_ecx = 42, tf_eax = 0, tf_trapno = 12,
tf_err = 2,
      tf_eip = -1070518621, tf_cs = 8, tf_eflags = 66050, tf_esp =
-113966560,
      tf_ss = -1071700836}) at ../../i386/i386/trap.c:466
#6  0xc0312ea3 in generic_bzero ()
#7  0xc0240d90 in nfs_nget (mntp=0xcc621400, fhp=0xc62fe84c, fhsize=32,
npp=0xf946cd34)
    at ../../nfs/nfs_node.c:143
#8  0xc026716f in nfs_lookup (ap=0xf946ce00) at ../../nfs/nfs_vnops.c:959
#9  0xc01f15e5 in lookup (ndp=0xf946ce7c) at vnode_if.h:52
#10 0xc01f10e0 in namei (ndp=0xf946ce7c) at ../../kern/vfs_lookup.c:153
#11 0xc01f7441 in lstat (p=0xf9350220, uap=0xf946cf80) at
../../kern/vfs_syscalls.c:1824
#12 0xc03148c9 in syscall2 (frame={tf_fs = -112852945, tf_es =
-1070464977, tf_ds = -869334993,
      tf_edi = -1077951584, tf_esi = -1077953648, tf_ebp = -1077953744,
tf_isp = -112799788,
      tf_ebx = -1077952624, tf_edx = 2, tf_ecx = -49, tf_eax = 190,
tf_trapno = 7, tf_err = 2,
      tf_eip = 1745642828, tf_cs = 31, tf_eflags = 659, tf_esp =
-1077953772, tf_ss = 47})
    at ../../i386/i386/trap.c:1175
#13 0xc03015fb in Xint0x80_syscall ()
cannot read proc at 0

Cheers.

-- 
Mark Powell - UNIX System Administrator - The University of Salford
Information Services Division, Clifford Whitworth Building,
Salford University, Manchester, M5 4WT, UK.
Tel: +44 161 295 5936  Fax: +44 161 295 5888  www.pgp.com for PGP key