ZFS-NFS kernel panic under load

Wed Aug 6 15:29:01 UTC 2008

Hello,

Please forgive me, I didn't really see this discussed in the archives but 
I am wondering if anyone has seen this issue.  I can replicate this issue 
under FreeBSD amd64 7.0-RELEASE and the latest -STABLE (RELENG_7).  I do 
not replicate any problems running 9 instances of postmark on the machine 
directly, so the issue appears to be isolated with NFS.

There are backtraces and more information in ticket kern/124280

I am experiencing random kernel panics while running postmark benchmark 
from 9 NFS clients (clients on RedHat) to a 3TB ZFS filesystem exported 
with NFS.  The panics happen as soon as 5 mins from starting the benchmark 
or may take hours before it panics and reboots.  It doesn't correspond 
to a time a cron job is going on.  I am using the following settings in 
postmark:

set number 20000
set transactions 10000000
set subdirectories 1000
set size 10000 15000
set report verbose
set location /var/mail/store1/X  (where X is a number 1-9 so each is 
operating in its own tree)

The problem happens if I run 1 postmark on 9 NFS clients at the same time 
(each client is its own server) or if I run 9 postmarks on one NFS client.

commands used to create filesystem:
zpool create tank mirror da0 da12 mirror da1 da13 mirror da2 da14 mirror 
da3 da15\
  mirror da4 da16 mirror da5 da17 mirror da6 da18 mirror da7 da19 mirror 
da8 da20 \
  mirror da9 da21 mirror da10 da22 spare da11 da23
zfs set atime=off tank
zfs create tank/mail
zfs set mountpoint=/var/mail tank/mail
zfs set sharenfs="-maproot=root  -network 192.168.2.0 -mask 255.255.255.0" 
tank/mail

I am using a 3ware 9690 SAS controller.  I have 2 IBM EXP3000 enclosures, 
each drive is shown as single disk by the controller.

this is my loader.conf:
vm.kmem_size_max="1073741824"
vm.kmem_size="1073741824"
kern.maxvnodes="800000"
vfs.zfs.prefetch_disable="1"
vfs.zfs.cache_flush_disable="1"

(I should note that kern.maxnodes in loader.conf does not appear to do 
anything, after boot, it is shown to be at 100000 with sysctl.  It does 
change to 800000 if I manually set it with sysctl.  However it appears my 
vnode usage sits at around 25-26K and is near that within 5s of the panic.

The server has 16GB of RAM, and 2 quad core XEON processors.

This server is only a NFS fileserver.  The only non-default daemon running 
is sshd.  It is running the GENERIC kernel, right now, unmodified.

I am using two NICs.  NFS is exported only on the secondary NIC.  Each NIC 
is in it's own subnet.

nothing in /var/log/messages near time of panic except:
Aug  6 08:45:30 store1 savecore: reboot after panic: page fault
Aug  6 08:45:30 store1 savecore: writing core to vmcore.2

I can provide cores if needed.

Thank you for your time!

Weldon

kgdb with backtrace:

store1# kgdb kernel.debug /var/crash/vmcore.2
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you 
are
welcome to change it and/or distribute copies of it under certain 
conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for 
details.
This GDB was configured as "amd64-marcel-freebsd"...

Unread portion of the kernel message buffer:

Fatal trap 12: page fault while in kernel mode
cpuid = 5; apic id = 05
fault virtual address   = 0xdc
fault code              = supervisor read data, page not present
instruction pointer     = 0x8:0xffffffff8063b3d8
stack pointer           = 0x10:0xffffffffdfbc5720
frame pointer           = 0x10:0xffffff00543ed000
code segment            = base 0x0, limit 0xfffff, type 0x1b
                         = DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags        = interrupt enabled, resume, IOPL = 0
current process         = 839 (nfsd)
trap number             = 12
panic: page fault
cpuid = 5
Uptime: 18m53s
Physical memory: 16366 MB
Dumping 1991 MB: 1976 1960 1944 1928 1912 1896 1880 1864 1848 1832 1816 
1800 1784 1768 1752 1736 1720 1704 1688 1672 1656 1640 1624 1608 1592 1576 
1560 1544 1528 1512 1496 1480 1464 1448 1432 1416 1400 1384 1368 1352 1336 
1320 1304 1288 1272 1256 1240 1224 1208 1192 1176 1160 1144 1128 1112 1096 
1080 1064 1048 1032 1016 1000 984 968 952 936 920 904 888 872 856 840 824 
808 792 776 760 744 728 712 696 680 664 648 632 616 600 584 568 552 536 
520 504 488 472 456 440 424 408 392 376 360 344 328 312 296 280 264 248 
232 216 200 184 168 152 136 120 104 88 72 56 40 24 8

Reading symbols from /boot/kernel/zfs.ko...Reading symbols from 
/boot/kernel/zfs.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/zfs.ko
#0  doadump () at pcpu.h:194
194             __asm __volatile("movq %%gs:0,%0" : "=r" (td));
(kgdb) backtrace
#0  doadump () at pcpu.h:194
#1  0x0000000000000004 in ?? ()
#2  0xffffffff804a7049 in boot (howto=260) at 
/usr/src/sys/kern/kern_shutdown.c:418
#3  0xffffffff804a744d in panic (fmt=0x104 <Address 0x104 out of bounds>) 
at /usr/src/sys/kern/kern_shutdown.c:572
#4  0xffffffff807780e4 in trap_fatal (frame=0xffffff000bce26c0, 
eva=18446742974395967712)
     at /usr/src/sys/amd64/amd64/trap.c:724
#5  0xffffffff807784b5 in trap_pfault (frame=0xffffffffdfbc5670, 
usermode=0) at /usr/src/sys/amd64/amd64/trap.c:641
#6  0xffffffff80778de8 in trap (frame=0xffffffffdfbc5670) at 
/usr/src/sys/amd64/amd64/trap.c:410
#7  0xffffffff8075e7ce in calltrap () at 
/usr/src/sys/amd64/amd64/exception.S:169
#8  0xffffffff8063b3d8 in nfsrv_access (vp=0xffffff00207d7dc8, flags=128, 
cred=0xffffff00403d4800, rdonly=0,
     td=0xffffff000bce26c0, override=0) at 
/usr/src/sys/nfsserver/nfs_serv.c:4284
#9  0xffffffff8063c4f1 in nfsrv3_access (nfsd=0xffffff00543ed000, 
slp=0xffffff0006396d00, td=0xffffff000bce26c0,
     mrq=0xffffffffdfbc5af0) at /usr/src/sys/nfsserver/nfs_serv.c:234
#10 0xffffffff8064cd1d in nfssvc (td=Variable "td" is not available.
) at /usr/src/sys/nfsserver/nfs_syscalls.c:456
#11 0xffffffff80778737 in syscall (frame=0xffffffffdfbc5c70) at 
/usr/src/sys/amd64/amd64/trap.c:852
#12 0xffffffff8075e9db in Xfast_syscall () at 
/usr/src/sys/amd64/amd64/exception.S:290
#13 0x0000000800687acc in ?? ()
Previous frame inner to this frame (corrupt stack?)