[nullfs] [panic] null with unref'ed lowervp

Mon Nov 23 02:05:08 UTC 2009

Hello,
my server recently had the kernel panic "null with unref'ed lowervp" in
null_subr.c:null_checkvp().
I am not sure whether I should open a PR for it.

The system is a SMP machine with SCSI RAID (asr), it runs several jails
and uses multiple null mounts between partitions, because there is not
enough disk space (~90% usage).

uname -a:
FreeBSD trinity.asta.uni-potsdam.de 7.2-RELEASE
FreeBSD 7.2-RELEASE #3: Tue May 12 18:53:06 CEST 2009
root at trinity.asta.uni-potsdam.de:/usr/obj/usr/src/sys/ASTA  i386

Kernel is compiled with:
options INVARIANT_SUPPORT
options INVARIANTS
options DIAGNOSTIC

Among the few occurances I was not able to observe a pattern or
reproduce the error (two crashes happened at 8am which is a cronjob
time, but not one with particulary high load).
I am also going to add new disks any time now in order to reduce the
number of null mounts, so I do not expect to see this error again.

I append three gdb backtraces, in case anyone finds them useful.

-- 
Martin Schütte
-------------- next part --------------
[root at trinity] /usr/obj/usr/src/sys/ASTA# cat /archiv/crash/info.2 && kgdb kernel.debug /archiv/crash/vmcore.2
Dump header from device /dev/da0s1b
  Architecture: i386
  Architecture Version: 2
  Dump Length: 354189312B (337 MB)
  Blocksize: 512
  Dumptime: Fri Oct 16 08:00:05 2009
  Hostname: trinity.asta.uni-potsdam.de
  Magic: FreeBSD Kernel Dump
  Version String: FreeBSD 7.2-RELEASE #3: Tue May 12 18:53:06 CEST 2009
    root at trinity.asta.uni-potsdam.de:/usr/obj/usr/src/sys/ASTA
  Panic String: null with unref'ed lowervp
  Dump Parity: 3870003723
  Bounds: 2
  Dump Status: good
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "i386-marcel-freebsd"...

Unread portion of the kernel message buffer:
vp = 0xcabd0000, unref'ed lowervp
 deadc0de deadc0de deadc0de c70fb2a0 deadc0de deadc0de deadc0de c70fb2a0
panic: null with unref'ed lowervp
cpuid = 2
Uptime: 31d4h28m45s
Physical memory: 3447 MB
Dumping 337 MB: 322 306 290 274 258 242 226 210 194 178 162 146 130 114 98 82 66 50 34 18 2

Reading symbols from /boot/kernel/accf_http.ko...Reading symbols from /boot/kernel/accf_http.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/accf_http.ko
Reading symbols from /boot/kernel/acpi.ko...Reading symbols from /boot/kernel/acpi.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/acpi.ko
Reading symbols from /boot/kernel/nullfs.ko...Reading symbols from /boot/kernel/nullfs.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/nullfs.ko
#0  doadump () at pcpu.h:196
196             __asm __volatile("movl %%fs:0,%0" : "=r" (td));
(kgdb) bt
#0  doadump () at pcpu.h:196
#1  0xc057ba0c in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:418
#2  0xc057bcac in panic (fmt=Variable "fmt" is not available.
) at /usr/src/sys/kern/kern_shutdown.c:574
#3  0xc70f86f3 in null_checkvp (vp=Variable "vp" is not available.
) at /usr/src/sys/modules/nullfs/../../fs/nullfs/null_subr.c:337
#4  0xc70f9557 in null_lock (ap=0xeb12c994) at /usr/src/sys/modules/nullfs/../../fs/nullfs/null_vnops.c:531
#5  0xc07d0a35 in VOP_LOCK1_APV (vop=0xc70fb4c0, a=0xeb12c994) at vnode_if.c:1618
#6  0xc0605bde in _vn_lock (vp=0xcabd0000, flags=8194, td=0xc6f52d20, file=0xc080b893 "/usr/src/sys/kern/vfs_subr.c", line=2159)
    at vnode_if.h:851
#7  0xc05fa0ee in vrele (vp=0xcabd0000) at /usr/src/sys/kern/vfs_subr.c:2159
#8  0xc05f05fe in namei (ndp=0xeb12cb7c) at /usr/src/sys/kern/vfs_lookup.c:202
#9  0xc0605572 in vn_open_cred (ndp=0xeb12cb7c, flagp=0xeb12cc78, cmode=0, cred=0xc717f600, fp=0xc70744c0)
    at /usr/src/sys/kern/vfs_vnops.c:188
#10 0xc06057f3 in vn_open (ndp=0xeb12cb7c, flagp=0xeb12cc78, cmode=0, fp=0xc70744c0) at /usr/src/sys/kern/vfs_vnops.c:94
#11 0xc06046b3 in kern_open (td=0xc6f52d20, path=0xbfbfa750 <Address 0xbfbfa750 out of bounds>, pathseg=UIO_USERSPACE, flags=3,
    mode=0) at /usr/src/sys/kern/vfs_syscalls.c:1042
#12 0xc0604ba0 in open (td=0xc6f52d20, uap=0xeb12ccfc) at /usr/src/sys/kern/vfs_syscalls.c:1009
#13 0xc07ba703 in syscall (frame=0xeb12cd38) at /usr/src/sys/i386/i386/trap.c:1090
#14 0xc079fd80 in Xint0x80_syscall () at /usr/src/sys/i386/i386/exception.s:255
#15 0x00000033 in ?? ()
Previous frame inner to this frame (corrupt stack?)
-------------- next part --------------
= info =
Dump header from device /dev/da0s1b
  Architecture: i386
  Architecture Version: 2
  Dump Length: 362143744B (345 MB)
  Blocksize: 512
  Dumptime: Sun Oct 25 08:00:16 2009
  Hostname: trinity.asta.uni-potsdam.de
  Magic: FreeBSD Kernel Dump
  Version String: FreeBSD 7.2-RELEASE #3: Tue May 12 18:53:06 CEST 2009
    root at trinity.asta.uni-potsdam.de:/usr/obj/usr/src/sys/ASTA
  Panic String: null with unref'ed lowervp
  Dump Parity: 2201427979
  Bounds: 3
  Dump Status: good

= kgdb =
Unread portion of the kernel message buffer:
vp = 0xc7f17000, unref'ed lowervp
 deadc0de deadc0de deadc0de c70e62a0 1fffff 0 0 0
panic: null with unref'ed lowervp
cpuid = 1
Uptime: 9d0h58m13s
Physical memory: 3447 MB
Dumping 345 MB: 330 314 298 282 266 250 234 218 202 186 170 154 138 122 106 90 74 58 42 26 10

(kgdb) bt
#0  doadump () at pcpu.h:196
#1  0xc057ba0c in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:418
#2  0xc057bcac in panic (fmt=Variable "fmt" is not available.
) at /usr/src/sys/kern/kern_shutdown.c:574
#3  0xc70e36f3 in null_checkvp (vp=Variable "vp" is not available.
) at /usr/src/sys/modules/nullfs/../../fs/nullfs/null_subr.c:337
#4  0xc70e4557 in null_lock (ap=0xebc0fb20) at /usr/src/sys/modules/nullfs/../../fs/nullfs/null_vnops.c:531
#5  0xc07d0a35 in VOP_LOCK1_APV (vop=0xc70e64c0, a=0xebc0fb20) at vnode_if.c:1618
#6  0xc0605bde in _vn_lock (vp=0xc7f17000, flags=8194, td=0xca5ffaf0, file=0xc080b893 "/usr/src/sys/kern/vfs_subr.c", line=2159)
    at vnode_if.h:851
#7  0xc05fa0ee in vrele (vp=0xc7f17000) at /usr/src/sys/kern/vfs_subr.c:2159
#8  0xc06004ca in kern_rename (td=0xca5ffaf0, from=0x287019b0 <Address 0x287019b0 out of bounds>,
    to=0x28701a20 <Address 0x28701a20 out of bounds>, pathseg=UIO_USERSPACE) at /usr/src/sys/kern/vfs_syscalls.c:3428
#9  0xc0600589 in rename (td=0xca5ffaf0, uap=0xebc0fcfc) at /usr/src/sys/kern/vfs_syscalls.c:3319
#10 0xc07ba703 in syscall (frame=0xebc0fd38) at /usr/src/sys/i386/i386/trap.c:1090
#11 0xc079fd80 in Xint0x80_syscall () at /usr/src/sys/i386/i386/exception.s:255
#12 0x00000033 in ?? ()
Previous frame inner to this frame (corrupt stack?)
-------------- next part --------------
= info =
Dump header from device /dev/da0s1b
  Architecture: i386
  Architecture Version: 2
  Dump Length: 365514752B (348 MB)
  Blocksize: 512
  Dumptime: Tue Nov 17 22:48:12 2009
  Hostname: trinity.asta.uni-potsdam.de
  Magic: FreeBSD Kernel Dump
  Version String: FreeBSD 7.2-RELEASE #3: Tue May 12 18:53:06 CEST 2009
    root at trinity.asta.uni-potsdam.de:/usr/obj/usr/src/sys/ASTA
  Panic String: null with unref'ed lowervp
  Dump Parity: 527080458
  Bounds: 4
  Dump Status: good

= kgdb =
Unread portion of the kernel message buffer:
vp = 0xc8e60228, unref'ed lowervp
 deadc0de deadc0de deadc0de c713b2a0 deadc0de deadc0de deadc0de c084e1a0
panic: null with unref'ed lowervp
cpuid = 0
Uptime: 23d14h45m58s
Physical memory: 3447 MB
Dumping 348 MB: 333 317 301 285 269 253 237 221 205 189 173 157 141 125 109 93 77 61 45 29 13

(kgdb) bt
#0  doadump () at pcpu.h:196
#1  0xc057ba0c in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:418
#2  0xc057bcac in panic (fmt=Variable "fmt" is not available.
) at /usr/src/sys/kern/kern_shutdown.c:574
#3  0xc71386f3 in null_checkvp (vp=Variable "vp" is not available.
) at /usr/src/sys/modules/nullfs/../../fs/nullfs/null_subr.c:337
#4  0xc7139557 in null_lock (ap=0xeb5af994) at /usr/src/sys/modules/nullfs/../../fs/nullfs/null_vnops.c:531
#5  0xc07d0a35 in VOP_LOCK1_APV (vop=0xc713b4c0, a=0xeb5af994) at vnode_if.c:1618
#6  0xc0605bde in _vn_lock (vp=0xc8e60228, flags=8194, td=0xc8e87460, file=0xc080b893 "/usr/src/sys/kern/vfs_subr.c", line=2159)
    at vnode_if.h:851
#7  0xc05fa0ee in vrele (vp=0xc8e60228) at /usr/src/sys/kern/vfs_subr.c:2159
#8  0xc05f05fe in namei (ndp=0xeb5afb7c) at /usr/src/sys/kern/vfs_lookup.c:202
#9  0xc0605572 in vn_open_cred (ndp=0xeb5afb7c, flagp=0xeb5afc78, cmode=0, cred=0xc76c2100, fp=0xcd28a6d4)
    at /usr/src/sys/kern/vfs_vnops.c:188
#10 0xc06057f3 in vn_open (ndp=0xeb5afb7c, flagp=0xeb5afc78, cmode=0, fp=0xcd28a6d4) at /usr/src/sys/kern/vfs_vnops.c:94
#11 0xc06046b3 in kern_open (td=0xc8e87460, path=0xbfbfa8f0 <Address 0xbfbfa8f0 out of bounds>, pathseg=UIO_USERSPACE, flags=3,
    mode=0) at /usr/src/sys/kern/vfs_syscalls.c:1042
#12 0xc0604ba0 in open (td=0xc8e87460, uap=0xeb5afcfc) at /usr/src/sys/kern/vfs_syscalls.c:1009
#13 0xc07ba703 in syscall (frame=0xeb5afd38) at /usr/src/sys/i386/i386/trap.c:1090
#14 0xc079fd80 in Xint0x80_syscall () at /usr/src/sys/i386/i386/exception.s:255
#15 0x00000033 in ?? ()
Previous frame inner to this frame (corrupt stack?)