i386/59945: nullfs bug: reboot after panic: null_checkvp

Admin root at foo.felvi.hu
Thu Dec 4 03:50:23 PST 2003 

>Number:         59945
>Category:       i386
>Synopsis:       nullfs bug: reboot after panic: null_checkvp
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    freebsd-i386
>State:          open
>Quarter:        
>Keywords:       
>Date-Required:
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Thu Dec 04 03:50:16 PST 2003
>Closed-Date:
>Last-Modified:
>Originator:     Admin
>Release:        FreeBSD 4.9-RELEASE i386
>Organization:
EDUCATIO
>Environment:
System: FreeBSD foo.felvi.hu 4.9-RELEASE FreeBSD 4.9-RELEASE #2: Fri Nov 14 10:52:35 CET 2003 root at foo.felvi.hu:/usr/obj/usr/src/sys/KAAMOS i386

Dmesg:
Copyright (c) 1992-2003 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
        The Regents of the University of California. All rights reserved.
FreeBSD 4.9-RELEASE #2: Fri Nov 14 10:52:35 CET 2003
    root at foo.felvi.hu:/usr/obj/usr/src/sys/KAAMOS
Timecounter "i8254"  frequency 1193182 Hz
CPU: Intel(R) Xeon(TM) CPU 2.80GHz (2793.90-MHz 686-class CPU)
  Origin = "GenuineIntel"  Id = 0xf27  Stepping = 7
  Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
  Hyperthreading: 2 logical CPUs
real memory  = 2147397632 (2097068K bytes)
avail memory = 2086674432 (2037768K bytes)
APIC_IO: MP table broken: 8259->APIC entry missing!
Programming 16 pins in IOAPIC #0
IOAPIC #0 intpin 2 -> irq 0
Programming 16 pins in IOAPIC #1
Programming 16 pins in IOAPIC #2
FreeBSD/SMP: Multiprocessor motherboard: 4 CPUs
 cpu0 (BSP): apic id:  0, version: 0x00050014, at 0xfee00000
 cpu1 (AP):  apic id:  1, version: 0x00050014, at 0xfee00000
 cpu2 (AP):  apic id:  6, version: 0x00050014, at 0xfee00000
 cpu3 (AP):  apic id:  7, version: 0x00050014, at 0xfee00000
 io0 (APIC): apic id: 14, version: 0x000f0011, at 0xfec00000
 io1 (APIC): apic id: 13, version: 0x000f0011, at 0xfec01000
 io2 (APIC): apic id: 12, version: 0x000f0011, at 0xfec02000
Preloaded elf kernel "kernel" at 0xc04a5000.
Warning: Pentium 4 CPU: PSE disabled
Pentium Pro MTRR support enabled
md0: Malloc disk
npx0: <math processor> on motherboard
npx0: INT 16 interface
pcib0: <Host to PCI bridge> on motherboard
IOAPIC #1 intpin 10 -> irq 2
pci0: <PCI bus> on pcib0
pci0: <ATI Mach64-GR graphics accelerator> at 6.0 irq 2
atapci0: <ServerWorks CSB5 ATA100 controller> port 0x700-0x70f,0x374-0x377,0x170
-0x177,0x3f4-0x3f7,0x1f0-0x1f7 at device 15.1 on pci0
ata0: at 0x1f0 irq 14 on atapci0
ata1: at 0x170 irq 15 on atapci0
pci0: <OHCI USB controller> at 15.2 irq 3
isab0: <PCI to ISA bridge (vendor=1166 device=0225)> at device 15.3 on pci0
isa0: <ISA bus> on isab0
pcib1: <Host to PCI bridge> on motherboard
pci1: <PCI bus> on pcib1
pcib2: <Host to PCI bridge> on motherboard
IOAPIC #1 intpin 4 -> irq 4
pci2: <PCI bus> on pcib2
em0: <Intel(R) PRO/1000 Network Connection, Version - 1.7.16> mem 0xfbfd0000-0xf
bfdffff,0xfbfe0000-0xfbffffff irq 4 at device 3.0 on pci2
em0:  Speed:1000 Mbps  Duplex:Full
pcib3: <Host to PCI bridge> on motherboard
pci3: <PCI bus> on pcib3
pcib4: <ServerWorks host to PCI bridge(unknown chipset)> on motherboard
IOAPIC #1 intpin 6 -> irq 5
pci4: <PCI bus> on pcib4
aac0: <Adaptec SCSI RAID 2120S> mem 0xf0000000-0xf7ffffff irq 5 at device 4.0 on
 pci4
aac0: i960RX 100MHz, 112MB cache memory, optional battery present
aac0: Kernel 4.0-0, Build 6008, S/N b8b31a
aac0: Supported Options=1f7e<CLUSTERS,WCACHE,DATA64,HOSTTIME,RAID50,WINDOW4GB,SO
FTERR,NORECOND,SGMAP64,ALARM,NONDASD>
pcib5: <ServerWorks host to PCI bridge(unknown chipset)> on motherboard
pci5: <PCI bus> on pcib5
pcib6: <ServerWorks host to PCI bridge(unknown chipset)> on motherboard
IOAPIC #1 intpin 13 -> irq 7
IOAPIC #1 intpin 14 -> irq 9
pci6: <PCI bus> on pcib6
em1: <Intel(R) PRO/1000 Network Connection, Version - 1.7.16> port 0x2500-0x253f
 mem 0xeefe0000-0xeeffffff irq 7 at device 8.0 on pci6
em1:  Speed:N/A  Duplex:N/A
em2: <Intel(R) PRO/1000 Network Connection, Version - 1.7.16> port 0x2540-0x257f
 mem 0xeefc0000-0xeefdffff irq 9 at device 8.1 on pci6
em2:  Speed:N/A  Duplex:N/A
pcib8: <ServerWorks host to PCI bridge(unknown chipset)> on motherboard
IOAPIC #1 intpin 11 -> irq 10
IOAPIC #1 intpin 12 -> irq 11
pci8: <PCI bus> on pcib8
mpt0: <LSILogic 1030 Ultra4 Adapter> port 0x2600-0x26ff mem 0xecfe0000-0xecfefff
f,0xecff0000-0xecffffff irq 10 at device 7.0 on pci8
mpt1: <LSILogic 1030 Ultra4 Adapter> port 0x2700-0x27ff mem 0xecfc0000-0xecfcfff
f,0xecfd0000-0xecfdffff irq 11 at device 7.1 on pci8
orm0: <Option ROMs> at iomem 0xc0000-0xc7fff,0xc8000-0xc97ff,0xc9800-0xc9fff,0xc
a000-0xcdfff on isa0
pmtimer0 on isa0
fdc0: <NEC 72065B or clone> at port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on isa0
fdc0: FIFO enabled, 8 bytes threshold
fd0: <1440-KB 3.5" drive> on fdc0 drive 0
atkbdc0: <Keyboard controller (i8042)> at port 0x60,0x64 on isa0
atkbd0: <AT Keyboard> flags 0x1 irq 1 on atkbdc0
kbd0 at atkbd0
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
sio0: configured irq 4 not in bitmap of probed irqs 0
sio0 at port 0x3f8-0x3ff irq 4 flags 0x10 on isa0
sio0: type 8250
sio1: configured irq 3 not in bitmap of probed irqs 0
APIC_IO: Testing 8254 interrupt delivery
APIC_IO: routing 8254 via IOAPIC #0 intpin 2
IP packet filtering initialized, divert enabled, rule-based forwarding enabled,
default to accept, logging limited to 100 packets/entry by default
IPsec: Initialized Security Association Processing.
IP Filter: v3.4.31 initialized.  Default = block all, Logging = enabled
SMP: AP CPU #1 Launched!
acd0: CDROM <CD-224E> at ata0-master PIO4
Waiting 15 seconds for SCSI devices to settle
aacd0: <RAID 1 (Mirror)> on aac0
aacd0: 34712MB (71091456 sectors)
aacd1: <Volume> on aac0
aacd1: 34712MB (71091456 sectors)
SMP: AP CPU #3 Launched!
SMP: AP CPU #2 Launched!


>Description:
	4.9 machine running various apps in null-mounted jails crash after various time:  reboot after panic: null_checkvp

	null-mounted jail: 
		jail root: /jail/app (/dev, /var, /tmp, /mnt; +symlinks {/bin,/etc,/usr}-> /mnt/$1)
		read-only content: /jail/ro (/bin, /etc, /usr)
		at startup: mount_null -o ro,noatime /jail/ro /jail/app/mnt

	After starting apps (espec. dnscache) systat shows, that allocated numvnodes started to increment (sometimes 100/sec) and also freevnodes the same way (but freevnodes < numvnodes).
	When numvnodes reach 33695 the system hang up: 2-3 sec while numvnodes does not change, after that kernel panic msg appear.


The gdb output:

GNU gdb 4.18 (FreeBSD)
Copyright 1998 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "i386-unknown-freebsd"...Deprecated bfd_read called a
t /usr/src/gnu/usr.bin/binutils/gdb/../../../../contrib/gdb/gdb/dbxread.c line 2
627 in elfstab_build_psymtabs
Deprecated bfd_read called at /usr/src/gnu/usr.bin/binutils/gdb/../../../../cont
rib/gdb/gdb/dbxread.c line 933 in fill_symbuf

SMP 4 cpus
IdlePTD at phsyical address 0x004c4000
initial pcb at physical address 0x003fc5c0
panicstr: rslock: cpu: 0, addr: 0xc03fd034, lock: 0x00000001
panic messages:
---
panic: null_checkvp
mp_lock = 00000001; cpuid = 0; lapic.id = 00000000
boot() called on cpu#0
syncing disks... 16 2 1 1 1 1 1 1 1 panic: rslock: cpu: 0, addr: 0xc03fd034, loc
k: 0x00000001
mp_lock = 00000001; cpuid = 0; lapic.id = 00000000
boot() called on cpu#0
Uptime: 30m53s

dumping to dev #aacd/0x20009, offset 4194472
dump 2047 2046 2045 2044 2043 2042 2041 2040 2039 2038 2037 2036 2035 2034 2033
2032 2031 2030 2029 2028 2027 2026 2025 2024 2023 2022 2021 2020 2019 2018 2017
... truncated ...
 6 5 4 3 2 1 0 succeeded
 aac0: shutting down controller...
 ---
#0  dumpsys () at /usr/src/sys/kern/kern_shutdown.c:487
487             if (dumping++) {
(kgdb) bt
#0  dumpsys () at /usr/src/sys/kern/kern_shutdown.c:487
#1  0xc01f930b in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:316
#2  0xc01f9764 in poweroff_wait (junk=0xc03318ce, howto=0)
    at /usr/src/sys/kern/kern_shutdown.c:595
#3  0xc03318ce in bsl1 ()
#4  0xc02285d8 in vdrop (vp=0xf6fd7c40) at /usr/src/sys/kern/vfs_subr.c:1692
#5  0xc0227b5a in brelvp (bp=0xd61f7b2c) at /usr/src/sys/kern/vfs_subr.c:1074
#6  0xc02204a8 in vfs_vmio_release (bp=0xd61f7b2c)
    at /usr/src/sys/kern/vfs_bio.c:1361
#7  0xc021fead in brelse (bp=0xd61f7b2c) at /usr/src/sys/kern/vfs_bio.c:1140
#8  0xc0227533 in vinvalbuf (vp=0xf6fd7c40, flags=0, cred=0x0, p=0xe84d2860,
    slpflag=0, slptimeo=0) at /usr/src/sys/kern/vfs_subr.c:872
#9  0xc02d7a8a in ffs_truncate (vp=0xf6fd7c40, length=0, flags=0, cred=0x0,
    p=0xe84d2860) at /usr/src/sys/ufs/ffs/ffs_inode.c:199
#10 0xc02e2b3c in ufs_inactive (ap=0xf7098b08)
    at /usr/src/sys/ufs/ufs/ufs_inode.c:89
#11 0xc02e8181 in ufs_vnoperate (ap=0xf7098b08)
    at /usr/src/sys/ufs/ufs/ufs_vnops.c:2376
#12 0xc02284ef in vput (vp=0xf6fd7c40) at vnode_if.h:815
#13 0xc02db6e4 in handle_workitem_remove (dirrem=0xc9cb3b40)
    at /usr/src/sys/ufs/ffs/ffs_softdep.c:2852
#14 0xc02d8d55 in process_worklist_item (matchmnt=0x0, flags=0)
    at /usr/src/sys/ufs/ffs/ffs_softdep.c:716
#15 0xc02d8bfe in softdep_process_worklist (matchmnt=0x0)
    at /usr/src/sys/ufs/ffs/ffs_softdep.c:622
#16 0xc01f9141 in boot (howto=256) at /usr/src/sys/kern/kern_shutdown.c:261
#17 0xc01f9764 in poweroff_wait (junk=0xc0383662, howto=-150369236)
    at /usr/src/sys/kern/kern_shutdown.c:595
#18 0xc0230819 in null_checkvp (vp=0xf70815c0,
    fil=0xc0383780 "/usr/src/sys/miscfs/nullfs/null_vnops.c", lno=824)
    at /usr/src/sys/miscfs/nullfs/null_subr.c:319
#19 0xc0231728 in null_getvobject (ap=0xf7098c2c)
    at /usr/src/sys/miscfs/nullfs/null_vnops.c:824
#20 0xc0227032 in getnewvnode (tag=VT_NULL, mp=0xc9ceb600, vops=0xc9785500,
    vpp=0xf7098c98) at vnode_if.h:1421
#21 0xc0230617 in null_node_alloc (mp=0xc9ceb600, lowervp=0xf7081500,
    vpp=0xf7098c98) at /usr/src/sys/miscfs/nullfs/null_subr.c:172
#22 0xc0230776 in null_node_create (mp=0xc9ceb600, lowervp=0xf7081500,
    newvpp=0xf7098cc8) at /usr/src/sys/miscfs/nullfs/null_subr.c:256
#23 0xc02310f7 in null_lookup (ap=0xf7098d18)
    at /usr/src/sys/miscfs/nullfs/null_vnops.c:396
#24 0xc0226289 in lookup (ndp=0xf7098edc) at vnode_if.h:52
#25 0xc0225d84 in namei (ndp=0xf7098edc) at /usr/src/sys/kern/vfs_lookup.c:153
#26 0xc01f0596 in execve (p=0xe84d2860, uap=0xf7098f80)
    at /usr/src/sys/kern/kern_exec.c:165
#27 0xc033380d in syscall2 (frame={tf_fs = 47, tf_es = 47, tf_ds = 47,
      tf_edi = 0, tf_esi = 14, tf_ebp = -1077936992, tf_isp = -150368300,
      tf_ebx = -1077936512, tf_edx = -1077936760, tf_ecx = 134528480,
      tf_eax = 59, tf_trapno = 12, tf_err = 2, tf_eip = 403307776, tf_cs = 31,
      tf_eflags = 663, tf_esp = -1077937036, tf_ss = 47})
    at /usr/src/sys/i386/i386/trap.c:1175
#28 0xc031e0db in Xint0x80_syscall ()
#29 0x8048d2b in ?? ()
#30 0x8048a66 in ?? ()
#31 0x80486b6 in ?? ()
(kgdb) q
	

	NULLFS presented in RELENG_4 years ago, but it still in beta: some env works fine for lots of people for years ago, but there lots of bugs - as man page describe it. When talking about a prod release, it must not contain not production stable parts. Instead replying "do not use" and "use nfs" please help to fix it.
(I help to test anything that need to test ;)
		
>How-To-Repeat:
	Just start up nullfs and daemons (tested: bind9, oops, djbdns's dnscache+daemontools).

	I dont know whether an user can do something nasty vnode allocation and halt the system.
	
>Fix:

	dont know. Do not use nullfs, even not for that apps but for any.


>Release-Note:
>Audit-Trail:
>Unformatted:

More information about the freebsd-i386 mailing list