kern/111734: FreeBSD 6.2-release (p3) panic in NFS client code/VFS subsystem

Kai Storbeck kai at xs4all.nl
Mon Apr 16 19:20:15 UTC 2007


>Number:         111734
>Category:       kern
>Synopsis:       FreeBSD 6.2-release (p3) panic in NFS client code/VFS subsystem
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    freebsd-bugs
>State:          open
>Quarter:        
>Keywords:       
>Date-Required:
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Mon Apr 16 19:20:14 GMT 2007
>Closed-Date:
>Last-Modified:
>Originator:     Kai Storbeck
>Release:        FreeBSD 6.2 stable (p3)
>Organization:
>Environment:
FreeBSD webserver 6.2-RELEASE-p3 FreeBSD 6.2-RELEASE-p3 #2: Thu Apr  5 14:15:19 CEST 2007     kai at buildmachine:/usr/src/sys/i386/compile/SMP-DEBUG  i386

>Description:
We have 4 machines on freebsd 6.2 (dual 2ghz dualcore xeon). They are
serving webpages (static content with a few default scripts) from an NFS
server. They _all_ regularly panic since we've upgraded the machines to
new hardware + freebsd 6.2 stable.

This is the backtrace: 

[GDB will not be able to debug user-mode threads: /usr/lib/libthread_db.so: Undefined symbol "ps_pglobal_lookup"]
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "i386-marcel-freebsd".

Unread portion of the kernel message buffer:
panic: bundirty: buffer 0xdc510d50 still on queue 1
cpuid = 0
Uptime: 8h14m42s
Dumping 3326 MB (2 chunks)
  chunk 0: 1MB (156 pages) ... ok
  chunk 1: 3327MB (851552 pages) 3311 3295 3279 3263 3247 3231 3215 3199 3183 3167 3151 3135 3119 3103 3087 3071 3055 3039 3023 3007 2991 2975 2959 2943 2927 2911 2895 2879 2863 2847 2831 2815 2799 2783 2767 2751 2735 2719 2703 2687 2671 2655 2639 2623 2607 2591 2575 2559 2543 2527 2511 2495 2479 2463 2447 2431 2415 2399 2383 2367 2351 2335 2319 2303 2287 2271 2255 2239 2223 2207 2191 2175 2159 2143 2127 2111 2095 2079 2063 2047 2031 2015 1999 1983 1967 1951 1935 1919 1903 1887 1871 1855 1839 1823 1807 1791 1775 1759 1743 1727 1711 1695 1679 1663 1647 1631 1615 1599 1583 1567 1551 1535 1519 1503 1487 1471 1455 1439 1423 1407 1391 1375 1359 1343 1327 1311 1295 1279 1263 1247 1231 1215 1199 1183 1167 1151 1135 1119 1103 1087 1071 1055 1039 1023 1007 991 975 959 943 927 911 895 879 863 847 831 815 799 783 767 751 735 719 703 687 671 655 639 623 607 591 575 559 543 527 511 495 479 463 447 431 415 399 383 367 351 335 319 303 287 271 255 239 223 207 191 175 159 143 127 111 95 79 63
  47 31 15

#0  doadump () at pcpu.h:165
165             __asm __volatile("movl %%fs:0,%0" : "=r" (td));
(kgdb) bt
#0  doadump () at pcpu.h:165
#1  0xc06708e8 in boot (howto=260) at ../../../kern/kern_shutdown.c:409
#2  0xc0670bca in panic (
    fmt=0xc08f7fec "bundirty: buffer %p still on queue %d")
    at ../../../kern/kern_shutdown.c:565
#3  0xc06b3409 in bundirty (bp=0xdc510d50) at ../../../kern/vfs_bio.c:1036
#4  0xc06b3d6c in brelse (bp=0xdc510d50) at ../../../kern/vfs_bio.c:1351
#5  0xc075e18e in nfs_writebp (bp=0xdc510d50, force=1, td=0xcf6db900)
    at ../../../nfsclient/nfs_vnops.c:2982
#6  0xc075e3f8 in nfs_bwrite (bp=0xdc510d50) at pcpu.h:162
#7  0xc06b5b7d in getblk (vp=0xca312aa0, blkno=8438, size=6535, slpflag=256,
    slptimeo=0, flags=0) at buf.h:412
#8  0xc0751299 in nfs_getcacheblk (vp=0xca312aa0, bn=8438, size=6535,
    td=0xcf6db900) at ../../../nfsclient/nfs_bio.c:1253
#9  0xc0750e62 in nfs_write (ap=0x0) at ../../../nfsclient/nfs_bio.c:1069
#10 0xc08894f2 in VOP_WRITE_APV (vop=0xc097fc00, a=0xec4babf4)
    at vnode_if.c:698
#11 0xc06cea9a in vn_write (fp=0xc96c7ca8, uio=0xec4bacbc,
    active_cred=0xc8e19380, flags=0, td=0xcf6db900) at vnode_if.h:372
#12 0xc0692867 in dofilewrite (td=0xcf6db900, fd=5, fp=0xc96c7ca8,
    auio=0xec4bacbc, offset=Unhandled dwarf expression opcode 0x93
) at file.h:252
#13 0xc069270b in kern_writev (td=0xcf6db900, fd=5, auio=0xec4bacbc)
    at ../../../kern/sys_generic.c:402
#14 0xc0692631 in write (td=0xcf6db900, uap=0x0)
    at ../../../kern/sys_generic.c:326
#15 0xc087842f in syscall (frame=
      {tf_fs = 137101371, tf_es = 1210646587, tf_ds = -1078001605, tf_edi = 137121792, tf_esi = 1210674680, tf_ebp = -1077942760, tf_isp = -330584732, tf_ebx = 1210584448, tf_edx = 4096, tf_ecx = 0, tf_eax = 4, tf_trapno = 0, tf_err = 2, tf_eip = 1210524463, tf_cs = 51, tf_eflags = 530, tf_esp = -1077942788, tf_ss = 59}) at ../../../i386/i386/trap.c:983
#16 0xc08649ef in Xint0x80_syscall () at ../../../i386/i386/exception.s:200
#17 0x00000033 in ?? ()
Previous frame inner to this frame (corrupt stack?)


This is the dmesg:

Copyright (c) 1992-2007 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
        The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 6.2-RELEASE-p3 #2: Thu Apr  5 14:15:19 CEST 2007
    kai at buildmachine:/usr/src/sys/i386/compile/SMP-DEBUG
WARNING: DIAGNOSTIC option enabled, expect reduced performance.
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: Intel(R) Xeon(R) CPU            5130  @ 2.00GHz (2000.08-MHz 686-class CPU)
  Origin = "GenuineIntel"  Id = 0x6f6  Stepping = 6
  Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
  Features2=0x4e33d<SSE3,RSVD2,MON,DS_CPL,VMX,TM2,<b9>,CX16,<b14>,<b15>,<b18>>
  AMD Features=0x20100000<NX,LM>
  AMD Features2=0x1<LAHF>
  Cores per package: 2
real memory  = 3489005568 (3327 MB)
avail memory = 3414216704 (3256 MB)
ACPI APIC Table: <PTLTD          APIC  >
FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
 cpu0 (BSP): APIC ID:  0
 cpu1 (AP): APIC ID:  1
 cpu2 (AP): APIC ID:  6
 cpu3 (AP): APIC ID:  7
ioapic0 <Version 2.0> irqs 0-23 on motherboard
ioapic1 <Version 2.0> irqs 24-47 on motherboard
kbd1 at kbdmux0
ath_hal: 0.9.17.2 (AR5210, AR5211, AR5212, RF5111, RF5112, RF2413, RF5413)
acpi0: <PTLTD   RSDT> on motherboard
acpi0: Power Button (fixed)
Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x1008-0x100b on acpi0
cpu0: <ACPI CPU> on acpi0
acpi_throttle0: <ACPI CPU Throttling> on cpu0
cpu1: <ACPI CPU> on acpi0
acpi_throttle1: <ACPI CPU Throttling> on cpu1
acpi_throttle1: failed to attach P_CNT
device_attach: acpi_throttle1 attach returned 6
cpu2: <ACPI CPU> on acpi0
acpi_throttle2: <ACPI CPU Throttling> on cpu2
acpi_throttle2: failed to attach P_CNT
device_attach: acpi_throttle2 attach returned 6
cpu3: <ACPI CPU> on acpi0
acpi_throttle3: <ACPI CPU Throttling> on cpu3
acpi_throttle3: failed to attach P_CNT
device_attach: acpi_throttle3 attach returned 6
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
pcib1: <ACPI PCI-PCI bridge> at device 2.0 on pci0
pci1: <ACPI PCI bus> on pcib1
pcib2: <ACPI PCI-PCI bridge> irq 16 at device 0.0 on pci1
pci2: <ACPI PCI bus> on pcib2
pcib3: <ACPI PCI-PCI bridge> irq 16 at device 0.0 on pci2
pci3: <ACPI PCI bus> on pcib3
pcib4: <ACPI PCI-PCI bridge> irq 18 at device 2.0 on pci2
pci4: <ACPI PCI bus> on pcib4
em0: <Intel(R) PRO/1000 Network Connection Version - 6.2.9> port 0x2000-0x201f mem 0xdd200000-0xdd21ffff irq 18 at device 0.0 on pci4
em0: Ethernet address: 00:30:48:32:b1:ac
em1: <Intel(R) PRO/1000 Network Connection Version - 6.2.9> port 0x2020-0x203f mem 0xdd220000-0xdd23ffff irq 19 at device 0.1 on pci4
em1: Ethernet address: 00:30:48:32:b1:ad
pcib5: <ACPI PCI-PCI bridge> at device 0.3 on pci1
pci5: <ACPI PCI bus> on pcib5
aac0: <Adaptec SCSI RAID 2020ZCR> mem 0xdc000000-0xdcffffff,0xdd000000-0xdd1fffff,0xd8000000-0xdbffffff irq 28 at device 2.0 on pci5
aac0: New comm. interface enabled
aac0: Adaptec Raid Controller 2.0.0-1
aacp0: <SCSI Passthrough Bus> on aac0
aacp1: <SCSI Passthrough Bus> on aac0
pcib6: <ACPI PCI-PCI bridge> at device 4.0 on pci0
pci6: <ACPI PCI bus> on pcib6
pcib7: <ACPI PCI-PCI bridge> at device 6.0 on pci0
pci7: <ACPI PCI bus> on pcib7
pci0: <base peripheral> at device 8.0 (no driver attached)
uhci0: <UHCI (generic) USB controller> port 0x1800-0x181f irq 17 at device 29.0 on pci0
uhci0: [GIANT-LOCKED]
usb0: <UHCI (generic) USB controller> on uhci0
usb0: USB revision 1.0
uhub0: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub0: 2 ports with 2 removable, self powered
uhci1: <UHCI (generic) USB controller> port 0x1820-0x183f irq 19 at device 29.1 on pci0
uhci1: [GIANT-LOCKED]
usb1: <UHCI (generic) USB controller> on uhci1
usb1: USB revision 1.0
uhub1: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub1: 2 ports with 2 removable, self powered
uhci2: <UHCI (generic) USB controller> port 0x1840-0x185f irq 18 at device 29.2 on pci0
uhci2: [GIANT-LOCKED]
usb2: <UHCI (generic) USB controller> on uhci2
usb2: USB revision 1.0
uhub2: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub2: 2 ports with 2 removable, self powered
uhci3: <UHCI (generic) USB controller> port 0x1860-0x187f irq 16 at device 29.3 on pci0
uhci3: [GIANT-LOCKED]
usb3: <UHCI (generic) USB controller> on uhci3
usb3: USB revision 1.0
uhub3: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub3: 2 ports with 2 removable, self powered
ehci0: <EHCI (generic) USB 2.0 controller> mem 0xdd700000-0xdd7003ff irq 17 at device 29.7 on pci0
ehci0: [GIANT-LOCKED]
usb4: EHCI version 1.0
usb4: companion controllers, 2 ports each: usb0 usb1 usb2 usb3
usb4: <EHCI (generic) USB 2.0 controller> on ehci0
usb4: USB revision 2.0
uhub4: Intel EHCI root hub, class 9/0, rev 2.00/1.00, addr 1
uhub4: 8 ports with 8 removable, self powered
pcib8: <ACPI PCI-PCI bridge> at device 30.0 on pci0
pci8: <ACPI PCI bus> on pcib8
pci8: <display, VGA> at device 1.0 (no driver attached)
isab0: <PCI-ISA bridge> at device 31.0 on pci0
isa0: <ISA bus> on isab0
atapci0: <Intel 63XXESB2 UDMA100 controller> port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0x1880-0x188f at device 31.1 on pci0
ata0: <ATA channel 0> on atapci0
ata1: <ATA channel 1> on atapci0
ichsmb0: <SMBus controller> port 0x1100-0x111f irq 19 at device 31.3 on pci0
ichsmb0: [GIANT-LOCKED]
smbus0: <System Management Bus> on ichsmb0
smb0: <SMBus generic I/O> on smbus0
acpi_button0: <Power Button> on acpi0
atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
sio0: <16550A-compatible COM port> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
sio0: type 16550A, console
sio1: <16550A-compatible COM port> port 0x2f8-0x2ff irq 3 on acpi0
sio1: type 16550A
fdc0: <floppy drive controller> port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on acpi0
fdc0: does not respond
device_attach: fdc0 attach returned 6
fdc0: <floppy drive controller> port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on acpi0
fdc0: does not respond
device_attach: fdc0 attach returned 6
pmtimer0 on isa0
orm0: <ISA Option ROMs> at iomem 0xc0000-0xcafff,0xcb000-0xcf7ff on isa0
ppc0: parallel port not found.
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
Timecounters tick every 1.000 msec
IPv6 packet filtering initialized, default to accept, logging limited to 100 packets/entry
ipfw2 (+ipv6) initialized, divert loadable, rule-based forwarding enabled, default to accept, logging limited to 100 packets/entry by default
acd0: DMA limited to UDMA33, controller found non-ATA66 cable
acd0: DVDROM <MATSHITADVD-ROM SR-8178/PZ16> at ata0-slave UDMA33
aacd0: <RAID 1 (Mirror)> on aac0
aacd0: 139981MB (286681856 sectors)
ses0 at aacp0 bus 0 target 6 lun 0
ses0: <SUPER GEM318 0> Fixed unknown SCSI-2 device 
ses0: 3.300MB/s transfers
ses0: SAF-TE Compliant Device
pass0 at aacp0 bus 0 target 0 lun 0
pass0: <SEAGATE ST3146707LC 0005> Fixed unknown SCSI-3 device 
pass0: 3.300MB/s transfers
pass1 at aacp0 bus 0 target 1 lun 0
pass1: <SEAGATE ST3146707LC 0005> Fixed unknown SCSI-3 device 
pass1: 3.300MB/s transfers
SMP: AP CPU #1 Launched!
SMP: AP CPU #2 Launched!
SMP: AP CPU #3 Launched!

Kernel config:

include GENERIC
# saves this config in the kernel (strings -n 3 kernel | sed -n 's/^___//p')
options         INCLUDE_CONFIG_FILE

# higher limits
options         MAXDSIZ="(1024UL*1024*1024)"
options         MAXSSIZ="(1024UL*1024*1024)"
options         DFLDSIZ="(256UL*1024*1024)"
# not needed options
nooptions MATH_EMULATE
nooptions MD_ROOT
nooptions UCONSOLE

options         ASR_COMPAT      # asr old ioctls support, needed by raidutils
options         SEMMNS=256      # Allow enough semaphores for pgsql

options         IPFIREWALL              # firewall
options         IPFIREWALL_VERBOSE
options         IPFIREWALL_VERBOSE_LIMIT=100
options         IPFIREWALL_FORWARD
options         IPFIREWALL_DEFAULT_TO_ACCEPT
options         IPV6FIREWALL
options         IPV6FIREWALL_VERBOSE
options         IPV6FIREWALL_VERBOSE_LIMIT=100
options         IPV6FIREWALL_DEFAULT_TO_ACCEPT
options         TCP_DROP_SYNFIN

options         SMP                     # Symmetric MultiProcessor Kernel

makeoptions DEBUG=-g
options     INVARIANTS
options     INVARIANT_SUPPORT
options     DIAGNOSTIC


device smb
device smbus
device ichsmb


device iic
device iicbus
device iicbb
device iicsmb


>How-To-Repeat:

>Fix:

>Release-Note:
>Audit-Trail:
>Unformatted:


More information about the freebsd-bugs mailing list