Occasional panics during shutdown

[Stefan Walter]

Hi,

for about half a year now (I think - it could be more), my 5-STABLE
machine has been panicing occasionally and irregularly - about once or
twice a week - during shutdown. (I did multiple updates of the system
during that time, which didn't change anything.) It always happens after
the disks are sync'd and when the machine should either turn off or
reboot.

Here's a backtrace of the latest core dump:

[GDB will not be able to debug user-mode threads: /usr/lib/libthread_db.so: Undefined symbol "ps_pglobal_lookup"]
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "i386-marcel-freebsd".

Unread portion of the kernel message buffer:
<118>Writing entropy file:
<118>.
<118>Terminated
<118>.
Waiting (max 60 seconds) for system process `vnlru' to stop...done
Waiting (max 60 seconds) for system process `bufdaemon' to stop...done
Waiting (max 60 seconds) for system process `syncer' to stop...
Syncing disks, vnodes remaining...5 2 3 1 0 0 0 done
No buffers busy after final sync
Uptime: 29m33s
ukphy0: detached
miibus0: detached
kernel trap 12 with interrupts disabled


Fatal trap 12: page fault while in kernel mode
fault virtual address	= 0x24
fault code		= supervisor read, page not present
instruction pointer	= 0x8:0xc0541143
stack pointer	        = 0x10:0xd4200c68
frame pointer	        = 0x10:0xd4200c7c
code segment		= base 0x0, limit 0xfffff, type 0x1b
			= DPL 0, pres 1, def32 1, gran 1
processor eflags	= resume, IOPL = 0
current process		= 22 (irq11: vr0 uhci0)
trap number		= 12
panic: page fault
Uptime: 29m38s
Dumping 511 MB
 16 32 48 64 80 96 112 128 144 160 176 192 208 224 240 256 272 288 304 320 336 352 368 384 400 416 432 448 464 480 496

#0  doadump () at pcpu.h:160
160	pcpu.h: No such file or directory.
	in pcpu.h
(kgdb) bt
#0  doadump () at pcpu.h:160
#1  0xc052217f in boot (howto=16644) at /usr/src/sys/kern/kern_shutdown.c:412
#2  0xc0522414 in panic (fmt=0xc06d1ab9 "%s")
    at /usr/src/sys/kern/kern_shutdown.c:568
#3  0xc06a82b8 in trap_fatal (frame=0xd4200c28, eva=36)
    at /usr/src/sys/i386/i386/trap.c:817
#4  0xc06a7aa5 in trap (frame=
      {tf_fs = 24, tf_es = 16, tf_ds = 16, tf_edi = -1047002366, tf_esi = -1046988544, tf_ebp = -736097156, tf_isp = -736097196, tf_ebx = -1047002368, tf_edx = -1047046128, tf_ecx = -1045009532, tf_eax = 0, tf_trapno = 12, tf_err = 0, tf_eip = -1068232381, tf_cs = 8, tf_eflags = 65683, tf_esp = 16, tf_ss = 0})
    at /usr/src/sys/i386/i386/trap.c:255
#5  0xc069b14a in calltrap () at /usr/src/sys/i386/i386/exception.s:140
#6  0x00000018 in ?? ()
#7  0x00000010 in ?? ()
#8  0x00000010 in ?? ()
#9  0xc1980302 in ?? ()
#10 0xc1983900 in ?? ()
#11 0xd4200c7c in ?? ()
#12 0xd4200c54 in ?? ()
#13 0xc1980300 in ?? ()
#14 0xc1975810 in ?? ()
#15 0xc1b66b84 in ?? ()
#16 0x00000000 in ?? ()
#17 0x0000000c in ?? ()
#18 0x00000000 in ?? ()
#19 0xc0541143 in propagate_priority (td=0xc1980300)
    at /usr/src/sys/kern/subr_turnstile.c:243
#20 0xc05417be in turnstile_wait (ts=0xc19758c0, lock=0xc1b66b84, owner=0xc1980300)
    at /usr/src/sys/kern/subr_turnstile.c:556
#21 0xc0519c3d in _mtx_lock_sleep (m=0xc1b66b84, td=0xc1983900, opts=0, file=0x0, 
    line=0) at /usr/src/sys/kern/kern_mutex.c:552
#22 0xc062b1ab in vr_intr (arg=0xc1b66000) at /usr/src/sys/pci/if_vr.c:1232
#23 0xc050e9fd in ithread_loop (arg=0xc197d700)
    at /usr/src/sys/kern/kern_intr.c:547
#24 0xc050dc90 in fork_exit (callout=0xc050e8ac <ithread_loop>, arg=0xc197d700, 
    frame=0xd4200d38) at /usr/src/sys/kern/kern_fork.c:791
#25 0xc069b1ac in fork_trampoline () at /usr/src/sys/i386/i386/exception.s:209
(kgdb) quit

And here the dmesg output:

Copyright (c) 1992-2005 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
	The Regents of the University of California. All rights reserved.
FreeBSD 5.4-STABLE #1: Thu Sep 29 11:30:38 CEST 2005
    root at kyuzo.dunkelkammer.void:/usr/obj/usr/src/sys/KYUZO
WARNING: debug.mpsafenet forced to 0 as ipsec requires Giant
WARNING: MPSAFE network stack disabled, expect reduced performance.
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: mobile AMD Athlon(tm) XP-M 2400+ (1800.08-MHz 686-class CPU)
  Origin = "AuthenticAMD"  Id = 0x6a0  Stepping = 0
  Features=0x383f9ff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,MMX,FXSR,SSE>
  AMD Features=0xc0480000<MP,AMIE,DSP,3DNow!>
real memory  = 536805376 (511 MB)
avail memory = 515624960 (491 MB)
netsmb_dev: loaded
npx0: <math processor> on motherboard
npx0: INT 16 interface
acpi0: <AMIINT VIA_K7> on motherboard
acpi0: Power Button (fixed)
Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x808-0x80b on acpi0
cpu0: <ACPI CPU> on acpi0
acpi_button0: <Power Button> on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
agp0: <VIA 8377 (Apollo KT400/KT400A/KT600) host to PCI bridge> mem 0xe0000000-0xe7ffffff at device 0.0 on pci0
pcib1: <PCI-PCI bridge> at device 1.0 on pci0
pci1: <PCI bus> on pcib1
pci1: <display, VGA> at device 0.0 (no driver attached)
pcm0: <CMedia CMI8738> port 0xec00-0xecff irq 5 at device 6.0 on pci0
sym0: <896> port 0xe400-0xe4ff mem 0xd6000000-0xd6001fff,0xd8000000-0xd80003ff irq 10 at device 7.0 on pci0
sym0: Tekram NVRAM, ID 7, Fast-40, LVD, parity checking
sym1: <896> port 0xe800-0xe8ff mem 0xdc000000-0xdc001fff,0xde000000-0xde0003ff irq 10 at device 7.1 on pci0
sym1: Tekram NVRAM, ID 7, Fast-40, SE, parity checking
fwohci0: <VIA Fire II (VT6306)> port 0xe000-0xe07f mem 0xdbfdf800-0xdbfdffff irq 5 at device 10.0 on pci0
fwohci0: OHCI version 1.0 (ROM=1)
fwohci0: No. of Isochronous channels is 4.
fwohci0: EUI64 20:00:00:00:04:00:20:a6
fwohci0: Phy 1394a available S400, 3 ports.
fwohci0: Link S400, max_rec 2048 bytes.
firewire0: <IEEE1394(FireWire) bus> on fwohci0
sbp0: <SBP-2/SCSI over FireWire> on firewire0
fwohci0: Initiate bus reset
fwohci0: node_id=0xc800ffc0, gen=1, CYCLEMASTER mode
firewire0: 1 nodes, maxhop <= 0, cable IRM = 0 (me)
firewire0: bus manager 0 (me)
uhci0: <VIA 83C572 USB controller> port 0xdc00-0xdc1f irq 11 at device 16.0 on pci0
uhci0: LegSup = 0xa000
usb0: <VIA 83C572 USB controller> on uhci0
usb0: USB revision 1.0
usbd_get_string: getting lang failed, using 0
uhub0: VIA UHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub0: 2 ports with 2 removable, self powered
pci0: <serial bus, USB> at device 16.3 (no driver attached)
isab0: <PCI-ISA bridge> at device 17.0 on pci0
isa0: <ISA bus> on isab0
atapci0: <VIA 8235 UDMA133 controller> port 0xfc00-0xfc0f,0x376,0x170-0x177,0x3f6,0x1f0-0x1f7 at device 17.1 on pci0
ata0: channel #0 on atapci0
ata1: channel #1 on atapci0
vr0: <VIA VT6102 Rhine II 10/100BaseTX> port 0xd800-0xd8ff mem 0xdbfdf600-0xdbfdf6ff irq 11 at device 18.0 on pci0
miibus0: <MII bus> on vr0
ukphy0: <Generic IEEE 802.3u media interface> on miibus0
ukphy0:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
vr0: Ethernet address: 00:0c:76:18:9c:4c
acpi_button1: <Sleep Button> on acpi0
speaker0: <PC speaker> port 0x61 on acpi0
atkbdc0: <Keyboard controller (i8042)> port 0x64,0x60 irq 1 on acpi0
atkbd0: <AT Keyboard> flags 0x1 irq 1 on atkbdc0
kbd0 at atkbd0
psm0: <PS/2 Mouse> irq 12 on atkbdc0
psm0: model Generic PS/2 mouse, device ID 0
orm0: <ISA Option ROMs> at iomem 0xcd000-0xcefff,0xc0000-0xccfff on isa0
pmtimer0 on isa0
ppc0: parallel port not found.
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <12 virtual consoles, flags=0x300>
sio0: configured irq 4 not in bitmap of probed irqs 0
sio0: port may not be enabled
sio0 at port 0x3f8-0x3ff irq 4 flags 0x10 on isa0
sio0: type 8250 or not responding
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
Timecounter "TSC" frequency 1800079258 Hz quality 800
Timecounters tick every 10.000 msec
IPsec: Initialized Security Association Processing.
ad0: 38166MB <ST340824A/2.03> [77545/16/63] at ata0-master UDMA100
Waiting 5 seconds for SCSI devices to settle
da0 at sym0 bus 0 target 0 lun 0
da0: <SEAGATE ST318438LW 0003> Fixed Direct Access SCSI-3 device 
da0: 80.000MB/s transfers (40.000MHz, offset 31, 16bit), Tagged Queueing Enabled
da0: 19001MB (38914049 512 byte sectors: 255H 63S/T 2422C)
cd0 at sym1 bus 0 target 4 lun 0
cd0: <TEAC CD-R55S 1.0L> Removable CD-ROM SCSI-2 device 
cd0: 10.000MB/s transfers (10.000MHz, offset 15)
cd0: Attempt to query device size failed: NOT READY, Medium not present
da1 at sym1 bus 0 target 1 lun 0
da1: <IBM DDRS-34560W S97B> Fixed Direct Access SCSI-2 device 
da1: 40.000MB/s transfers (20.000MHz, offset 15, 16bit), Tagged Queueing Enabled
da1: 4357MB (8925000 512 byte sectors: 255H 63S/T 555C)
Mounting root from ufs:/dev/da0s1a
drm0: <ATI Radeon If R250 9000> port 0xb800-0xb8ff mem 0xcfef0000-0xcfefffff,0xa0000000-0xafffffff irq 11 at device 0.0 on pci1
info: [drm] AGP at 0xe0000000 128MB
info: [drm] Initialized radeon 1.11.0 20020828 on minor 0
info: [drm] Loading R200 Microcode

Any ideas?

Stefan