Experiencing hangs on SMP box with no console messages given for
clues. Details inside.
Tillman Hodgson
tillman at seekingfire.com
Thu Mar 8 13:30:55 UTC 2007
Howdy folks,
These has been happening every few days for a few weeks now. When it
occurs, there's no messages logged to the console or to syslog -- it
just silently hangs. I added the break-to-debugger option so that I can
at least reboot it remotely via the serial console.
I've been following the -current kernel fairly closely in hopes that it
was just due to a transitory -current problem. I don't mind rebuilding a
kernel with special options if it's useful -- I'll be rebuilding this
morning with WITNESS and INVARIANTS for sure. I have the core saved,
though I'm inexperienced with gdb.
FreeBSD/i386 (athena.seekingfire.prv) (ttyd0)
login:
telnet> send brk
KDB: enter: Line break on console
[thread pid 11 tid 100005 ]
Stopped at kdb_enter+0x2c: leave
db> ?
Bad character
?
db> help
print p examine x search set
write w delete d break b
dwatch watch dhwatch hwatch step s
continue c until next match trace
t alltrace where bt call show
ps gdb halt reboot reset kill
watchdog thread panic ahd_dump ahd_out ahd_in
ahd_unpause ahd_pause ahd_sunit
db> bt
Tracing pid 11 tid 100005 td 0xc3afe6c0
kdb_enter(c0956f95,c0,c3afe6c0,c3af7cc8,c3afb880,...) at kdb_enter+0x2c
siointr1(c3cb7b80,e25f0c84,c08cd60f,c3cb4000,c3afe6c0,...) at siointr1+0x3be
siointr(c3cb4000,c3afe6c0,0,0,c3bfb400,...) at siointr+0x4c
intr_execute_handlers(c3af7cc8,e25f0c94) at intr_execute_handlers+0xf3
Xapic_isr1() at Xapic_isr1+0x34
--- interrupt, eip = 0xc0baf599, esp = 0xe25f0cd4, ebp = 0xe25f0cd4 ---
acpi_cpu_c1(e25f0cec,c06e382d,c0a5cb60,c3afe6c0,c06e3ccc,...) at acpi_cpu_c1+0x5
acpi_cpu_idle(0,e25f0d24,c06b5db1,0,e25f0d38,...) at acpi_cpu_idle+0x15a
sched_idletd(0,e25f0d38,0,c3afdb40,0,...) at sched_idletd+0x8a
fork_exit(c06e3ccc,0,e25f0d38) at fork_exit+0x61
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe25f0d70, ebp = 0 ---
db> show proc
Process 11 (idle: cpu0) at 0xc3afdb40:
state: NORMAL
uid: 0 gids: 0
parent: pid 0 at 0xc0a58d80
ABI: null
threads: 1
100005 Run CPU 0 [idle: cpu0]
db> panic
panic: from debugger
cpuid = 0
Uptime: 2d22h24m3s
Physical memory: 1015 MB
Dumping 200 MB: 185 169 153 137 121 105 89 73 57 41 25 9
Dump complete
Automatic reboot in 15 seconds - press a key on the console to abort
[root at athena ~]# uname -a
FreeBSD athena.seekingfire.prv 7.0-CURRENT FreeBSD 7.0-CURRENT #0: Sun
Mar 4 21:08:19 CST 2007 toor at athena.seekingfire.prv
(/usr/src was synced the same day)
[root at athena /usr/src/sys/i386/conf]# diff ATHENA GENERIC
24c24
< ident ATHENA
---
> ident GENERIC
29c29
< ### makeoptions DEBUG=-g # Build kernel with gdb(1) debug symbols
---
> makeoptions DEBUG=-g # Build kernel with gdb(1) debug symbols
67,73c67,70
< ###options INVARIANTS # Enable calls of extra sanity checking
< ###options INVARIANT_SUPPORT # Extra sanity checks of internal structures, required by INVARIANTS
< ###options WITNESS # Enable checks to detect deadlocks and cycles
< ###options WITNESS_SKIPSPIN # Don't run witness on spinlocks for speed
<
< ### Tillman added 26Feb07 as per http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/serialconsole-setup.html
< options BREAK_TO_DEBUGGER
---
> options INVARIANTS # Enable calls of extra sanity checking
> options INVARIANT_SUPPORT # Extra sanity checks of internal structures, required by INVARIANTS
> options WITNESS # Enable checks to detect deadlocks and cycles
> options WITNESS_SKIPSPIN # Don't run witness on spinlocks for speed
[root at athena ~]# dmesg
Copyright (c) 1992-2007 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 7.0-CURRENT #0: Sun Mar 4 21:08:19 CST 2007
toor at athena.seekingfire.prv:/usr/obj/usr/src/sys/ATHENA
ACPI APIC Table: <VIA694 AWRDACPI>
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: Intel Pentium III (997.17-MHz 686-class CPU)
Origin = "GenuineIntel" Id = 0x68a Stepping = 10
Features=0x387fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,PN,MMX,FXSR,SSE>
real memory = 1073676288 (1023 MB)
avail memory = 1041326080 (993 MB)
FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs
cpu0 (BSP): APIC ID: 0
cpu1 (AP): APIC ID: 1
ioapic0 <Version 1.1> irqs 0-23 on motherboard
kbd1 at kbdmux0
ath_hal: 0.9.20.3 (AR5210, AR5211, AR5212, RF5111, RF5112, RF2413, RF5413)
acpi0: <VIA694 AWRDACPI> on motherboard
acpi0: [ITHREAD]
acpi0: Power Button (fixed)
Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x4008-0x400b on acpi0
cpu0: <ACPI CPU> on acpi0
cpu1: <ACPI CPU> on acpi0
acpi_button0: <Power Button> on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff,0x4000-0x407f,0x4080-0x40ff,0x5000-0x500f,0x6000-0x607f on acpi0
pci0: <ACPI PCI bus> on pcib0
agp0: <VIA 82C691 (Apollo Pro) host to PCI bridge> on hostb0
pcib1: <PCI-PCI bridge> at device 1.0 on pci0
pci1: <PCI bus> on pcib1
vgapci0: <VGA-compatible display> port 0xd000-0xd0ff mem 0xf4000000-0xf4ffffff,0xf6241000-0xf6241fff irq 19 at device 6.0 on pci0
isab0: <PCI-ISA bridge> at device 7.0 on pci0
isa0: <ISA bus> on isab0
atapci0: <VIA 82C686B UDMA100 controller> port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xd400-0xd40f at device 7.1 on pci0
ata0: <ATA channel 0> on atapci0
ata0: [ITHREAD]
ata1: <ATA channel 1> on atapci0
ata1: [ITHREAD]
uhci0: <VIA 83C572 USB controller> port 0xd800-0xd81f irq 12 at device 7.2 on pci0
uhci0: [GIANT-LOCKED]
uhci0: [ITHREAD]
usb0: <VIA 83C572 USB controller> on uhci0
usb0: USB revision 1.0
uhub0: <VIA UHCI root hub, class 9/0, rev 1.00/1.00, addr 1> on usb0
uhub0: 2 ports with 2 removable, self powered
uhci1: <VIA 83C572 USB controller> port 0xdc00-0xdc1f irq 12 at device 7.3 on pci0
uhci1: [GIANT-LOCKED]
uhci1: [ITHREAD]
usb1: <VIA 83C572 USB controller> on uhci1
usb1: USB revision 1.0
uhub1: <VIA UHCI root hub, class 9/0, rev 1.00/1.00, addr 1> on usb1
uhub1: 2 ports with 2 removable, self powered
pci0: <bridge> at device 7.4 (no driver attached)
fxp0: <Intel 82559 Pro/100 Ethernet> port 0xe000-0xe03f mem 0xf6240000-0xf6240fff,0xf6000000-0xf60fffff irq 17 at device 13.0 on pci0
miibus0: <MII bus> on fxp0
inphy0: <i82555 10/100 media interface> PHY 1 on miibus0
inphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
fxp0: Ethernet address: 00:e0:81:21:ad:e0
fxp0: [ITHREAD]
fxp1: <Intel 82559 Pro/100 Ethernet> port 0xe400-0xe43f mem 0xf6242000-0xf6242fff,0xf6100000-0xf61fffff irq 18 at device 14.0 on pci0
miibus1: <MII bus> on fxp1
inphy1: <i82555 10/100 media interface> PHY 1 on miibus1
inphy1: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
fxp1: Ethernet address: 00:e0:81:21:ad:e1
fxp1: [ITHREAD]
em0: <Intel(R) PRO/1000 Network Connection Version - 6.2.9> port 0xe800-0xe83f mem 0xf6200000-0xf621ffff,0xf6220000-0xf623ffff irq 18 at device 16.0 on pci0
em0: Ethernet address: 00:0e:0c:c2:ce:4f
em0: [FILTER]
sio0: <16550A-compatible COM port> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
sio0: type 16550A, console
sio0: [FILTER]
sio1: <16550A-compatible COM port> port 0x2f8-0x2ff irq 3 on acpi0
sio1: type 16550A
sio1: [FILTER]
pmtimer0 on isa0
orm0: <ISA Option ROM> at iomem 0xc0000-0xc7fff pnpid ORM0000 on isa0
atkbdc0: <Keyboard controller (i8042)> at port 0x60,0x64 on isa0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
atkbd0: [ITHREAD]
ppc0: <Parallel port> at port 0x378-0x37f irq 7 on isa0
ppc0: Generic chipset (EPP/NIBBLE) in COMPATIBLE mode
ppbus0: <Parallel port bus> on ppc0
plip0: <PLIP network interface> on ppbus0
lpt0: <Printer> on ppbus0
lpt0: Interrupt-driven port
ppi0: <Parallel I/O> on ppbus0
ppc0: [GIANT-LOCKED]
ppc0: [ITHREAD]
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
Timecounters tick every 1.000 msec
ad0: 38166MB <Seagate ST340016A 3.75> at ata0-master UDMA100
acd0: CDROM <CDU5211/YYS7> at ata1-master UDMA33
SMP: AP CPU #1 Launched!
Trying to mount root from ufs:/dev/ad0s1a
WARNING: / was not properly dismounted
WARNING: /tmp was not properly dismounted
WARNING: /usr was not properly dismounted
WARNING: /var was not properly dismounted
/var: mount pending error: blocks 56 files 3
USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME COMMAND
root 11 97.6 0.0 0 8 ?? RL 6:39AM 13:14.72 [idle: cpu0]
-T
--
"To be nobody but yourself in a world which is doing its best to make
you everybody else, means to fight the hardest human battle ever and to
never stop fighting."
-- e.e. cummings
More information about the freebsd-current
mailing list