deadlock or bad disk ? RELENG_8
Mike Tancsa
mike at sentex.net
Sun Jul 18 21:08:11 UTC 2010
On the serial console I see
swap_pager: indefinite wait buffer: bufobj: 0, blkno: 74, size: 4096
swap_pager: indefinite wait buffer: bufobj: 0, blkno: 128, size: 20480
swap_pager: indefinite wait buffer: bufobj: 0, blkno: 69, size: 4096
swap_pager: indefinite wait buffer: bufobj: 0, blkno: 6, size: 4096
swap_pager: indefinite wait buffer: bufobj: 0, blkno: 74, size: 4096
swap_pager: indefinite wait buffer: bufobj: 0, blkno: 128, size: 20480
swap_pager: indefinite wait buffer: bufobj: 0, blkno: 69, size: 4096
swap_pager: indefinite wait buffer: bufobj: 0, blkno: 6, size: 4096
swap_pager: indefinite wait buffer: bufobj: 0, blkno: 74, size: 4096
swap_pager: indefinite wait buffer: bufobj: 0, blkno: 128, size: 20480
swap_pager: indefinite wait buffer: bufobj: 0, blkno: 69, size: 4096
swap_pager: indefinite wait buffer: bufobj: 0, blkno: 6, size: 4096
swap_pager: indefinite wait buffer: bufobj: 0, blkno: 74, size: 4096
swap_pager: indefinite wait buffer: bufobj: 0, blkno: 128, size: 20480
and on a session I had open from before
# killall -9 watchdogd
just hangs, I guess because its having trouble reading from the disk.
If I hit CTRL+t, I see
load: 0.00 cmd: csh 73167 [vnread] 22.32r 0.00u 0.00s 0% 3232k
load: 0.00 cmd: csh 73167 [vnread] 22.65r 0.00u 0.00s 0% 3232k
load: 0.00 cmd: csh 73167 [vnread] 22.96r 0.00u 0.00s 0% 3232k
load: 0.00 cmd: csh 73167 [vnread] 23.20r 0.00u 0.00s 0% 3232k
load: 0.00 cmd: csh 73167 [vnread] 23.40r 0.00u 0.00s 0% 3232k
load: 0.00 cmd: csh 73167 [vnread] 23.61r 0.00u 0.00s 0% 3232k
Its RELENG_8 amd64 from July 13th and the swap is on an ARECA drive
and I dont see any errors on any of the raidset members. I also have
a large zfs spool and a small mount point on a 3ware controller but
unfortunately, nothing in the logs post reboot and nothing from smartctl
cat /var/run/dmesg.boot
Copyright (c) 1992-2010 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 8.1-PRERELEASE #0: Tue Jul 13 09:55:48 EDT 2010
mdtancsa at backup3.sentex.ca:/usr/obj/usr/src/sys/backup amd64
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: Intel(R) Core(TM)2 Quad CPU Q6600 @ 2.40GHz (2400.10-MHz K8-class CPU)
Origin = "GenuineIntel" Id = 0x6fb Family = 6 Model = f Stepping = 11
Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
Features2=0xe3bd<SSE3,DTES64,MON,DS_CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM>
AMD Features=0x20100800<SYSCALL,NX,LM>
AMD Features2=0x1<LAHF>
TSC: P-state invariant
real memory = 8589934592 (8192 MB)
avail memory = 8267673600 (7884 MB)
ACPI APIC Table: <A_M_I_ OEMAPIC >
FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
FreeBSD/SMP: 1 package(s) x 4 core(s)
cpu0 (BSP): APIC ID: 0
cpu1 (AP): APIC ID: 1
cpu2 (AP): APIC ID: 2
cpu3 (AP): APIC ID: 3
ioapic0 <Version 2.0> irqs 0-23 on motherboard
kbd1 at kbdmux0
acpi0: <A_M_I_ OEMXSDT> on motherboard
acpi0: [ITHREAD]
acpi0: Power Button (fixed)
acpi0: reservation of fed08000, 1000 (3) failed
acpi0: reservation of fed1c000, 4000 (3) failed
acpi0: reservation of fed20000, 20000 (3) failed
acpi0: reservation of fed50000, 40000 (3) failed
acpi0: reservation of ffc00000, 300000 (3) failed
acpi0: reservation of fec00000, 1000 (3) failed
acpi0: reservation of fee00000, 1000 (3) failed
acpi0: reservation of e0000000, 10000000 (3) failed
acpi0: reservation of 0, a0000 (3) failed
acpi0: reservation of 100000, dff00000 (3) failed
Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x808-0x80b on acpi0
cpu0: <ACPI CPU> on acpi0
ACPI Warning: Incorrect checksum in table [OEMB] - 0xD1, should be
0xD0 (20100331/tbutils-354)
cpu1: <ACPI CPU> on acpi0
cpu2: <ACPI CPU> on acpi0
cpu3: <ACPI CPU> on acpi0
acpi_hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0
Timecounter "HPET" frequency 14318180 Hz quality 900
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
pcib1: <ACPI PCI-PCI bridge> irq 16 at device 1.0 on pci0
pci1: <ACPI PCI bus> on pcib1
pcib2: <PCI-PCI bridge> at device 0.0 on pci1
pci3: <PCI bus> on pcib2
arcmsr0: <Areca SATA Host Adapter RAID Controller
> mem 0xfc9ff000-0xfc9fffff irq 18 at device 14.0 on pci3
ARECA RAID ADAPTER0: Driver Version 1.20.00.16 2009-10-10
ARECA RAID ADAPTER0: FIRMWARE VERSION V1.44 2008-2-1
arcmsr0: [ITHREAD]
pcib3: <PCI-PCI bridge> at device 0.2 on pci1
pci2: <PCI bus> on pcib3
uhci0: <Intel 82801JI (ICH10) USB controller USB-D> port
0x7800-0x781f irq 16 at device 26.0 on pci0
uhci0: [ITHREAD]
usbus0: <Intel 82801JI (ICH10) USB controller USB-D> on uhci0
uhci1: <Intel 82801JI (ICH10) USB controller USB-E> port
0x7880-0x789f irq 21 at device 26.1 on pci0
uhci1: [ITHREAD]
usbus1: <Intel 82801JI (ICH10) USB controller USB-E> on uhci1
uhci2: <Intel 82801JI (ICH10) USB controller USB-F> port
0x7c00-0x7c1f irq 18 at device 26.2 on pci0
uhci2: [ITHREAD]
usbus2: <Intel 82801JI (ICH10) USB controller USB-F> on uhci2
ehci0: <Intel 82801JI (ICH10) USB 2.0 controller USB-B> mem
0xfc8ffc00-0xfc8fffff irq 18 at device 26.7 on pci0
ehci0: [ITHREAD]
usbus3: EHCI version 1.0
usbus3: <Intel 82801JI (ICH10) USB 2.0 controller USB-B> on ehci0
pci0: <multimedia, HDA> at device 27.0 (no driver attached)
pcib4: <ACPI PCI-PCI bridge> irq 17 at device 28.0 on pci0
pci9: <ACPI PCI bus> on pcib4
em0: <Intel(R) PRO/1000 Network Connection 7.0.5> port 0xdc00-0xdc1f
mem 0xfcfe0000-0xfcffffff,0xfcf00000-0xfcf7ffff,0xfcfdc000-0xfcfdffff
irq 16 at device 0.0 on pci9
em0: Using MSI interrupt
em0: [FILTER]
em0: Ethernet address: 00:1b:21:3f:62:72
pcib5: <ACPI PCI-PCI bridge> irq 16 at device 28.1 on pci0
pci8: <ACPI PCI bus> on pcib5
siis0: <SiI3132 SATA controller> port 0xcc00-0xcc7f mem
0xfceffc00-0xfceffc7f,0xfcef8000-0xfcefbfff irq 17 at device 0.0 on pci8
siis0: [ITHREAD]
siisch0: <SIIS channel> at channel 0 on siis0
siisch0: [ITHREAD]
siisch1: <SIIS channel> at channel 1 on siis0
siisch1: [ITHREAD]
pcib6: <ACPI PCI-PCI bridge> irq 18 at device 28.2 on pci0
pci7: <ACPI PCI bus> on pcib6
3ware device driver for 9000 series storage controllers, version: 3.80.06.002
twa0: <3ware 9000 series Storage Controller> port 0xb800-0xb8ff mem
0xfa000000-0xfbffffff,0xfcdff000-0xfcdfffff irq 18 at device 0.0 on pci7
twa0: [ITHREAD]
twa0: WARNING: (0x04: 0x0008): Unclean shutdown detected: unit=0
twa0: INFO: (0x15: 0x1300): Controller details:: Model 9650SE-2LP, 2
ports, Firmware FE9X 3.08.00.016, BIOS BE9X 3.08.00.004
pcib7: <ACPI PCI-PCI bridge> irq 19 at device 28.3 on pci0
pci6: <ACPI PCI bus> on pcib7
fwohci0: <1394 Open Host Controller Interface> port 0xa800-0xa8ff mem
0xfccff800-0xfccfffff irq 19 at device 0.0 on pci6
fwohci0: [ITHREAD]
fwohci0: OHCI version 1.10 (ROM=1)
fwohci0: No. of Isochronous channels is 4.
fwohci0: EUI64 00:1e:8c:00:00:c4:10:80
fwohci0: Phy 1394a available S400, 2 ports.
fwohci0: Link S400, max_rec 2048 bytes.
firewire0: <IEEE1394(FireWire) bus> on fwohci0
dcons_crom0: <dcons configuration ROM> on firewire0
dcons_crom0: bus_addr 0x8eacc0
fwe0: <Ethernet over FireWire> on firewire0
if_fwe0: Fake Ethernet address: 02:1e:8c:c4:10:80
fwe0: Ethernet address: 02:1e:8c:c4:10:80
fwip0: <IP over FireWire> on firewire0
fwip0: Firewire address: 00:1e:8c:00:00:c4:10:80 @ 0xfffe00000000,
S400, maxrec 2048
fwohci0: Initiate bus reset
fwohci0: fwohci_intr_core: BUS reset
fwohci0: fwohci_intr_core: node_id=0x00000000, SelfID Count=1, CYCLEMASTER mode
pcib8: <ACPI PCI-PCI bridge> irq 17 at device 28.4 on pci0
pci5: <ACPI PCI bus> on pcib8
ahci0: <JMicron JMB361 AHCI SATA controller> mem
0xfcbfa000-0xfcbfbfff irq 16 at device 0.0 on pci5
ahci0: [ITHREAD]
ahci0: AHCI v1.00 with 2 3Gbps ports, Port Multiplier supported
ahcich0: <AHCI channel> at channel 0 on ahci0
ahcich0: [ITHREAD]
ahcich1: <AHCI channel> at channel 1 on ahci0
ahcich1: [ITHREAD]
atapci0: <JMicron JMB361 UDMA133 controller> port
0x9c00-0x9c07,0x9880-0x9883,0x9800-0x9807,0x9480-0x9483,0x9400-0x940f
irq 17 at device 0.1 on pci5
atapci0: [ITHREAD]
ata2: <ATA channel 0> on atapci0
ata2: [ITHREAD]
pcib9: <ACPI PCI-PCI bridge> irq 16 at device 28.5 on pci0
pci4: <ACPI PCI bus> on pcib9
ale0: <Atheros AR8121/AR8113/AR8114 PCIe Ethernet> port 0x8c00-0x8c7f
mem 0xfcac0000-0xfcafffff irq 17 at device 0.0 on pci4
ale0: 960 Tx FIFO, 1024 Rx FIFO
ale0: Using 1 MSI messages.
miibus0: <MII bus> on ale0
atphy0: <Atheros F1 10/100/1000 PHY> PHY 0 on miibus0
atphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT-FDX, auto
ale0: Ethernet address: e0:cb:4e:42:4b:37
ale0: [FILTER]
uhci3: <Intel 82801JI (ICH10) USB controller USB-A> port
0x7080-0x709f irq 23 at device 29.0 on pci0
uhci3: [ITHREAD]
usbus4: <Intel 82801JI (ICH10) USB controller USB-A> on uhci3
uhci4: <Intel 82801JI (ICH10) USB controller USB-B> port
0x7400-0x741f irq 19 at device 29.1 on pci0
uhci4: [ITHREAD]
usbus5: <Intel 82801JI (ICH10) USB controller USB-B> on uhci4
uhci5: <Intel 82801JI (ICH10) USB controller USB-C> port
0x7480-0x749f irq 18 at device 29.2 on pci0
uhci5: [ITHREAD]
usbus6: <Intel 82801JI (ICH10) USB controller USB-C> on uhci5
ehci1: <Intel 82801JI (ICH10) USB 2.0 controller USB-A> mem
0xfc8ff800-0xfc8ffbff irq 23 at device 29.7 on pci0
ehci1: [ITHREAD]
usbus7: EHCI version 1.0
usbus7: <Intel 82801JI (ICH10) USB 2.0 controller USB-A> on ehci1
pcib10: <ACPI PCI-PCI bridge> at device 30.0 on pci0
pci10: <ACPI PCI bus> on pcib10
vgapci0: <VGA-compatible display> port 0xe000-0xe0ff mem
0xfd000000-0xfdffffff,0xfebff000-0xfebfffff irq 16 at device 0.0 on pci10
isab0: <PCI-ISA bridge> at device 31.0 on pci0
isa0: <ISA bus> on isab0
ahci1: <Intel ICH10 AHCI SATA controller> port
0x6c00-0x6c07,0x6880-0x6883,0x6800-0x6807,0x6480-0x6483,0x6400-0x641f
mem 0xfc8fe800-0xfc8fefff irq 19 at device 31.2 on pci0
ahci1: [ITHREAD]
ahci1: AHCI v1.20 with 6 3Gbps ports, Port Multiplier supported
ahcich2: <AHCI channel> at channel 0 on ahci1
ahcich2: [ITHREAD]
ahcich3: <AHCI channel> at channel 1 on ahci1
ahcich3: [ITHREAD]
ahcich4: <AHCI channel> at channel 2 on ahci1
ahcich4: [ITHREAD]
ahcich5: <AHCI channel> at channel 3 on ahci1
ahcich5: [ITHREAD]
ahcich6: <AHCI channel> at channel 4 on ahci1
ahcich6: [ITHREAD]
ahcich7: <AHCI channel> at channel 5 on ahci1
ahcich7: [ITHREAD]
pci0: <serial bus, SMBus> at device 31.3 (no driver attached)
acpi_button0: <Power Button> on acpi0
atrtc0: <AT realtime clock> port 0x70-0x71 irq 8 on acpi0
uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
uart0: [FILTER]
uart0: console (9600,n,8,1)
atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
atkbd0: [ITHREAD]
orm0: <ISA Option ROMs> at iomem
0xc0000-0xc97ff,0xc9800-0xca7ff,0xca800-0xcc7ff,0xd4800-0xd77ff,0xd7800-0xd87ff
on isa0
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
est0: <Enhanced SpeedStep Frequency Control> on cpu0
p4tcc0: <CPU Frequency Thermal Control> on cpu0
est1: <Enhanced SpeedStep Frequency Control> on cpu1
p4tcc1: <CPU Frequency Thermal Control> on cpu1
est2: <Enhanced SpeedStep Frequency Control> on cpu2
p4tcc2: <CPU Frequency Thermal Control> on cpu2
est3: <Enhanced SpeedStep Frequency Control> on cpu3
p4tcc3: <CPU Frequency Thermal Control> on cpu3
Timecounters tick every 1.000 msec
firewire0: 1 nodes, maxhop <= 0 cable IRM irm(0) (me)
firewire0: bus manager 0
usbus0: 12Mbps Full Speed USB v1.0
usbus1: 12Mbps Full Speed USB v1.0
usbus2: 12Mbps Full Speed USB v1.0
usbus3: 480Mbps High Speed USB v2.0
usbus4: 12Mbps Full Speed USB v1.0
usbus5: 12Mbps Full Speed USB v1.0
usbus6: 12Mbps Full Speed USB v1.0
usbus7: 480Mbps High Speed USB v2.0
ugen0.1: <Intel> at usbus0
uhub0: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus0
ugen1.1: <Intel> at usbus1
uhub1: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus1
ugen2.1: <Intel> at usbus2
uhub2: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus2
ugen3.1: <Intel> at usbus3
uhub3: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus3
ugen4.1: <Intel> at usbus4
uhub4: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus4
ugen5.1: <Intel> at usbus5
uhub5: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus5
ugen6.1: <Intel> at usbus6
uhub6: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus6
ugen7.1: <Intel> at usbus7
uhub7: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus7
uhub0: 2 ports with 2 removable, self powered
uhub1: 2 ports with 2 removable, self powered
uhub2: 2 ports with 2 removable, self powered
uhub4: 2 ports with 2 removable, self powered
uhub5: 2 ports with 2 removable, self powered
uhub6: 2 ports with 2 removable, self powered
(probe16:arcmsr0:0:16:0): inquiry data fails comparison at DV1 step
da0 at arcmsr0 bus 0 scbus0 target 0 lun 0
da0: <Areca usrvar R001> Fixed Direct Access SCSI-5 device
da0: 166.666MB/s transfers (83.333MHz, offset 32, 16bit)
da0: Command Queueing enabled
da0: 76293MB (156249600 512 byte sectors: 255H 63S/T 9726C)
da1 at arcmsr0 bus 0 scbus0 target 0 lun 1
da1: <Areca backup1 R001> Fixed Direct Access SCSI-5 device
da1: 166.666MB/s transfers (83.333MHz, offset 32, 16bit)
da1: Command Queueing enabled
da1: 2784728MB (5703123456 512 byte sectors: 255H 63S/T 355003C)
ada0 at ahcich2 bus 0 scbus6 target 0 lun 0da2 at twa0 bus 0 scbus3
target 0 lun 0
da2: <AMCC 9650SE-2LP DISK 3.08> Fixed Direct Access SCSI-5 device
da2: 100.000MB/s transfers
da2: 66747MB (136697856 512 byte sectors: 255H 63S/T 8509C)
ada0: <ST31000340AS SD1A> ATA-8 SATA 2.x device
ada0: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada0: Command Queueing enabled
ada0: 953869MB (1953525168 512 byte sectors: 16H 63S/T 16383C)
ada1 at ahcich3 bus 0 scbus7 target 0 lun 0
ada1: <ST31000340AS SD15> ATA-8 SATA 2.x device
ada1: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada1: Command Queueing enabled
ada1: 953869MB (1953525168 512 byte sectors: 16H 63S/T 16383C)
ada2 at ahcich4 bus 0 scbus8 target 0 lun 0
ada2: <ST31000333AS SD35> ATA-8 SATA 2.x device
ada2: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada2: Command Queueing enabled
ada2: 953869MB (1953525168 512 byte sectors: 16H 63S/T 16383C)
ada3 at ahcich5 bus 0 scbus9 target 0 lun 0
ada3: <ST31000528AS CC35> ATA-8 SATA 2.x device
ada3: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada3: Command Queueing enabled
ada3: 953869MB (1953525168 512 byte sectors: 16H 63S/T 16383C)
pass2 at arcmsr0 bus 0 scbus0 target 16 lun 0
pass2: <Areca RAID controller R001> Fixed Processor SCSI-0 device
SMP: AP CPU #1 Launched!
SMP: AP CPU #2 Launched!
SMP: AP CPU #3 Launched!
uhub3: 6 ports with 6 removable, self powered
uhub7: 6 ports with 6 removable, self powered
Root mount waiting for: usbus7
Trying to mount root from ufs:/dev/da2s1a
WARNING: / was not properly dismounted
ZFS filesystem version 3
ZFS storage pool version 14
ugen5.2: <American Power Conversion> at usbus5
twa0: INFO: (0x04: 0x000C): Initialize started: unit=0
em0: link state changed to UP
ale0: link state changed to UP
---Mike
--------------------------------------------------------------------
Mike Tancsa, tel +1 519 651 3400
Sentex Communications, mike at sentex.net
Providing Internet since 1994 www.sentex.net
Cambridge, Ontario Canada www.sentex.net/mike
More information about the freebsd-stable
mailing list