FreeBSD 7-STABLE, isp(4), QLE2462: panic & deadlocks

Panagiotis Christias p.christias at noc.ntua.gr
Tue Oct 14 15:23:48 PDT 2008


Hello everybody,

we recently got three Dell PowerEdge servers equipped with Qlogic
QLE2462 cards (dual FC 4Gbps ports) and an EMC CLARiiON CX3-40 SAN.
The servers with the FC cards were successfully and extensively
tested with an older SUN StorEdge T3 SAN.

However, when we connect them to the CX3-40, create and mount a new
partition and then do something as simple as "tar -C /san -xf ports.tgz"
the system panics and deadlocks. We have tried several FreeBSD versions
(6.3 i386/adm64, 7.0 i386/adm64, 7.1 i386/adm64 and lastly 7-STABLE i386
- we also tried the latest 8-CURRENT snapshot but it panicked too soon).
The result is always the same; panic and deadlock.

In case it helps, we have also tried all available connection speeds
(4, 2 and 1Gbps), the QLE2462 cards have installed the latest firwmare
version (4.03.10) and we have verified our hardware setup by testing
it with Linux (CentOS 5.2), although we would try anything in order
to keep the whole of our server base to FreeBSD (we have succeeded
till now).

Currently, we have installed the latest version of 7-STABLE (csuped
today) along with a debug-enabled kernel and collected as much output
as we can. See below.

We are open to any suggestion and any kind of tests in order to help
resolving the problem.

Thanks!


COLLECTED DATA:

1. uname -a:

FreeBSD 7x86.noc.ntua.gr 7.1-PRERELEASE FreeBSD 7.1-PRERELEASE #0: Tue Oct 14 21:12:57 EEST 2008     root at 7x86.noc.ntua.gr:/usr/obj/usr/src/sys/GENERIC-DEBUG  i386

---------------------------------------------------------------

2. Kernel configuration file /sys/i386/conf/GENERIC-DEBUG:

include  GENERIC
ident    GENERIC-DEBUG

# force isp firmware to fully loaded
# device ispfw

options KDB
options DDB

options INVARIANTS
options INVARIANT_SUPPORT
options WITNESS
options DEBUG_LOCKS
options DEBUG_VFS_LOCKS
options DIAGNOSTIC

options CAMDEBUG
options CAM_DEBUG_BUS=-1
options CAM_DEBUG_TARGET=-1
options CAM_DEBUG_LUN=-1
options CAM_DEBUG_FLAGS=CAM_DEBUG_INFO
 
---------------------------------------------------------------

3. dmesg output, also available at http://noc.ntua.gr/~christia/san/7-STABLE.dmesg.out :

Copyright (c) 1992-2008 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
        The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 7.1-PRERELEASE #0: Tue Oct 14 21:12:57 EEST 2008
    root at 7x86.noc.ntua.gr:/usr/obj/usr/src/sys/GENERIC-DEBUG
WARNING: WITNESS option enabled, expect reduced performance.
WARNING: DIAGNOSTIC option enabled, expect reduced performance.
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: Intel(R) Xeon(R) CPU           E5405  @ 2.00GHz (1997.03-MHz 686-class CPU)
  Origin = "GenuineIntel"  Id = 0x10676  Stepping = 6
  Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
  Features2=0xce33d<SSE3,RSVD2,MON,DS_CPL,VMX,TM2,SSSE3,CX16,xTPR,PDCM,DCA,<b19>>
  AMD Features=0x20100000<NX,LM>
  AMD Features2=0x1<LAHF>
  Cores per package: 4
real memory  = 3484745728 (3323 MB)
avail memory = 3405070336 (3247 MB)
ACPI APIC Table: <DELL   PE_SC3  >
FreeBSD/SMP: Multiprocessor System Detected: 8 CPUs
 cpu0 (BSP): APIC ID:  0
 cpu1 (AP): APIC ID:  1
 cpu2 (AP): APIC ID:  2
 cpu3 (AP): APIC ID:  3
 cpu4 (AP): APIC ID:  4
 cpu5 (AP): APIC ID:  5
 cpu6 (AP): APIC ID:  6
 cpu7 (AP): APIC ID:  7
WITNESS: spin lock cpuset not in order list
WITNESS: spin lock intrcnt not in order list
ioapic0: Changing APIC ID to 8
ioapic1: Changing APIC ID to 9
ioapic2: Changing APIC ID to 10
ioapic0 <Version 2.0> irqs 0-23 on motherboard
ioapic1 <Version 2.0> irqs 160-183 on motherboard
ioapic2 <Version 2.0> irqs 224-247 on motherboard
kbd1 at kbdmux0
ath_hal: 0.9.20.3 (AR5210, AR5211, AR5212, RF5111, RF5112, RF2413, RF5413)
acpi0: <DELL PE_SC3> on motherboard
acpi0: [ITHREAD]
acpi0: Power Button (fixed)
Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x808-0x80b on acpi0
acpi_hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0
Timecounter "HPET" frequency 14318180 Hz quality 900
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
pcib1: <ACPI PCI-PCI bridge> at device 2.0 on pci0
pci4: <ACPI PCI bus> on pcib1
pcib2: <ACPI PCI-PCI bridge> at device 0.0 on pci4
pci5: <ACPI PCI bus> on pcib2
pcib3: <ACPI PCI-PCI bridge> at device 0.0 on pci5
pci6: <ACPI PCI bus> on pcib3
pcib4: <PCI-PCI bridge> at device 0.0 on pci6
pci7: <PCI bus> on pcib4
bce0: <Broadcom NetXtreme II BCM5708 1000Base-T (B2)> mem 0xf4000000-0xf5ffffff irq 16 at device 0.0 on pci7
miibus0: <MII bus> on bce0
brgphy0: <BCM5708C 10/100/1000baseTX PHY> PHY 1 on miibus0
brgphy0:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-FDX, auto
bce0: Ethernet address: 00:1e:c9:e8:13:e9
bce0: [ITHREAD]
bce0: ASIC (0x57081020); Rev (B2); Bus (PCI-X, 64-bit, 133MHz); F/W (0x04000305); Flags( MFW MSI )
pcib5: <ACPI PCI-PCI bridge> at device 1.0 on pci5
pci8: <ACPI PCI bus> on pcib5
pcib6: <PCI-PCI bridge> at device 0.3 on pci4
pci9: <PCI bus> on pcib6
pcib7: <ACPI PCI-PCI bridge> at device 3.0 on pci0
pci1: <ACPI PCI bus> on pcib7
mfi0: <Dell PERC 6> port 0xec00-0xecff mem 0xfc780000-0xfc7bffff,0xfc740000-0xfc77ffff irq 16 at device 0.0 on pci1
mfi0: Megaraid SAS driver Ver 2.00 
mfi0: 3299 (277344669s/0x0020/info) - Shutdown command received from host
mfi0: 3300 (boot + 3s/0x0020/info) - Firmware initialization started (PCI ID 0060/1000/1f0c/1028)
mfi0: 3301 (boot + 3s/0x0020/info) - Firmware version 1.11.82-0473
mfi0: 3302 (boot + 3s/0x0008/info) - Battery Present
mfi0: 3303 (boot + 3s/0x0020/info) - Package version 6.0.3-0002
mfi0: 3304 (boot + 23s/0x0004/info) - Enclosure (SES) discovered on PD 20(c None/p0)
mfi0: 3305 (boot + 23s/0x0002/info) - Unexpected sense: Encl PD 20, CDB: 12 00 00 00 00 00, Sense: 70 00 06 00 00 00 00 0a 00 00 00 00 29 00 00 00 00 00
mfi0: 3306 (boot + 24s/0x0002/info) - Inserted: Encl PD 20
mfi0: 3307 (boot + 24s/0x0002/info) - Inserted: PD 20(c None/p0) Info: enclPd=20, scsiType=d, portMap=09, sasAddr=5001e090d62bfb00,0000000000000000
mfi0: 3308 (boot + 24s/0x0002/info) - Inserted: PD 00(e0x20/s0)
mfi0: 3309 (boot + 24s/0x0002/info) - Inserted: PD 00(e0x20/s0) Info: enclPd=20, scsiType=0, portMap=00, sasAddr=5000cca005b1bf89,0000000000000000
mfi0: 3310 (boot + 24s/0x0002/info) - Inserted: PD 01(e0x20/s1)
mfi0: 3311 (boot + 24s/0x0002/info) - Inserted: PD 01(e0x20/s1) Info: enclPd=20, scsiType=0, portMap=01, sasAddr=5000cca005b1427d,0000000000000000
mfi0: 3312 (boot + 24s/0x0002/info) - Inserted: PD 02(e0x20/s2)
mfi0: 3313 (boot + 24s/0x0002/info) - Inserted: PD 02(e0x20/s2) Info: enclPd=20, scsiType=0, portMap=02, sasAddr=5000cca005b13c61,0000000000000000
mfi0: 3314 (boot + 24s/0x0042/info) - Global Hot Spare created on PD 01(e0x20/s1) (global)
mfi0: 3315 (boot + 24s/0x0002/info) - State change on PD 01(e0x20/s1) from UNCONFIGURED_GOOD(0) to HOT SPARE(2)
mfi0: 3316 (boot + 68s/0x0008/info) - Battery temperature is normal
mfi0: 3317 (boot + 68s/0x0008/info) - Battery started charging
mfi0: 3318 (boot + 68s/0x0008/info) - Current capacity of the battery is above threshold
mfi0: [ITHREAD]
pcib8: <ACPI PCI-PCI bridge> at device 4.0 on pci0
pci10: <ACPI PCI bus> on pcib8
isp0: <Qlogic ISP 2432 PCI FC-AL Adapter> port 0xdc00-0xdcff mem 0xfc5bc000-0xfc5bffff irq 16 at device 0.0 on pci10
firmware_get: failed to load firmware image isp_2400
isp0: [ITHREAD]
isp0: Board Type 2422, Chip Revision 0x3, resident F/W Revision 4.3.1
isp1: <Qlogic ISP 2432 PCI FC-AL Adapter> port 0xd800-0xd8ff mem 0xfc5b8000-0xfc5bbfff irq 17 at device 0.1 on pci10
firmware_get: failed to load firmware image isp_2400
isp1: [ITHREAD]
isp1: Board Type 2422, Chip Revision 0x3, resident F/W Revision 4.3.1
pcib9: <PCI-PCI bridge> at device 5.0 on pci0
pci11: <PCI bus> on pcib9
pcib10: <ACPI PCI-PCI bridge> at device 6.0 on pci0
pci12: <ACPI PCI bus> on pcib10
pcib11: <ACPI PCI-PCI bridge> at device 0.0 on pci12
pci13: <ACPI PCI bus> on pcib11
pcib12: <ACPI PCI-PCI bridge> at device 0.2 on pci12
pci14: <ACPI PCI bus> on pcib12
pcib13: <PCI-PCI bridge> at device 7.0 on pci0
pci15: <PCI bus> on pcib13
pcib14: <ACPI PCI-PCI bridge> at device 28.0 on pci0
pci2: <ACPI PCI bus> on pcib14
pcib15: <PCI-PCI bridge> at device 0.0 on pci2
pci3: <PCI bus> on pcib15
bce1: <Broadcom NetXtreme II BCM5708 1000Base-T (B2)> mem 0xf8000000-0xf9ffffff irq 16 at device 0.0 on pci3
miibus1: <MII bus> on bce1
brgphy1: <BCM5708C 10/100/1000baseTX PHY> PHY 1 on miibus1
brgphy1:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-FDX, auto
bce1: Ethernet address: 00:1e:c9:e8:13:e7
bce1: [ITHREAD]
bce1: ASIC (0x57081020); Rev (B2); Bus (PCI-X, 64-bit, 133MHz); F/W (0x04000305); Flags( MFW MSI )
uhci0: <Intel 631XESB/632XESB/3100 USB controller USB-1> port 0xbce0-0xbcff irq 21 at device 29.0 on pci0
uhci0: [GIANT-LOCKED]
uhci0: [ITHREAD]
usb0: <Intel 631XESB/632XESB/3100 USB controller USB-1> on uhci0
usb0: USB revision 1.0
uhub0: <Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1> on usb0
uhub0: 2 ports with 2 removable, self powered
uhci1: <Intel 631XESB/632XESB/3100 USB controller USB-2> port 0xbcc0-0xbcdf irq 20 at device 29.1 on pci0
uhci1: [GIANT-LOCKED]
uhci1: [ITHREAD]
usb1: <Intel 631XESB/632XESB/3100 USB controller USB-2> on uhci1
usb1: USB revision 1.0
uhub1: <Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1> on usb1
uhub1: 2 ports with 2 removable, self powered
uhci2: <Intel 631XESB/632XESB/3100 USB controller USB-3> port 0xbca0-0xbcbf irq 21 at device 29.2 on pci0
uhci2: [GIANT-LOCKED]
uhci2: [ITHREAD]
usb2: <Intel 631XESB/632XESB/3100 USB controller USB-3> on uhci2
usb2: USB revision 1.0
uhub2: <Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1> on usb2
uhub2: 2 ports with 2 removable, self powered
uhci3: <Intel 631XESB/632XESB/3100 USB controller USB-4> port 0xbc80-0xbc9f irq 20 at device 29.3 on pci0
uhci3: [GIANT-LOCKED]
uhci3: [ITHREAD]
usb3: <Intel 631XESB/632XESB/3100 USB controller USB-4> on uhci3
usb3: USB revision 1.0
uhub3: <Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1> on usb3
uhub3: 2 ports with 2 removable, self powered
ehci0: <Intel 63XXESB USB 2.0 controller> mem 0xfc800000-0xfc8003ff irq 21 at device 29.7 on pci0
ehci0: [GIANT-LOCKED]
ehci0: [ITHREAD]
usb4: EHCI version 1.0
usb4: companion controllers, 2 ports each: usb0 usb1 usb2 usb3
usb4: <Intel 63XXESB USB 2.0 controller> on ehci0
usb4: USB revision 2.0
uhub4: <Intel EHCI root hub, class 9/0, rev 2.00/1.00, addr 1> on usb4
uhub4: 8 ports with 8 removable, self powered
uhub5: <vendor 0x413c product 0xa001, class 9/0, rev 2.00/0.00, addr 2> on uhub4
uhub5: multiple transaction translators
uhub5: 2 ports with 2 removable, self powered
ehci_device_intr_done: not done, ex=0xc699e800
ukbd0: <Dell DRAC5, class 0/0, rev 1.10/0.00, addr 3> on uhub5
ehci_device_intr_done: not done, ex=0xc699e000
kbd2 at ukbd0
ums0: <Dell DRAC5, class 0/0, rev 1.10/0.00, addr 3> on uhub5
ums0: X report 0x0002 not supported
device_attach: ums0 attach returned 6
umass0: <DELL  INC. DRAC5 VIRTUAL  MEDIA, class 0/0, rev 2.00/0.00, addr 4> on uhub5
umass1: <DELL  INC. DRAC5 VIRTUAL  MEDIA, class 0/0, rev 2.00/0.00, addr 4> on uhub5
uhub6: <vendor 0x04b4 product 0x6560, class 9/0, rev 2.00/0.0b, addr 5> on uhub4
uhub6: multiple transaction translators
uhub6: 4 ports with 4 removable, self powered
ehci_device_intr_done: not done, ex=0xc69a9400
pcib16: <ACPI PCI-PCI bridge> at device 30.0 on pci0
pci16: <ACPI PCI bus> on pcib16
vgapci0: <VGA-compatible display> port 0xcc00-0xccff mem 0xd8000000-0xdfffffff,0xfc2d0000-0xfc2dffff irq 19 at device 13.0 on pci16
isab0: <PCI-ISA bridge> at device 31.0 on pci0
isa0: <ISA bus> on isab0
atapci0: <Intel 63XXESB2 UDMA100 controller> port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xfc00-0xfc0f at device 31.1 on pci0
ata0: <ATA channel 0> on atapci0
ata0: [ITHREAD]
ata1: <ATA channel 1> on atapci0
ata1: [ITHREAD]
fdc0: <floppy drive controller> port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on acpi0
fdc0: does not respond
device_attach: fdc0 attach returned 6
sio0: <16550A-compatible COM port> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
sio0: type 16550A, console
sio0: [FILTER]
sio1: <16550A-compatible COM port> port 0x2f8-0x2ff irq 3 on acpi0
sio1: type 16550A
sio1: [FILTER]
cpu0: <ACPI CPU> on acpi0
p4tcc0: <CPU Frequency Thermal Control> on cpu0
cpu1: <ACPI CPU> on acpi0
p4tcc1: <CPU Frequency Thermal Control> on cpu1
cpu2: <ACPI CPU> on acpi0
p4tcc2: <CPU Frequency Thermal Control> on cpu2
cpu3: <ACPI CPU> on acpi0
p4tcc3: <CPU Frequency Thermal Control> on cpu3
cpu4: <ACPI CPU> on acpi0
p4tcc4: <CPU Frequency Thermal Control> on cpu4
cpu5: <ACPI CPU> on acpi0
p4tcc5: <CPU Frequency Thermal Control> on cpu5
cpu6: <ACPI CPU> on acpi0
p4tcc6: <CPU Frequency Thermal Control> on cpu6
cpu7: <ACPI CPU> on acpi0
p4tcc7: <CPU Frequency Thermal Control> on cpu7
fdc0: <floppy drive controller> port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on acpi0
fdc0: does not respond
device_attach: fdc0 attach returned 6
pmtimer0 on isa0
orm0: <ISA Option ROMs> at iomem 0xc0000-0xc8fff,0xc9000-0xc9fff,0xd4800-0xd57ff,0xec000-0xeffff pnpid ORM0000 on isa0
atkbdc0: <Keyboard controller (i8042)> at port 0x60,0x64 on isa0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
atkbd0: [ITHREAD]
ppc0: parallel port not found.
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x100>
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
Timecounters tick every 1.000 msec
bce0: link state changed to DOWN
bce1: link state changed to DOWN
acd0: DVDROM <TEAC DVD-ROM DV28EV/D.AE> at ata0-master UDMA33
mfi0: 3319 (277344766s/0x0020/info) - Time established as 10/15/08  0:12:46; (71 seconds since power on)
Expensive timeout(9) function: 0xc06f9de0(0xc0c52520) 0.113017715 s
mfid0: <MFI Logical Disk> on mfi0
mfid0: 69376MB (142082048 sectors) RAID volume '' is optimal
(probe376:isp0:0:376:0): cam completion status 0x4a
(probe376:isp0:0:376:0): cam completion status 0x4a
(probe395:isp0:0:395:0): cam completion status 0x4a
(probe395:isp0:0:395:0): cam completion status 0x4a
(probe400:isp0:0:400:0): cam completion status 0x4a
(probe400:isp0:0:400:0): cam completion status 0x4a
(probe411:isp0:0:411:0): cam completion status 0x4a
(probe411:isp0:0:411:0): cam completion status 0x4a
(probe423:isp0:0:423:0): cam completion status 0x4a
(probe423:isp0:0:423:0): cam completion status 0x4a
(probe437:isp0:0:437:0): cam completion status 0x4a
(probe437:isp0:0:437:0): cam completion status 0x4a
(probe449:isp0:0:449:0): cam completion status 0x4a
(probe449:isp0:0:449:0): cam completion status 0x4a
(probe459:isp0:0:459:0): cam completion status 0x4a
(probe459:isp0:0:459:0): cam completion status 0x4a
(probe464:isp0:0:464:0): cam completion status 0x4a
(probe464:isp0:0:464:0): cam completion status 0x4a
(probe471:isp0:0:471:0): cam completion status 0x4a
(probe471:isp0:0:471:0): cam completion status 0x4a
(probe478:isp0:0:478:0): cam completion status 0x4a
(probe478:isp0:0:478:0): cam completion status 0x4a
(probe488:isp0:0:488:0): cam completion status 0x4a
(probe488:isp0:0:488:0): cam completion status 0x4a
(probe504:isp0:0:504:0): cam completion status 0x4a
(probe504:isp0:0:504:0): cam completion status 0x4a
(probe505:isp0:0:505:0): cam completion status 0x4a
(probe505:isp0:0:505:0): cam completion status 0x4a
(probe506:isp0:0:506:0): cam completion status 0x4a
(probe506:isp0:0:506:0): cam completion status 0x4a
(probe507:isp0:0:507:0): cam completion status 0x4a
(probe507:isp0:0:507:0): cam completion status 0x4a
(probe508:isp0:0:508:0): cam completion status 0x4a
(probe508:isp0:0:508:0): cam completion status 0x4a
(probe509:isp0:0:509:0): cam completion status 0x4a
(probe509:isp0:0:509:0): cam completion status 0x4a
(probe510:isp0:0:510:0): cam completion status 0x4a
(probe510:isp0:0:510:0): cam completion status 0x4a
(probe511:isp0:0:511:0): cam completion status 0x4a
(probe511:isp0:0:511:0): cam completion status 0x4a
(probe1:isp0:0:1:0): cam completion status 0x4a
(probe1:isp0:0:1:0): cam completion status 0x4a
(probe2:isp0:0:2:0): cam completion status 0x4a
(probe2:isp0:0:2:0): cam completion status 0x4a
(probe3:isp0:0:3:0): cam completion status 0x4a
(probe0:isp0:0:0:0): cam completion status 0xcc
(probe3:isp0:0:3:0): cam completion status 0x4a
(probe4:isp0:0:4:0): cam completion status 0x4a
(probe4:isp0:0:4:0): cam completion status 0x4a
(probe5:isp0:0:5:0): cam completion status 0x4a
(probe5:isp0:0:5:0): cam completion status 0x4a
(probe7:isp0:0:7:0): cam completion status 0x4a
(probe7:isp0:0:7:0): cam completion status 0x4a
(probe8:isp0:0:8:0): cam completion status 0x4a
(probe8:isp0:0:8:0): cam completion status 0x4a
(probe9:isp0:0:9:0): cam completion status 0x4a
(probe9:isp0:0:9:0): cam completion status 0x4a
(probe10:isp0:0:10:0): cam completion status 0x4a
(probe10:isp0:0:10:0): cam completion status 0x4a
(probe11:isp0:0:11:0): cam completion status 0x4a
(probe11:isp0:0:11:0): cam completion status 0x4a
(probe12:isp0:0:12:0): cam completion status 0x4a
(probe12:isp0:0:12:0): cam completion status 0x4a
(probe13:isp0:0:13:0): cam completion status 0x4a
(probe13:isp0:0:13:0): cam completion status 0x4a
(probe14:isp0:0:14:0): cam completion status 0x4a
(probe14:isp0:0:14:0): cam completion status 0x4a
(probe15:isp0:0:15:0): cam completion status 0x4a
(probe15:isp0:0:15:0): cam completion status 0x4a
(probe16:isp0:0:16:0): cam completion status 0x4a
(probe16:isp0:0:16:0): cam completion status 0x4a
(probe17:isp0:0:17:0): cam completion status 0x4a
(probe17:isp0:0:17:0): cam completion status 0x4a
(probe18:isp0:0:18:0): cam completion status 0x4a
(probe18:isp0:0:18:0): cam completion status 0x4a
(probe19:isp0:0:19:0): cam completion status 0x4a
(probe19:isp0:0:19:0): cam completion status 0x4a
(probe20:isp0:0:20:0): cam completion status 0x4a
(probe20:isp0:0:20:0): cam completion status 0x4a
(probe21:isp0:0:21:0): cam completion status 0x4a
(probe21:isp0:0:21:0): cam completion status 0x4a
(probe22:isp0:0:22:0): cam completion status 0x4a
(probe22:isp0:0:22:0): cam completion status 0x4a
(probe23:isp0:0:23:0): cam completion status 0x4a
(probe23:isp0:0:23:0): cam completion status 0x4a
(probe24:isp0:0:24:0): cam completion status 0x4a
(probe24:isp0:0:24:0): cam completion status 0x4a
(probe26:isp0:0:26:0): cam completion status 0x4a
(probe26:isp0:0:26:0): cam completion status 0x4a
(probe27:isp0:0:27:0): cam completion status 0x4a
(probe27:isp0:0:27:0): cam completion status 0x4a
(probe28:isp0:0:28:0): cam completion status 0x4a
(probe28:isp0:0:28:0): cam completion status 0x4a
(probe29:isp0:0:29:0): cam completion status 0x4a
(probe29:isp0:0:29:0): cam completion status 0x4a
(probe30:isp0:0:30:0): cam completion status 0x4a
(probe30:isp0:0:30:0): cam completion status 0x4a
(probe31:isp0:0:31:0): cam completion status 0x4a
(probe31:isp0:0:31:0): cam completion status 0x4a
(probe32:isp0:0:32:0): cam completion status 0x4a
(probe32:isp0:0:32:0): cam completion status 0x4a
(probe33:isp0:0:33:0): cam completion status 0x4a
(probe33:isp0:0:33:0): cam completion status 0x4a
(probe34:isp0:0:34:0): cam completion status 0x4a
(probe34:isp0:0:34:0): cam completion status 0x4a
(probe35:isp0:0:35:0): cam completion status 0x4a
(probe35:isp0:0:35:0): cam completion status 0x4a
(probe36:isp0:0:36:0): cam completion status 0x4a
(probe36:isp0:0:36:0): cam completion status 0x4a
(probe37:isp0:0:37:0): cam completion status 0x4a
(probe37:isp0:0:37:0): cam completion status 0x4a
(probe38:isp0:0:38:0): cam completion status 0x4a
(probe38:isp0:0:38:0): cam completion status 0x4a
(probe39:isp0:0:39:0): cam completion status 0x4a
(probe39:isp0:0:39:0): cam completion status 0x4a
(probe40:isp0:0:40:0): cam completion status 0x4a
(probe40:isp0:0:40:0): cam completion status 0x4a
(probe41:isp0:0:41:0): cam completion status 0x4a
(probe41:isp0:0:41:0): cam completion status 0x4a
(probe42:isp0:0:42:0): cam completion status 0x4a
(probe42:isp0:0:42:0): cam completion status 0x4a
(probe43:isp0:0:43:0): cam completion status 0x4a
(probe43:isp0:0:43:0): cam completion status 0x4a
(probe44:isp0:0:44:0): cam completion status 0x4a
(probe44:isp0:0:44:0): cam completion status 0x4a
(probe45:isp0:0:45:0): cam completion status 0x4a
(probe45:isp0:0:45:0): cam completion status 0x4a
(probe46:isp0:0:46:0): cam completion status 0x4a
(probe46:isp0:0:46:0): cam completion status 0x4a
(probe47:isp0:0:47:0): cam completion status 0x4a
(probe47:isp0:0:47:0): cam completion status 0x4a
(probe48:isp0:0:48:0): cam completion status 0x4a
(probe48:isp0:0:48:0): cam completion status 0x4a
(probe49:isp0:0:49:0): cam completion status 0x4a
(probe49:isp0:0:49:0): cam completion status 0x4a
(probe50:isp0:0:50:0): cam completion status 0x4a
(probe50:isp0:0:50:0): cam completion status 0x4a
(probe51:isp0:0:51:0): cam completion status 0x4a
(probe51:isp0:0:51:0): cam completion status 0x4a
(probe52:isp0:0:52:0): cam completion status 0x4a
(probe52:isp0:0:52:0): cam completion status 0x4a
(probe53:isp0:0:53:0): cam completion status 0x4a
(probe53:isp0:0:53:0): cam completion status 0x4a
(probe54:isp0:0:54:0): cam completion status 0x4a
(probe54:isp0:0:54:0): cam completion status 0x4a
(probe55:isp0:0:55:0): cam completion status 0x4a
(probe55:isp0:0:55:0): cam completion status 0x4a
(probe56:isp0:0:56:0): cam completion status 0x4a
(probe56:isp0:0:56:0): cam completion status 0x4a
(probe57:isp0:0:57:0): cam completion status 0x4a
(probe57:isp0:0:57:0): cam completion status 0x4a
(probe58:isp0:0:58:0): cam completion status 0x4a
(probe58:isp0:0:58:0): cam completion status 0x4a
(probe59:isp0:0:59:0): cam completion status 0x4a
(probe59:isp0:0:59:0): cam completion status 0x4a
(probe60:isp0:0:60:0): cam completion status 0x4a
(probe60:isp0:0:60:0): cam completion status 0x4a
(probe61:isp0:0:61:0): cam completion status 0x4a
(probe61:isp0:0:61:0): cam completion status 0x4a
(probe63:isp0:0:63:0): cam completion status 0x4a
(probe63:isp0:0:63:0): cam completion status 0x4a
(probe64:isp0:0:64:0): cam completion status 0x4a
(probe64:isp0:0:64:0): cam completion status 0x4a
(probe65:isp0:0:65:0): cam completion status 0x4a
(probe65:isp0:0:65:0): cam completion status 0x4a
(probe66:isp0:0:66:0): cam completion status 0x4a
(probe66:isp0:0:66:0): cam completion status 0x4a
(probe67:isp0:0:67:0): cam completion status 0x4a
(probe67:isp0:0:67:0): cam completion status 0x4a
(probe68:isp0:0:68:0): cam completion status 0x4a
(probe68:isp0:0:68:0): cam completion status 0x4a
(probe69:isp0:0:69:0): cam completion status 0x4a
(probe69:isp0:0:69:0): cam completion status 0x4a
(probe70:isp0:0:70:0): cam completion status 0x4a
(probe70:isp0:0:70:0): cam completion status 0x4a
(probe71:isp0:0:71:0): cam completion status 0x4a
(probe71:isp0:0:71:0): cam completion status 0x4a
(probe72:isp0:0:72:0): cam completion status 0x4a
(probe72:isp0:0:72:0): cam completion status 0x4a
(probe73:isp0:0:73:0): cam completion status 0x4a
(probe73:isp0:0:73:0): cam completion status 0x4a
(probe75:isp0:0:75:0): cam completion status 0x4a
(probe75:isp0:0:75:0): cam completion status 0x4a
(probe76:isp0:0:76:0): cam completion status 0x4a
(probe76:isp0:0:76:0): cam completion status 0x4a
(probe77:isp0:0:77:0): cam completion status 0x4a
(probe77:isp0:0:77:0): cam completion status 0x4a
(probe78:isp0:0:78:0): cam completion status 0x4a
(probe78:isp0:0:78:0): cam completion status 0x4a
(probe79:isp0:0:79:0): cam completion status 0x4a
(probe79:isp0:0:79:0): cam completion status 0x4a
(probe81:isp0:0:81:0): cam completion status 0x4a
(probe81:isp0:0:81:0): cam completion status 0x4a
(probe82:isp0:0:82:0): cam completion status 0x4a
(probe82:isp0:0:82:0): cam completion status 0x4a
(probe83:isp0:0:83:0): cam completion status 0x4a
(probe83:isp0:0:83:0): cam completion status 0x4a
(probe84:isp0:0:84:0): cam completion status 0x4a
(probe84:isp0:0:84:0): cam completion status 0x4a
(probe85:isp0:0:85:0): cam completion status 0x4a
(probe85:isp0:0:85:0): cam completion status 0x4a
(probe86:isp0:0:86:0): cam completion status 0x4a
(probe86:isp0:0:86:0): cam completion status 0x4a
(probe87:isp0:0:87:0): cam completion status 0x4a
(probe87:isp0:0:87:0): cam completion status 0x4a
(probe88:isp0:0:88:0): cam completion status 0x4a
(probe88:isp0:0:88:0): cam completion status 0x4a
(probe89:isp0:0:89:0): cam completion status 0x4a
(probe89:isp0:0:89:0): cam completion status 0x4a
(probe90:isp0:0:90:0): cam completion status 0x4a
(probe90:isp0:0:90:0): cam completion status 0x4a
(probe91:isp0:0:91:0): cam completion status 0x4a
(probe91:isp0:0:91:0): cam completion status 0x4a
(probe92:isp0:0:92:0): cam completion status 0x4a
(probe92:isp0:0:92:0): cam completion status 0x4a
(probe93:isp0:0:93:0): cam completion status 0x4a
(probe93:isp0:0:93:0): cam completion status 0x4a
(probe94:isp0:0:94:0): cam completion status 0x4a
(probe94:isp0:0:94:0): cam completion status 0x4a
(probe95:isp0:0:95:0): cam completion status 0x4a
(probe95:isp0:0:95:0): cam completion status 0x4a
(probe96:isp0:0:96:0): cam completion status 0x4a
(probe96:isp0:0:96:0): cam completion status 0x4a
(probe97:isp0:0:97:0): cam completion status 0x4a
(probe97:isp0:0:97:0): cam completion status 0x4a
(probe98:isp0:0:98:0): cam completion status 0x4a
(probe98:isp0:0:98:0): cam completion status 0x4a
(probe99:isp0:0:99:0): cam completion status 0x4a
(probe99:isp0:0:99:0): cam completion status 0x4a
(probe100:isp0:0:100:0): cam completion status 0x4a
(probe100:isp0:0:100:0): cam completion status 0x4a
(probe101:isp0:0:101:0): cam completion status 0x4a
(probe101:isp0:0:101:0): cam completion status 0x4a
(probe102:isp0:0:102:0): cam completion status 0x4a
(probe102:isp0:0:102:0): cam completion status 0x4a
(probe103:isp0:0:103:0): cam completion status 0x4a
(probe103:isp0:0:103:0): cam completion status 0x4a
(probe104:isp0:0:104:0): cam completion status 0x4a
(probe104:isp0:0:104:0): cam completion status 0x4a
(probe105:isp0:0:105:0): cam completion status 0x4a
(probe105:isp0:0:105:0): cam completion status 0x4a
(probe106:isp0:0:106:0): cam completion status 0x4a
(probe106:isp0:0:106:0): cam completion status 0x4a
(probe107:isp0:0:107:0): cam completion status 0x4a
(probe107:isp0:0:107:0): cam completion status 0x4a
(probe108:isp0:0:108:0): cam completion status 0x4a
(probe108:isp0:0:108:0): cam completion status 0x4a
(probe110:isp0:0:110:0): cam completion status 0x4a
(probe110:isp0:0:110:0): cam completion status 0x4a
(probe111:isp0:0:111:0): cam completion status 0x4a
(probe111:isp0:0:111:0): cam completion status 0x4a
(probe112:isp0:0:112:0): cam completion status 0x4a
(probe112:isp0:0:112:0): cam completion status 0x4a
(probe113:isp0:0:113:0): cam completion status 0x4a
(probe113:isp0:0:113:0): cam completion status 0x4a
(probe114:isp0:0:114:0): cam completion status 0x4a
(probe114:isp0:0:114:0): cam completion status 0x4a
(probe115:isp0:0:115:0): cam completion status 0x4a
(probe115:isp0:0:115:0): cam completion status 0x4a
(probe116:isp0:0:116:0): cam completion status 0x4a
(probe116:isp0:0:116:0): cam completion status 0x4a
(probe117:isp0:0:117:0): cam completion status 0x4a
(probe117:isp0:0:117:0): cam completion status 0x4a
(probe118:isp0:0:118:0): cam completion status 0x4a
(probe118:isp0:0:118:0): cam completion status 0x4a
(probe119:isp0:0:119:0): cam completion status 0x4a
(probe119:isp0:0:119:0): cam completion status 0x4a
(probe120:isp0:0:120:0): cam completion status 0x4a
(probe120:isp0:0:120:0): cam completion status 0x4a
(probe121:isp0:0:121:0): cam completion status 0x4a
(probe121:isp0:0:121:0): cam completion status 0x4a
(probe122:isp0:0:122:0): cam completion status 0x4a
(probe122:isp0:0:122:0): cam completion status 0x4a
(probe123:isp0:0:123:0): cam completion status 0x4a
(probe123:isp0:0:123:0): cam completion status 0x4a
(probe124:isp0:0:124:0): cam completion status 0x4a
(probe124:isp0:0:124:0): cam completion status 0x4a
(probe125:isp0:0:125:0): cam completion status 0x4a
(probe125:isp0:0:125:0): cam completion status 0x4a
(probe126:isp0:0:126:0): cam completion status 0x4a
(probe126:isp0:0:126:0): cam completion status 0x4a
(probe127:isp0:0:127:0): cam completion status 0x4a
(probe127:isp0:0:127:0): cam completion status 0x4a
(probe128:isp0:0:128:0): cam completion status 0x4a
(probe128:isp0:0:128:0): cam completion status 0x4a
(probe129:isp0:0:129:0): cam completion status 0x4a
(probe129:isp0:0:129:0): cam completion status 0x4a
(probe130:isp0:0:130:0): cam completion status 0x4a
(probe130:isp0:0:130:0): cam completion status 0x4a
(probe132:isp0:0:132:0): cam completion status 0x4a
(probe132:isp0:0:132:0): cam completion status 0x4a
(probe133:isp0:0:133:0): cam completion status 0x4a
(probe133:isp0:0:133:0): cam completion status 0x4a
(probe134:isp0:0:134:0): cam completion status 0x4a
(probe134:isp0:0:134:0): cam completion status 0x4a
(probe135:isp0:0:135:0): cam completion status 0x4a
(probe135:isp0:0:135:0): cam completion status 0x4a
(probe136:isp0:0:136:0): cam completion status 0x4a
(probe136:isp0:0:136:0): cam completion status 0x4a
(probe137:isp0:0:137:0): cam completion status 0x4a
(probe137:isp0:0:137:0): cam completion status 0x4a
(probe138:isp0:0:138:0): cam completion status 0x4a
(probe138:isp0:0:138:0): cam completion status 0x4a
(probe139:isp0:0:139:0): cam completion status 0x4a
(probe139:isp0:0:139:0): cam completion status 0x4a
(probe140:isp0:0:140:0): cam completion status 0x4a
(probe140:isp0:0:140:0): cam completion status 0x4a
(probe141:isp0:0:141:0): cam completion status 0x4a
(probe141:isp0:0:141:0): cam completion status 0x4a
(probe142:isp0:0:142:0): cam completion status 0x4a
(probe142:isp0:0:142:0): cam completion status 0x4a
(probe143:isp0:0:143:0): cam completion status 0x4a
(probe143:isp0:0:143:0): cam completion status 0x4a
(probe144:isp0:0:144:0): cam completion status 0x4a
(probe144:isp0:0:144:0): cam completion status 0x4a
(probe145:isp0:0:145:0): cam completion status 0x4a
(probe145:isp0:0:145:0): cam completion status 0x4a
(probe146:isp0:0:146:0): cam completion status 0x4a
(probe146:isp0:0:146:0): cam completion status 0x4a
(probe148:isp0:0:148:0): cam completion status 0x4a
(probe148:isp0:0:148:0): cam completion status 0x4a
(probe149:isp0:0:149:0): cam completion status 0x4a
(probe149:isp0:0:149:0): cam completion status 0x4a
(probe150:isp0:0:150:0): cam completion status 0x4a
(probe150:isp0:0:150:0): cam completion status 0x4a
(probe151:isp0:0:151:0): cam completion status 0x4a
(probe151:isp0:0:151:0): cam completion status 0x4a
(probe152:isp0:0:152:0): cam completion status 0x4a
(probe152:isp0:0:152:0): cam completion status 0x4a
(probe153:isp0:0:153:0): cam completion status 0x4a
(probe153:isp0:0:153:0): cam completion status 0x4a
(probe154:isp0:0:154:0): cam completion status 0x4a
(probe154:isp0:0:154:0): cam completion status 0x4a
(probe155:isp0:0:155:0): cam completion status 0x4a
(probe155:isp0:0:155:0): cam completion status 0x4a
(probe156:isp0:0:156:0): cam completion status 0x4a
(probe156:isp0:0:156:0): cam completion status 0x4a
(probe157:isp0:0:157:0): cam completion status 0x4a
(probe157:isp0:0:157:0): cam completion status 0x4a
(probe158:isp0:0:158:0): cam completion status 0x4a
(probe158:isp0:0:158:0): cam completion status 0x4a
(probe159:isp0:0:159:0): cam completion status 0x4a
(probe159:isp0:0:159:0): cam completion status 0x4a
(probe160:isp0:0:160:0): cam completion status 0x4a
(probe160:isp0:0:160:0): cam completion status 0x4a
(probe161:isp0:0:161:0): cam completion status 0x4a
(probe161:isp0:0:161:0): cam completion status 0x4a
(probe162:isp0:0:162:0): cam completion status 0x4a
(probe162:isp0:0:162:0): cam completion status 0x4a
(probe163:isp0:0:163:0): cam completion status 0x4a
(probe163:isp0:0:163:0): cam completion status 0x4a
(probe164:isp0:0:164:0): cam completion status 0x4a
(probe164:isp0:0:164:0): cam completion status 0x4a
(probe165:isp0:0:165:0): cam completion status 0x4a
(probe165:isp0:0:165:0): cam completion status 0x4a
(probe166:isp0:0:166:0): cam completion status 0x4a
(probe166:isp0:0:166:0): cam completion status 0x4a
(probe167:isp0:0:167:0): cam completion status 0x4a
(probe167:isp0:0:167:0): cam completion status 0x4a
(probe168:isp0:0:168:0): cam completion status 0x4a
(probe168:isp0:0:168:0): cam completion status 0x4a
(probe169:isp0:0:169:0): cam completion status 0x4a
(probe169:isp0:0:169:0): cam completion status 0x4a
(probe170:isp0:0:170:0): cam completion status 0x4a
(probe170:isp0:0:170:0): cam completion status 0x4a
(probe171:isp0:0:171:0): cam completion status 0x4a
(probe171:isp0:0:171:0): cam completion status 0x4a
(probe172:isp0:0:172:0): cam completion status 0x4a
(probe172:isp0:0:172:0): cam completion status 0x4a
(probe173:isp0:0:173:0): cam completion status 0x4a
(probe173:isp0:0:173:0): cam completion status 0x4a
(probe174:isp0:0:174:0): cam completion status 0x4a
(probe174:isp0:0:174:0): cam completion status 0x4a
(probe175:isp0:0:175:0): cam completion status 0x4a
(probe175:isp0:0:175:0): cam completion status 0x4a
(probe176:isp0:0:176:0): cam completion status 0x4a
(probe176:isp0:0:176:0): cam completion status 0x4a
(probe177:isp0:0:177:0): cam completion status 0x4a
(probe177:isp0:0:177:0): cam completion status 0x4a
(probe178:isp0:0:178:0): cam completion status 0x4a
(probe178:isp0:0:178:0): cam completion status 0x4a
(probe179:isp0:0:179:0): cam completion status 0x4a
(probe179:isp0:0:179:0): cam completion status 0x4a
(probe180:isp0:0:180:0): cam completion status 0x4a
(probe180:isp0:0:180:0): cam completion status 0x4a
(probe181:isp0:0:181:0): cam completion status 0x4a
(probe181:isp0:0:181:0): cam completion status 0x4a
(probe182:isp0:0:182:0): cam completion status 0x4a
(probe182:isp0:0:182:0): cam completion status 0x4a
(probe183:isp0:0:183:0): cam completion status 0x4a
(probe183:isp0:0:183:0): cam completion status 0x4a
(probe184:isp0:0:184:0): cam completion status 0x4a
(probe184:isp0:0:184:0): cam completion status 0x4a
(probe185:isp0:0:185:0): cam completion status 0x4a
(probe185:isp0:0:185:0): cam completion status 0x4a
(probe186:isp0:0:186:0): cam completion status 0x4a
(probe186:isp0:0:186:0): cam completion status 0x4a
(probe187:isp0:0:187:0): cam completion status 0x4a
(probe187:isp0:0:187:0): cam completion status 0x4a
(probe188:isp0:0:188:0): cam completion status 0x4a
(probe188:isp0:0:188:0): cam completion status 0x4a
(probe189:isp0:0:189:0): cam completion status 0x4a
(probe189:isp0:0:189:0): cam completion status 0x4a
(probe190:isp0:0:190:0): cam completion status 0x4a
(probe190:isp0:0:190:0): cam completion status 0x4a
(probe191:isp0:0:191:0): cam completion status 0x4a
(probe191:isp0:0:191:0): cam completion status 0x4a
(probe192:isp0:0:192:0): cam completion status 0x4a
(probe192:isp0:0:192:0): cam completion status 0x4a
(probe193:isp0:0:193:0): cam completion status 0x4a
(probe193:isp0:0:193:0): cam completion status 0x4a
(probe194:isp0:0:194:0): cam completion status 0x4a
(probe194:isp0:0:194:0): cam completion status 0x4a
(probe195:isp0:0:195:0): cam completion status 0x4a
(probe195:isp0:0:195:0): cam completion status 0x4a
(probe196:isp0:0:196:0): cam completion status 0x4a
(probe196:isp0:0:196:0): cam completion status 0x4a
(probe197:isp0:0:197:0): cam completion status 0x4a
(probe197:isp0:0:197:0): cam completion status 0x4a
(probe198:isp0:0:198:0): cam completion status 0x4a
(probe198:isp0:0:198:0): cam completion status 0x4a
(probe199:isp0:0:199:0): cam completion status 0x4a
(probe199:isp0:0:199:0): cam completion status 0x4a
(probe200:isp0:0:200:0): cam completion status 0x4a
(probe200:isp0:0:200:0): cam completion status 0x4a
(probe201:isp0:0:201:0): cam completion status 0x4a
(probe201:isp0:0:201:0): cam completion status 0x4a
(probe202:isp0:0:202:0): cam completion status 0x4a
(probe202:isp0:0:202:0): cam completion status 0x4a
(probe203:isp0:0:203:0): cam completion status 0x4a
(probe203:isp0:0:203:0): cam completion status 0x4a
(probe204:isp0:0:204:0): cam completion status 0x4a
(probe204:isp0:0:204:0): cam completion status 0x4a
(probe205:isp0:0:205:0): cam completion status 0x4a
(probe205:isp0:0:205:0): cam completion status 0x4a
(probe207:isp0:0:207:0): cam completion status 0x4a
(probe207:isp0:0:207:0): cam completion status 0x4a
(probe208:isp0:0:208:0): cam completion status 0x4a
(probe208:isp0:0:208:0): cam completion status 0x4a
(probe209:isp0:0:209:0): cam completion status 0x4a
(probe209:isp0:0:209:0): cam completion status 0x4a
(probe210:isp0:0:210:0): cam completion status 0x4a
(probe210:isp0:0:210:0): cam completion status 0x4a
(probe211:isp0:0:211:0): cam completion status 0x4a
(probe211:isp0:0:211:0): cam completion status 0x4a
(probe212:isp0:0:212:0): cam completion status 0x4a
(probe212:isp0:0:212:0): cam completion status 0x4a
(probe213:isp0:0:213:0): cam completion status 0x4a
(probe213:isp0:0:213:0): cam completion status 0x4a
(probe214:isp0:0:214:0): cam completion status 0x4a
(probe214:isp0:0:214:0): cam completion status 0x4a
(probe215:isp0:0:215:0): cam completion status 0x4a
(probe215:isp0:0:215:0): cam completion status 0x4a
(probe216:isp0:0:216:0): cam completion status 0x4a
(probe216:isp0:0:216:0): cam completion status 0x4a
(probe217:isp0:0:217:0): cam completion status 0x4a
(probe217:isp0:0:217:0): cam completion status 0x4a
(probe218:isp0:0:218:0): cam completion status 0x4a
(probe218:isp0:0:218:0): cam completion status 0x4a
(probe219:isp0:0:219:0): cam completion status 0x4a
(probe219:isp0:0:219:0): cam completion status 0x4a
(probe220:isp0:0:220:0): cam completion status 0x4a
(probe220:isp0:0:220:0): cam completion status 0x4a
(probe221:isp0:0:221:0): cam completion status 0x4a
(probe221:isp0:0:221:0): cam completion status 0x4a
(probe222:isp0:0:222:0): cam completion status 0x4a
(probe222:isp0:0:222:0): cam completion status 0x4a
(probe223:isp0:0:223:0): cam completion status 0x4a
(probe223:isp0:0:223:0): cam completion status 0x4a
(probe224:isp0:0:224:0): cam completion status 0x4a
(probe224:isp0:0:224:0): cam completion status 0x4a
(probe225:isp0:0:225:0): cam completion status 0x4a
(probe225:isp0:0:225:0): cam completion status 0x4a
(probe226:isp0:0:226:0): cam completion status 0x4a
(probe226:isp0:0:226:0): cam completion status 0x4a
(probe227:isp0:0:227:0): cam completion status 0x4a
(probe227:isp0:0:227:0): cam completion status 0x4a
(probe228:isp0:0:228:0): cam completion status 0x4a
(probe228:isp0:0:228:0): cam completion status 0x4a
(probe229:isp0:0:229:0): cam completion status 0x4a
(probe229:isp0:0:229:0): cam completion status 0x4a
(probe230:isp0:0:230:0): cam completion status 0x4a
(probe230:isp0:0:230:0): cam completion status 0x4a
(probe231:isp0:0:231:0): cam completion status 0x4a
(probe231:isp0:0:231:0): cam completion status 0x4a
(probe232:isp0:0:232:0): cam completion status 0x4a
(probe232:isp0:0:232:0): cam completion status 0x4a
(probe233:isp0:0:233:0): cam completion status 0x4a
(probe233:isp0:0:233:0): cam completion status 0x4a
(probe234:isp0:0:234:0): cam completion status 0x4a
(probe234:isp0:0:234:0): cam completion status 0x4a
(probe235:isp0:0:235:0): cam completion status 0x4a
(probe235:isp0:0:235:0): cam completion status 0x4a
(probe236:isp0:0:236:0): cam completion status 0x4a
(probe236:isp0:0:236:0): cam completion status 0x4a
(probe237:isp0:0:237:0): cam completion status 0x4a
(probe237:isp0:0:237:0): cam completion status 0x4a
(probe238:isp0:0:238:0): cam completion status 0x4a
(probe238:isp0:0:238:0): cam completion status 0x4a
(probe239:isp0:0:239:0): cam completion status 0x4a
(probe239:isp0:0:239:0): cam completion status 0x4a
(probe240:isp0:0:240:0): cam completion status 0x4a
(probe240:isp0:0:240:0): cam completion status 0x4a
(probe241:isp0:0:241:0): cam completion status 0x4a
(probe241:isp0:0:241:0): cam completion status 0x4a
(probe242:isp0:0:242:0): cam completion status 0x4a
(probe242:isp0:0:242:0): cam completion status 0x4a
(probe243:isp0:0:243:0): cam completion status 0x4a
(probe243:isp0:0:243:0): cam completion status 0x4a
(probe244:isp0:0:244:0): cam completion status 0x4a
(probe244:isp0:0:244:0): cam completion status 0x4a
(probe245:isp0:0:245:0): cam completion status 0x4a
(probe245:isp0:0:245:0): cam completion status 0x4a
(probe246:isp0:0:246:0): cam completion status 0x4a
(probe246:isp0:0:246:0): cam completion status 0x4a
(probe247:isp0:0:247:0): cam completion status 0x4a
(probe247:isp0:0:247:0): cam completion status 0x4a
(probe248:isp0:0:248:0): cam completion status 0x4a
(probe248:isp0:0:248:0): cam completion status 0x4a
(probe249:isp0:0:249:0): cam completion status 0x4a
(probe249:isp0:0:249:0): cam completion status 0x4a
(probe250:isp0:0:250:0): cam completion status 0x4a
(probe250:isp0:0:250:0): cam completion status 0x4a
(probe251:isp0:0:251:0): cam completion status 0x4a
(probe251:isp0:0:251:0): cam completion status 0x4a
(probe252:isp0:0:252:0): cam completion status 0x4a
(probe252:isp0:0:252:0): cam completion status 0x4a
(probe253:isp0:0:253:0): cam completion status 0x4a
(probe253:isp0:0:253:0): cam completion status 0x4a
(probe254:isp0:0:254:0): cam completion status 0x4a
(probe254:isp0:0:254:0): cam completion status 0x4a
(probe255:isp0:0:255:0): cam completion status 0x4a
(probe255:isp0:0:255:0): cam completion status 0x4a
(probe256:isp0:0:256:0): cam completion status 0x4a
(probe256:isp0:0:256:0): cam completion status 0x4a
(probe257:isp0:0:257:0): cam completion status 0x4a
(probe257:isp0:0:257:0): cam completion status 0x4a
(probe258:isp0:0:258:0): cam completion status 0x4a
(probe258:isp0:0:258:0): cam completion status 0x4a
(probe259:isp0:0:259:0): cam completion status 0x4a
(probe259:isp0:0:259:0): cam completion status 0x4a
(probe260:isp0:0:260:0): cam completion status 0x4a
(probe260:isp0:0:260:0): cam completion status 0x4a
(probe261:isp0:0:261:0): cam completion status 0x4a
(probe261:isp0:0:261:0): cam completion status 0x4a
(probe262:isp0:0:262:0): cam completion status 0x4a
(probe262:isp0:0:262:0): cam completion status 0x4a
(probe263:isp0:0:263:0): cam completion status 0x4a
(probe263:isp0:0:263:0): cam completion status 0x4a
(probe264:isp0:0:264:0): cam completion status 0x4a
(probe264:isp0:0:264:0): cam completion status 0x4a
(probe265:isp0:0:265:0): cam completion status 0x4a
(probe265:isp0:0:265:0): cam completion status 0x4a
(probe266:isp0:0:266:0): cam completion status 0x4a
(probe266:isp0:0:266:0): cam completion status 0x4a
(probe267:isp0:0:267:0): cam completion status 0x4a
(probe267:isp0:0:267:0): cam completion status 0x4a
(probe268:isp0:0:268:0): cam completion status 0x4a
(probe268:isp0:0:268:0): cam completion status 0x4a
(probe269:isp0:0:269:0): cam completion status 0x4a
(probe269:isp0:0:269:0): cam completion status 0x4a
(probe270:isp0:0:270:0): cam completion status 0x4a
(probe270:isp0:0:270:0): cam completion status 0x4a
(probe271:isp0:0:271:0): cam completion status 0x4a
(probe271:isp0:0:271:0): cam completion status 0x4a
(probe272:isp0:0:272:0): cam completion status 0x4a
(probe272:isp0:0:272:0): cam completion status 0x4a
(probe273:isp0:0:273:0): cam completion status 0x4a
(probe273:isp0:0:273:0): cam completion status 0x4a
(probe274:isp0:0:274:0): cam completion status 0x4a
(probe274:isp0:0:274:0): cam completion status 0x4a
(probe275:isp0:0:275:0): cam completion status 0x4a
(probe275:isp0:0:275:0): cam completion status 0x4a
(probe276:isp0:0:276:0): cam completion status 0x4a
(probe276:isp0:0:276:0): cam completion status 0x4a
(probe277:isp0:0:277:0): cam completion status 0x4a
(probe277:isp0:0:277:0): cam completion status 0x4a
(probe278:isp0:0:278:0): cam completion status 0x4a
(probe278:isp0:0:278:0): cam completion status 0x4a
(probe279:isp0:0:279:0): cam completion status 0x4a
(probe279:isp0:0:279:0): cam completion status 0x4a
(probe280:isp0:0:280:0): cam completion status 0x4a
(probe280:isp0:0:280:0): cam completion status 0x4a
(probe281:isp0:0:281:0): cam completion status 0x4a
(probe281:isp0:0:281:0): cam completion status 0x4a
(probe282:isp0:0:282:0): cam completion status 0x4a
(probe282:isp0:0:282:0): cam completion status 0x4a
(probe283:isp0:0:283:0): cam completion status 0x4a
(probe283:isp0:0:283:0): cam completion status 0x4a
(probe284:isp0:0:284:0): cam completion status 0x4a
(probe284:isp0:0:284:0): cam completion status 0x4a
(probe285:isp0:0:285:0): cam completion status 0x4a
(probe285:isp0:0:285:0): cam completion status 0x4a
(probe286:isp0:0:286:0): cam completion status 0x4a
(probe286:isp0:0:286:0): cam completion status 0x4a
(probe287:isp0:0:287:0): cam completion status 0x4a
(probe287:isp0:0:287:0): cam completion status 0x4a
(probe288:isp0:0:288:0): cam completion status 0x4a
(probe288:isp0:0:288:0): cam completion status 0x4a
(probe289:isp0:0:289:0): cam completion status 0x4a
(probe289:isp0:0:289:0): cam completion status 0x4a
(probe290:isp0:0:290:0): cam completion status 0x4a
(probe290:isp0:0:290:0): cam completion status 0x4a
(probe291:isp0:0:291:0): cam completion status 0x4a
(probe291:isp0:0:291:0): cam completion status 0x4a
(probe292:isp0:0:292:0): cam completion status 0x4a
(probe292:isp0:0:292:0): cam completion status 0x4a
(probe293:isp0:0:293:0): cam completion status 0x4a
(probe293:isp0:0:293:0): cam completion status 0x4a
(probe294:isp0:0:294:0): cam completion status 0x4a
(probe294:isp0:0:294:0): cam completion status 0x4a
(probe295:isp0:0:295:0): cam completion status 0x4a
(probe295:isp0:0:295:0): cam completion status 0x4a
(probe296:isp0:0:296:0): cam completion status 0x4a
(probe296:isp0:0:296:0): cam completion status 0x4a
(probe297:isp0:0:297:0): cam completion status 0x4a
(probe297:isp0:0:297:0): cam completion status 0x4a
(probe298:isp0:0:298:0): cam completion status 0x4a
(probe298:isp0:0:298:0): cam completion status 0x4a
(probe299:isp0:0:299:0): cam completion status 0x4a
(probe299:isp0:0:299:0): cam completion status 0x4a
(probe300:isp0:0:300:0): cam completion status 0x4a
(probe300:isp0:0:300:0): cam completion status 0x4a
(probe301:isp0:0:301:0): cam completion status 0x4a
(probe301:isp0:0:301:0): cam completion status 0x4a
(probe302:isp0:0:302:0): cam completion status 0x4a
(probe302:isp0:0:302:0): cam completion status 0x4a
(probe303:isp0:0:303:0): cam completion status 0x4a
(probe303:isp0:0:303:0): cam completion status 0x4a
(probe304:isp0:0:304:0): cam completion status 0x4a
(probe304:isp0:0:304:0): cam completion status 0x4a
(probe305:isp0:0:305:0): cam completion status 0x4a
(probe305:isp0:0:305:0): cam completion status 0x4a
(probe306:isp0:0:306:0): cam completion status 0x4a
(probe306:isp0:0:306:0): cam completion status 0x4a
(probe307:isp0:0:307:0): cam completion status 0x4a
(probe307:isp0:0:307:0): cam completion status 0x4a
(probe308:isp0:0:308:0): cam completion status 0x4a
(probe308:isp0:0:308:0): cam completion status 0x4a
(probe309:isp0:0:309:0): cam completion status 0x4a
(probe309:isp0:0:309:0): cam completion status 0x4a
(probe310:isp0:0:310:0): cam completion status 0x4a
(probe310:isp0:0:310:0): cam completion status 0x4a
(probe311:isp0:0:311:0): cam completion status 0x4a
(probe311:isp0:0:311:0): cam completion status 0x4a
(probe312:isp0:0:312:0): cam completion status 0x4a
(probe312:isp0:0:312:0): cam completion status 0x4a
(probe313:isp0:0:313:0): cam completion status 0x4a
(probe313:isp0:0:313:0): cam completion status 0x4a
(probe314:isp0:0:314:0): cam completion status 0x4a
(probe314:isp0:0:314:0): cam completion status 0x4a
(probe315:isp0:0:315:0): cam completion status 0x4a
(probe315:isp0:0:315:0): cam completion status 0x4a
(probe316:isp0:0:316:0): cam completion status 0x4a
(probe316:isp0:0:316:0): cam completion status 0x4a
(probe317:isp0:0:317:0): cam completion status 0x4a
(probe317:isp0:0:317:0): cam completion status 0x4a
(probe318:isp0:0:318:0): cam completion status 0x4a
(probe318:isp0:0:318:0): cam completion status 0x4a
(probe319:isp0:0:319:0): cam completion status 0x4a
(probe319:isp0:0:319:0): cam completion status 0x4a
(probe320:isp0:0:320:0): cam completion status 0x4a
(probe320:isp0:0:320:0): cam completion status 0x4a
(probe321:isp0:0:321:0): cam completion status 0x4a
(probe321:isp0:0:321:0): cam completion status 0x4a
(probe322:isp0:0:322:0): cam completion status 0x4a
(probe322:isp0:0:322:0): cam completion status 0x4a
(probe323:isp0:0:323:0): cam completion status 0x4a
(probe323:isp0:0:323:0): cam completion status 0x4a
(probe324:isp0:0:324:0): cam completion status 0x4a
(probe324:isp0:0:324:0): cam completion status 0x4a
(probe325:isp0:0:325:0): cam completion status 0x4a
(probe325:isp0:0:325:0): cam completion status 0x4a
(probe326:isp0:0:326:0): cam completion status 0x4a
(probe326:isp0:0:326:0): cam completion status 0x4a
(probe327:isp0:0:327:0): cam completion status 0x4a
(probe327:isp0:0:327:0): cam completion status 0x4a
(probe328:isp0:0:328:0): cam completion status 0x4a
(probe328:isp0:0:328:0): cam completion status 0x4a
(probe329:isp0:0:329:0): cam completion status 0x4a
(probe329:isp0:0:329:0): cam completion status 0x4a
(probe330:isp0:0:330:0): cam completion status 0x4a
(probe330:isp0:0:330:0): cam completion status 0x4a
(probe331:isp0:0:331:0): cam completion status 0x4a
(probe331:isp0:0:331:0): cam completion status 0x4a
(probe332:isp0:0:332:0): cam completion status 0x4a
(probe332:isp0:0:332:0): cam completion status 0x4a
(probe333:isp0:0:333:0): cam completion status 0x4a
(probe333:isp0:0:333:0): cam completion status 0x4a
(probe334:isp0:0:334:0): cam completion status 0x4a
(probe334:isp0:0:334:0): cam completion status 0x4a
(probe335:isp0:0:335:0): cam completion status 0x4a
(probe335:isp0:0:335:0): cam completion status 0x4a
(probe336:isp0:0:336:0): cam completion status 0x4a
(probe336:isp0:0:336:0): cam completion status 0x4a
(probe337:isp0:0:337:0): cam completion status 0x4a
(probe337:isp0:0:337:0): cam completion status 0x4a
(probe338:isp0:0:338:0): cam completion status 0x4a
(probe338:isp0:0:338:0): cam completion status 0x4a
(probe339:isp0:0:339:0): cam completion status 0x4a
(probe339:isp0:0:339:0): cam completion status 0x4a
(probe340:isp0:0:340:0): cam completion status 0x4a
(probe340:isp0:0:340:0): cam completion status 0x4a
(probe341:isp0:0:341:0): cam completion status 0x4a
(probe341:isp0:0:341:0): cam completion status 0x4a
(probe342:isp0:0:342:0): cam completion status 0x4a
(probe342:isp0:0:342:0): cam completion status 0x4a
(probe343:isp0:0:343:0): cam completion status 0x4a
(probe343:isp0:0:343:0): cam completion status 0x4a
(probe344:isp0:0:344:0): cam completion status 0x4a
(probe344:isp0:0:344:0): cam completion status 0x4a
(probe345:isp0:0:345:0): cam completion status 0x4a
(probe345:isp0:0:345:0): cam completion status 0x4a
(probe346:isp0:0:346:0): cam completion status 0x4a
(probe346:isp0:0:346:0): cam completion status 0x4a
(probe347:isp0:0:347:0): cam completion status 0x4a
(probe347:isp0:0:347:0): cam completion status 0x4a
(probe348:isp0:0:348:0): cam completion status 0x4a
(probe348:isp0:0:348:0): cam completion status 0x4a
(probe349:isp0:0:349:0): cam completion status 0x4a
(probe349:isp0:0:349:0): cam completion status 0x4a
(probe350:isp0:0:350:0): cam completion status 0x4a
(probe350:isp0:0:350:0): cam completion status 0x4a
(probe351:isp0:0:351:0): cam completion status 0x4a
(probe351:isp0:0:351:0): cam completion status 0x4a
(probe352:isp0:0:352:0): cam completion status 0x4a
(probe352:isp0:0:352:0): cam completion status 0x4a
(probe353:isp0:0:353:0): cam completion status 0x4a
(probe353:isp0:0:353:0): cam completion status 0x4a
(probe354:isp0:0:354:0): cam completion status 0x4a
(probe354:isp0:0:354:0): cam completion status 0x4a
(probe355:isp0:0:355:0): cam completion status 0x4a
(probe355:isp0:0:355:0): cam completion status 0x4a
(probe356:isp0:0:356:0): cam completion status 0x4a
(probe356:isp0:0:356:0): cam completion status 0x4a
(probe357:isp0:0:357:0): cam completion status 0x4a
(probe357:isp0:0:357:0): cam completion status 0x4a
(probe358:isp0:0:358:0): cam completion status 0x4a
(probe358:isp0:0:358:0): cam completion status 0x4a
(probe359:isp0:0:359:0): cam completion status 0x4a
(probe359:isp0:0:359:0): cam completion status 0x4a
(probe360:isp0:0:360:0): cam completion status 0x4a
(probe360:isp0:0:360:0): cam completion status 0x4a
(probe361:isp0:0:361:0): cam completion status 0x4a
(probe361:isp0:0:361:0): cam completion status 0x4a
(probe362:isp0:0:362:0): cam completion status 0x4a
(probe362:isp0:0:362:0): cam completion status 0x4a
(probe363:isp0:0:363:0): cam completion status 0x4a
(probe363:isp0:0:363:0): cam completion status 0x4a
(probe364:isp0:0:364:0): cam completion status 0x4a
(probe364:isp0:0:364:0): cam completion status 0x4a
(probe365:isp0:0:365:0): cam completion status 0x4a
(probe365:isp0:0:365:0): cam completion status 0x4a
(probe366:isp0:0:366:0): cam completion status 0x4a
(probe366:isp0:0:366:0): cam completion status 0x4a
(probe367:isp0:0:367:0): cam completion status 0x4a
(probe367:isp0:0:367:0): cam completion status 0x4a
(probe368:isp0:0:368:0): cam completion status 0x4a
(probe368:isp0:0:368:0): cam completion status 0x4a
(probe369:isp0:0:369:0): cam completion status 0x4a
(probe369:isp0:0:369:0): cam completion status 0x4a
(probe370:isp0:0:370:0): cam completion status 0x4a
(probe370:isp0:0:370:0): cam completion status 0x4a
(probe371:isp0:0:371:0): cam completion status 0x4a
(probe371:isp0:0:371:0): cam completion status 0x4a
(probe372:isp0:0:372:0): cam completion status 0x4a
(probe372:isp0:0:372:0): cam completion status 0x4a
(probe373:isp0:0:373:0): cam completion status 0x4a
(probe373:isp0:0:373:0): cam completion status 0x4a
(probe374:isp0:0:374:0): cam completion status 0x4a
(probe374:isp0:0:374:0): cam completion status 0x4a
(probe375:isp0:0:375:0): cam completion status 0x4a
(probe375:isp0:0:375:0): cam completion status 0x4a
(probe377:isp0:0:377:0): cam completion status 0x4a
(probe377:isp0:0:377:0): cam completion status 0x4a
(probe378:isp0:0:378:0): cam completion status 0x4a
(probe378:isp0:0:378:0): cam completion status 0x4a
(probe379:isp0:0:379:0): cam completion status 0x4a
(probe379:isp0:0:379:0): cam completion status 0x4a
(probe380:isp0:0:380:0): cam completion status 0x4a
(probe380:isp0:0:380:0): cam completion status 0x4a
(probe381:isp0:0:381:0): cam completion status 0x4a
(probe381:isp0:0:381:0): cam completion status 0x4a
(probe382:isp0:0:382:0): cam completion status 0x4a
(probe382:isp0:0:382:0): cam completion status 0x4a
(probe383:isp0:0:383:0): cam completion status 0x4a
(probe383:isp0:0:383:0): cam completion status 0x4a
(probe384:isp0:0:384:0): cam completion status 0x4a
(probe384:isp0:0:384:0): cam completion status 0x4a
(probe385:isp0:0:385:0): cam completion status 0x4a
(probe385:isp0:0:385:0): cam completion status 0x4a
(probe386:isp0:0:386:0): cam completion status 0x4a
(probe386:isp0:0:386:0): cam completion status 0x4a
(probe387:isp0:0:387:0): cam completion status 0x4a
(probe387:isp0:0:387:0): cam completion status 0x4a
(probe388:isp0:0:388:0): cam completion status 0x4a
(probe388:isp0:0:388:0): cam completion status 0x4a
(probe389:isp0:0:389:0): cam completion status 0x4a
(probe389:isp0:0:389:0): cam completion status 0x4a
(probe390:isp0:0:390:0): cam completion status 0x4a
(probe390:isp0:0:390:0): cam completion status 0x4a
(probe391:isp0:0:391:0): cam completion status 0x4a
(probe391:isp0:0:391:0): cam completion status 0x4a
(probe392:isp0:0:392:0): cam completion status 0x4a
(probe392:isp0:0:392:0): cam completion status 0x4a
(probe393:isp0:0:393:0): cam completion status 0x4a
(probe393:isp0:0:393:0): cam completion status 0x4a
(probe394:isp0:0:394:0): cam completion status 0x4a
(probe394:isp0:0:394:0): cam completion status 0x4a
(probe396:isp0:0:396:0): cam completion status 0x4a
(probe396:isp0:0:396:0): cam completion status 0x4a
(probe397:isp0:0:397:0): cam completion status 0x4a
(probe397:isp0:0:397:0): cam completion status 0x4a
(probe398:isp0:0:398:0): cam completion status 0x4a
(probe398:isp0:0:398:0): cam completion status 0x4a
(probe399:isp0:0:399:0): cam completion status 0x4a
(probe399:isp0:0:399:0): cam completion status 0x4a
(probe401:isp0:0:401:0): cam completion status 0x4a
(probe401:isp0:0:401:0): cam completion status 0x4a
(probe402:isp0:0:402:0): cam completion status 0x4a
(probe402:isp0:0:402:0): cam completion status 0x4a
(probe403:isp0:0:403:0): cam completion status 0x4a
(probe403:isp0:0:403:0): cam completion status 0x4a
(probe404:isp0:0:404:0): cam completion status 0x4a
(probe404:isp0:0:404:0): cam completion status 0x4a
(probe405:isp0:0:405:0): cam completion status 0x4a
(probe405:isp0:0:405:0): cam completion status 0x4a
(probe406:isp0:0:406:0): cam completion status 0x4a
(probe406:isp0:0:406:0): cam completion status 0x4a
(probe407:isp0:0:407:0): cam completion status 0x4a
(probe407:isp0:0:407:0): cam completion status 0x4a
(probe408:isp0:0:408:0): cam completion status 0x4a
(probe408:isp0:0:408:0): cam completion status 0x4a
(probe409:isp0:0:409:0): cam completion status 0x4a
(probe409:isp0:0:409:0): cam completion status 0x4a
(probe410:isp0:0:410:0): cam completion status 0x4a
(probe410:isp0:0:410:0): cam completion status 0x4a
(probe412:isp0:0:412:0): cam completion status 0x4a
(probe412:isp0:0:412:0): cam completion status 0x4a
(probe413:isp0:0:413:0): cam completion status 0x4a
(probe413:isp0:0:413:0): cam completion status 0x4a
(probe414:isp0:0:414:0): cam completion status 0x4a
(probe414:isp0:0:414:0): cam completion status 0x4a
(probe415:isp0:0:415:0): cam completion status 0x4a
(probe415:isp0:0:415:0): cam completion status 0x4a
(probe416:isp0:0:416:0): cam completion status 0x4a
(probe416:isp0:0:416:0): cam completion status 0x4a
(probe417:isp0:0:417:0): cam completion status 0x4a
(probe417:isp0:0:417:0): cam completion status 0x4a
(probe418:isp0:0:418:0): cam completion status 0x4a
(probe418:isp0:0:418:0): cam completion status 0x4a
(probe419:isp0:0:419:0): cam completion status 0x4a
(probe419:isp0:0:419:0): cam completion status 0x4a
(probe420:isp0:0:420:0): cam completion status 0x4a
(probe420:isp0:0:420:0): cam completion status 0x4a
(probe421:isp0:0:421:0): cam completion status 0x4a
(probe421:isp0:0:421:0): cam completion status 0x4a
(probe422:isp0:0:422:0): cam completion status 0x4a
(probe422:isp0:0:422:0): cam completion status 0x4a
(probe424:isp0:0:424:0): cam completion status 0x4a
(probe424:isp0:0:424:0): cam completion status 0x4a
(probe425:isp0:0:425:0): cam completion status 0x4a
(probe425:isp0:0:425:0): cam completion status 0x4a
(probe426:isp0:0:426:0): cam completion status 0x4a
(probe426:isp0:0:426:0): cam completion status 0x4a
(probe427:isp0:0:427:0): cam completion status 0x4a
(probe427:isp0:0:427:0): cam completion status 0x4a
(probe428:isp0:0:428:0): cam completion status 0x4a
(probe428:isp0:0:428:0): cam completion status 0x4a
(probe429:isp0:0:429:0): cam completion status 0x4a
(probe429:isp0:0:429:0): cam completion status 0x4a
(probe430:isp0:0:430:0): cam completion status 0x4a
(probe430:isp0:0:430:0): cam completion status 0x4a
(probe431:isp0:0:431:0): cam completion status 0x4a
(probe431:isp0:0:431:0): cam completion status 0x4a
(probe432:isp0:0:432:0): cam completion status 0x4a
(probe432:isp0:0:432:0): cam completion status 0x4a
(probe433:isp0:0:433:0): cam completion status 0x4a
(probe433:isp0:0:433:0): cam completion status 0x4a
(probe434:isp0:0:434:0): cam completion status 0x4a
(probe434:isp0:0:434:0): cam completion status 0x4a
(probe435:isp0:0:435:0): cam completion status 0x4a
(probe435:isp0:0:435:0): cam completion status 0x4a
(probe436:isp0:0:436:0): cam completion status 0x4a
(probe436:isp0:0:436:0): cam completion status 0x4a
(probe438:isp0:0:438:0): cam completion status 0x4a
(probe438:isp0:0:438:0): cam completion status 0x4a
(probe439:isp0:0:439:0): cam completion status 0x4a
(probe439:isp0:0:439:0): cam completion status 0x4a
(probe440:isp0:0:440:0): cam completion status 0x4a
(probe440:isp0:0:440:0): cam completion status 0x4a
(probe441:isp0:0:441:0): cam completion status 0x4a
(probe441:isp0:0:441:0): cam completion status 0x4a
(probe442:isp0:0:442:0): cam completion status 0x4a
(probe442:isp0:0:442:0): cam completion status 0x4a
(probe443:isp0:0:443:0): cam completion status 0x4a
(probe443:isp0:0:443:0): cam completion status 0x4a
(probe444:isp0:0:444:0): cam completion status 0x4a
(probe444:isp0:0:444:0): cam completion status 0x4a
(probe445:isp0:0:445:0): cam completion status 0x4a
(probe445:isp0:0:445:0): cam completion status 0x4a
(probe446:isp0:0:446:0): cam completion status 0x4a
(probe446:isp0:0:446:0): cam completion status 0x4a
(probe447:isp0:0:447:0): cam completion status 0x4a
(probe447:isp0:0:447:0): cam completion status 0x4a
(probe448:isp0:0:448:0): cam completion status 0x4a
(probe448:isp0:0:448:0): cam completion status 0x4a
(probe450:isp0:0:450:0): cam completion status 0x4a
(probe450:isp0:0:450:0): cam completion status 0x4a
(probe451:isp0:0:451:0): cam completion status 0x4a
(probe451:isp0:0:451:0): cam completion status 0x4a
(probe452:isp0:0:452:0): cam completion status 0x4a
(probe452:isp0:0:452:0): cam completion status 0x4a
(probe453:isp0:0:453:0): cam completion status 0x4a
(probe453:isp0:0:453:0): cam completion status 0x4a
(probe454:isp0:0:454:0): cam completion status 0x4a
(probe454:isp0:0:454:0): cam completion status 0x4a
(probe455:isp0:0:455:0): cam completion status 0x4a
(probe455:isp0:0:455:0): cam completion status 0x4a
(probe456:isp0:0:456:0): cam completion status 0x4a
(probe456:isp0:0:456:0): cam completion status 0x4a
(probe457:isp0:0:457:0): cam completion status 0x4a
(probe457:isp0:0:457:0): cam completion status 0x4a
(probe458:isp0:0:458:0): cam completion status 0x4a
(probe458:isp0:0:458:0): cam completion status 0x4a
(probe460:isp0:0:460:0): cam completion status 0x4a
(probe460:isp0:0:460:0): cam completion status 0x4a
(probe461:isp0:0:461:0): cam completion status 0x4a
(probe461:isp0:0:461:0): cam completion status 0x4a
(probe462:isp0:0:462:0): cam completion status 0x4a
(probe462:isp0:0:462:0): cam completion status 0x4a
(probe463:isp0:0:463:0): cam completion status 0x4a
(probe463:isp0:0:463:0): cam completion status 0x4a
(probe465:isp0:0:465:0): cam completion status 0x4a
(probe465:isp0:0:465:0): cam completion status 0x4a
(probe466:isp0:0:466:0): cam completion status 0x4a
(probe466:isp0:0:466:0): cam completion status 0x4a
(probe467:isp0:0:467:0): cam completion status 0x4a
(probe467:isp0:0:467:0): cam completion status 0x4a
(probe468:isp0:0:468:0): cam completion status 0x4a
(probe468:isp0:0:468:0): cam completion status 0x4a
(probe469:isp0:0:469:0): cam completion status 0x4a
(probe469:isp0:0:469:0): cam completion status 0x4a
(probe470:isp0:0:470:0): cam completion status 0x4a
(probe470:isp0:0:470:0): cam completion status 0x4a
(probe472:isp0:0:472:0): cam completion status 0x4a
(probe472:isp0:0:472:0): cam completion status 0x4a
(probe473:isp0:0:473:0): cam completion status 0x4a
(probe473:isp0:0:473:0): cam completion status 0x4a
(probe474:isp0:0:474:0): cam completion status 0x4a
(probe474:isp0:0:474:0): cam completion status 0x4a
(probe475:isp0:0:475:0): cam completion status 0x4a
(probe475:isp0:0:475:0): cam completion status 0x4a
(probe476:isp0:0:476:0): cam completion status 0x4a
(probe476:isp0:0:476:0): cam completion status 0x4a
(probe477:isp0:0:477:0): cam completion status 0x4a
(probe477:isp0:0:477:0): cam completion status 0x4a
(probe479:isp0:0:479:0): cam completion status 0x4a
(probe479:isp0:0:479:0): cam completion status 0x4a
(probe480:isp0:0:480:0): cam completion status 0x4a
(probe480:isp0:0:480:0): cam completion status 0x4a
(probe481:isp0:0:481:0): cam completion status 0x4a
(probe481:isp0:0:481:0): cam completion status 0x4a
(probe482:isp0:0:482:0): cam completion status 0x4a
(probe482:isp0:0:482:0): cam completion status 0x4a
(probe483:isp0:0:483:0): cam completion status 0x4a
(probe483:isp0:0:483:0): cam completion status 0x4a
(probe484:isp0:0:484:0): cam completion status 0x4a
(probe484:isp0:0:484:0): cam completion status 0x4a
(probe485:isp0:0:485:0): cam completion status 0x4a
(probe485:isp0:0:485:0): cam completion status 0x4a
(probe486:isp0:0:486:0): cam completion status 0x4a
(probe486:isp0:0:486:0): cam completion status 0x4a
(probe487:isp0:0:487:0): cam completion status 0x4a
(probe487:isp0:0:487:0): cam completion status 0x4a
(probe489:isp0:0:489:0): cam completion status 0x4a
(probe489:isp0:0:489:0): cam completion status 0x4a
(probe490:isp0:0:490:0): cam completion status 0x4a
(probe490:isp0:0:490:0): cam completion status 0x4a
(probe491:isp0:0:491:0): cam completion status 0x4a
(probe491:isp0:0:491:0): cam completion status 0x4a
(probe492:isp0:0:492:0): cam completion status 0x4a
(probe492:isp0:0:492:0): cam completion status 0x4a
(probe493:isp0:0:493:0): cam completion status 0x4a
(probe493:isp0:0:493:0): cam completion status 0x4a
(probe494:isp0:0:494:0): cam completion status 0x4a
(probe494:isp0:0:494:0): cam completion status 0x4a
(probe495:isp0:0:495:0): cam completion status 0x4a
(probe495:isp0:0:495:0): cam completion status 0x4a
(probe496:isp0:0:496:0): cam completion status 0x4a
(probe496:isp0:0:496:0): cam completion status 0x4a
(probe497:isp0:0:497:0): cam completion status 0x4a
(probe497:isp0:0:497:0): cam completion status 0x4a
(probe498:isp0:0:498:0): cam completion status 0x4a
(probe498:isp0:0:498:0): cam completion status 0x4a
(probe499:isp0:0:499:0): cam completion status 0x4a
(probe499:isp0:0:499:0): cam completion status 0x4a
(probe500:isp0:0:500:0): cam completion status 0x4a
(probe500:isp0:0:500:0): cam completion status 0x4a
(probe501:isp0:0:501:0): cam completion status 0x4a
(probe501:isp0:0:501:0): cam completion status 0x4a
(probe502:isp0:0:502:0): cam completion status 0x4a
(probe502:isp0:0:502:0): cam completion status 0x4a
(probe503:isp0:0:503:0): cam completion status 0x4a
(probe503:isp0:0:503:0): cam completion status 0x4a
(probe6:isp0:0:6:0): cam completion status 0x4a
(probe6:isp0:0:6:0): cam completion status 0x4a
(probe25:isp0:0:25:0): cam completion status 0x4a
(probe25:isp0:0:25:0): cam completion status 0x4a
(probe62:isp0:0:62:0): cam completion status 0x4a
(probe62:isp0:0:62:0): cam completion status 0x4a
(probe74:isp0:0:74:0): cam completion status 0x4a
(probe74:isp0:0:74:0): cam completion status 0x4a
(probe80:isp0:0:80:0): cam completion status 0x4a
(probe80:isp0:0:80:0): cam completion status 0x4a
(probe109:isp0:0:109:0): cam completion status 0x4a
(probe109:isp0:0:109:0): cam completion status 0x4a
(probe131:isp0:0:131:0): cam completion status 0x4a
(probe131:isp0:0:131:0): cam completion status 0x4a
(probe147:isp0:0:147:0): cam completion status 0x4a
(probe147:isp0:0:147:0): cam completion status 0x4a
(probe206:isp0:0:206:0): cam completion status 0x4a
(probe206:isp0:0:206:0): cam completion status 0x4a
da0 at isp0 bus 0 target 0 lun 0
da0: <DGC RAID 5 0326> Fixed Direct Access SCSI-4 device 
da0: 100.000MB/s transfers
da0: Command Queueing Enabled
da0: 512000MB (1048576000 512 byte sectors: 255H 63S/T 65270C)
cd0 at umass-sim0 bus 0 target 0 lun 0
cd0: <Dell Virtual  CDROM 123> Removable CD-ROM SCSI-0 device 
cd0: 40.000MB/s transfers
cd0: Attempt to query device size failed: NOT READY, Medium not present
SMP: AP CPU #1 Launched!
SMP: AP CPU #2 Launched!
SMP: AP CPU #3 Launched!
SMP: AP CPU #6 Launched!
SMP: AP CPU #7 Launched!
SMP: AP CPU #5 Launched!
SMP: AP CPU #4 Launched!
WARNING: WITNESS option enabled, expect reduced performance.
WARNING: DIAGNOSTIC option enabled, expect reduced performance.
da1 at umass-sim1 bus 1 target 0 lun 0
da1: <Dell Virtual  Floppy 123> Removable Direct Access SCSI-0 device 
da1: 40.000MB/s transfers
da1: Attempt to query device size failed: NOT READY, Medium not present
Trying to mount root from ufs:/dev/mfid0s1a
bce0: link state changed to UP
mfi0: 3320 (277345023s/0x0008/info) - Battery charge complete

---------------------------------------------------------------

4. panic data and ddb output, also available at
http://noc.ntua.gr/~christia/san/7-STABLE.deadlock-ddb.out :

KKDDpanic: ufs_dirbad: /a: bad dir ino 3370734 at ofBBfset 0: mangled entry
cpuid = 1
KDB: enter: panic
[thread pid 768 tid 100104 ]
Stopped at      kdb_enter_why+0x3a:     movl    $0,kdb_why
db> trace
Tracing pid 768 tid 100104 td 0xc74c6690
kdb_enter_why(c0b3b0a1,c0b3b0a1,c0b5d0ec,ed2dd85c,1,...) at kdb_enter_why+0x3a
panic(c0b5d0ec,c70365c0,336eee,0,c0b5d164,...) at panic+0x136
ufs_dirbad(c9c40180,0,c0b5d164,0,ed2dd904,...) at ufs_dirbad+0x49
ufs_lookup(ed2dd94c,c0b6f08d,cb090160,cb090160,ed2ddba4,...) at ufs_lookup+0x377
VOP_CACHEDLOOKUP_APV(c0c0c1a0,ed2dd94c,ed2ddba4,ed2ddb90,c75dae00,...) at VOP_CACHEDLOOKUP_APV+0xc5
vfs_cache_lookup(ed2dd9d0,c0b471e4,cb090160,2,cb090160,...) at vfs_cache_lookup+0xd3
VOP_LOOKUP_APV(c0c0c1a0,ed2dd9d0,c74c6690,c0b46172,19b,...) at VOP_LOOKUP_APV+0xe5
lookup(ed2ddb7c,c0b46172,c6,bf,c703a72c,...) at lookup+0x59e
namei(ed2ddb7c,c74c6690,ed2dda74,246,c0c61ab8,...) at namei+0x35e
vn_open_cred(ed2ddb7c,ed2ddc78,1a4,c75dae00,c74871c8,...) at vn_open_cred+0x62
vn_open(ed2ddb7c,ed2ddc78,1a4,c74871c8,0,...) at vn_open+0x33
kern_open(c74c6690,8164100,0,a02,1a4,...) at kern_open+0xe7
open(c74c6690,ed2ddcfc,c,c0b40d50,c0beb278,...) at open+0x30
syscall(ed2ddd38) at syscall+0x2d3
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (5, FreeBSD ELF32, open), eip = 0x281a161b, esp = 0xbfbfe8cc, ebp = 0xbfbfe8f8 ---
db> ps
  pid  ppid  pgrp   uid   state   wmesg     wchan    cmd
  768   728   768     0  R+      CPU 1               bsdtar
  766   728   766     0  R+      CPU 4               bsdtar
  728   724   728     0  Ss+     ttyin    0xc68ce810 csh
  724   645   724     0  Ss      select   0xc0ca821c sshd
  720   711   720     0  S+      ttyin    0xc69a7810 csh
  719     1   719     0  Ss+     ttyin    0xc69db410 getty
  718     1   718     0  Ss+     ttyin    0xc69dc810 getty
  717     1   717     0  Ss+     ttyin    0xc69de410 getty
  716     1   716     0  Ss+     ttyin    0xc69d3810 getty
  715     1   715     0  Ss+     ttyin    0xc69d5810 getty
  714     1   714     0  Ss+     ttyin    0xc69d5c10 getty
  713     1   713     0  Ss+     ttyin    0xc69dc010 getty
  712     1   712     0  Ss+     ttyin    0xc69db010 getty
  711     1   711     0  Ss+     wait     0xc6feb2b8 login
  672     1   672     0  Ss      nanslp   0xc0c56844 cron
  666     1   666    25  Ss      pause    0xc7049b40 sendmail
  660     1   660     0  Ss      select   0xc0ca821c sendmail
  645     1   645     0  Ss      select   0xc0ca821c sshd
  527     1   527     0  Ss      select   0xc0ca821c syslogd
  473     1   473     0  Ss      select   0xc0ca821c devd
  143     1   143     0  Ss      pause    0xc704ab40 adjkerntz
   56     0     0     0  SL      sdflush  0xc0cb4668 [softdepflush]
   55     0     0     0  SL      syncer   0xc0c5666c [syncer]
   54     0     0     0  SL      vlruwt   0xc6fec000 [vnlru]
   53     0     0     0  RL      CPU 5               [bufdaemon]
   52     0     0     0  SL      pgzero   0xc0cb5220 [pagezero]
   51     0     0     0  SL      psleep   0xc0cb4e38 [vmdaemon]
   50     0     0     0  SL      psleep   0xc0cb4e00 [pagedaemon]
   49     0     0     0  SL      waiting_ 0xc0caa434 [sctp_iterator]
   48     0     0     0  WL                          [irq1: atkbd0]
   47     0     0     0  WL                          [swi0: sio]
   46     0     0     0  WL                          [irq15: ata1]
   45     0     0     0  WL                          [irq14: ata0]
   44     0     0     0  SL      usbevt   0xc68a8210 [usb4]
   43     0     0     0  SL      usbevt   0xc6994210 [usb3]
   42     0     0     0  SL      usbevt   0xc6977210 [usb2]
   41     0     0     0  SL      usbevt   0xc697c210 [usb1]
   40     0     0     0  WL                          [irq20: uhci1 uhci3]
   39     0     0     0  SL      usbtsk   0xc0c53e14 [usbtask-dr]
   38     0     0     0  SL      usbtsk   0xc0c53e00 [usbtask-hc]
   37     0     0     0  SL      usbevt   0xc692d210 [usb0]
   36     0     0     0  WL                          [irq21: uhci0 uhci+]
   35     0     0     0  WL                          [irq259: bce1]
   34     0     0     0  SL      ispf     0xc67e14d4 [isp1: fc_thrd]
   33     0     0     0  WL                          [irq258: isp1]
   32     0     0     0  SL      ispf     0xc67e06d4 [isp0: fc_thrd]
   31     0     0     0  RL                          [irq257: isp0]
   30     0     0     0  WL                          [irq16: mfi0]
   29     0     0     0  WL                          [irq256: bce0]
   28     0     0     0  WL                          [irq9: acpi0]
   27     0     0     0  SL      -        0xc685e900 [thread taskq]
   26     0     0     0  WL                          [swi5: +]
    9     0     0     0  SL      -        0xc685eb00 [kqueue taskq]
    8     0     0     0  SL      -        0xc685eb80 [acpi_task_2]
    7     0     0     0  SL      -        0xc685eb80 [acpi_task_1]
    6     0     0     0  SL      -        0xc685eb80 [acpi_task_0]
   25     0     0     0  WL                          [swi2: cambio]
    5     0     0     0  SL      ccb_scan 0xc0c24394 [xpt_thrd]
   24     0     0     0  WL                          [swi6: task queue]
   23     0     0     0  WL                          [swi6: Giant taskq]
   22     0     0     0  SL      -        0xc0c56674 [yarrow]
    4     0     0     0  SL      -        0xc0c5420c [g_down]
    3     0     0     0  RL      CPU 3               [g_up]
    2     0     0     0  SL      -        0xc0c54200 [g_event]
   21     0     0     0  WL                          [swi3: vm]
   20     0     0     0  WL                          [swi4: clock sio]
   19     0     0     0  WL                          [swi1: net]
   18     0     0     0  RL      CPU 0               [idle: cpu0]
   17     0     0     0  RL                          [idle: cpu1]
   16     0     0     0  RL      CPU 2               [idle: cpu2]
   15     0     0     0  RL                          [idle: cpu3]
   14     0     0     0  RL                          [idle: cpu4]
   13     0     0     0  RL                          [idle: cpu5]
   12     0     0     0  RL      CPU 6               [idle: cpu6]
   11     0     0     0  RL      CPU 7               [idle: cpu7]
    1     0     1     0  SLs     wait     0xc6793ae0 [init]
   10     0     0     0  SL      audit_wo 0xc0cb40a4 [audit]
    0     0     0     0  SLs     sched    0xc0c542c0 [swapper]
db> show pcpu
cpuid        = 1
curthread    = 0xc74c6690: pid 768 "bsdtar"
curpcb       = 0xed2ddd90
fpcurthread  = none
idlethread   = 0xc67a7460: pid 17 "idle: cpu1"
APIC ID      = 1
currentldt   = 0x50
spin locks held:
db> show allpcpu
Current CPU: 1

cpuid        = 0
curthread    = 0xc67a7230: pid 18 "idle: cpu0"
curpcb       = 0xe5424d90
fpcurthread  = none
idlethread   = 0xc67a7230: pid 18 "idle: cpu0"
APIC ID      = 0
currentldt   = 0x50
spin locks held:

cpuid        = 1
curthread    = 0xc74c6690: pid 768 "bsdtar"
curpcb       = 0xed2ddd90
fpcurthread  = none
idlethread   = 0xc67a7460: pid 17 "idle: cpu1"
APIC ID      = 1
currentldt   = 0x50
spin locks held:

cpuid        = 2
curthread    = 0xc6795000: pid 16 "idle: cpu2"
curpcb       = 0xe541ed90
fpcurthread  = none
idlethread   = 0xc6795000: pid 16 "idle: cpu2"
APIC ID      = 2
currentldt   = 0x50
spin locks held:

cpuid        = 3
curthread    = 0xc6797690: pid 3 "g_up"
curpcb       = 0xe6e36d90
fpcurthread  = none
idlethread   = 0xc6795230: pid 15 "idle: cpu3"
APIC ID      = 3
currentldt   = 0x50
spin locks held:

cpuid        = 4
curthread    = 0xc75c3690: pid 766 "bsdtar"
curpcb       = 0xed348d90
fpcurthread  = none
idlethread   = 0xc6795460: pid 14 "idle: cpu4"
APIC ID      = 4
currentldt   = 0x50
spin locks held:

cpuid        = 5
curthread    = 0xc6fea8c0: pid 53 "bufdaemon"
curpcb       = 0xeafead90
fpcurthread  = none
idlethread   = 0xc6795690: pid 13 "idle: cpu5"
APIC ID      = 5
currentldt   = 0x50
spin locks held:

cpuid        = 6
curthread    = 0xc67958c0: pid 12 "idle: cpu6"
curpcb       = 0xe5412d90
fpcurthread  = none
idlethread   = 0xc67958c0: pid 12 "idle: cpu6"
APIC ID      = 6
currentldt   = 0x50
spin locks held:

cpuid        = 7
curthread    = 0xc6795af0: pid 11 "idle: cpu7"
curpcb       = 0xe540fd90
fpcurthread  = none
idlethread   = 0xc6795af0: pid 11 "idle: cpu7"
APIC ID      = 7
currentldt   = 0x50
spin locks held:

db> show locks
db> show alllocks
Process 4 (g_down) thread 0xc6797460 (100015)
exclusive sleep mutex isp r = 0 (0xc67e0690) locked @ /usr/src/sys/cam/cam_periph.h:182
db> show lockedvnods
Locked vnodes

0xcb090160: tag ufs, type VDIR
    usecount 1, writecount 0, refcount 3 mountedhere 0
    flags ()
    v_object 0xcaa70c1c ref 0 pages 1
     lock type ufs: EXCL (count 1) by thread 0xc74c6690 (pid 768)#0 0xc077f6c0 at _lockmgr+0x560
#1 0xc099cb51 at ffs_lock+0x91
#2 0xc0aafcc5 at VOP_LOCK1_APV+0xa5
#3 0xc081f7b8 at _vn_lock+0xf8
#4 0xc0813c20 at vget+0x110
#5 0xc08029b7 at cache_lookup+0x4e7
#6 0xc0802aea at vfs_cache_lookup+0xaa
#7 0xc0ab0e65 at VOP_LOOKUP_APV+0xe5
#8 0xc0808d2e at lookup+0x59e
#9 0xc08098de at namei+0x35e
#10 0xc081f022 at vn_open_cred+0x62
#11 0xc081f513 at vn_open+0x33
#12 0xc081e177 at kern_open+0xe7
#13 0xc081e660 at open+0x30
#14 0xc0a98853 at syscall+0x2d3
#15 0xc0a7ea80 at Xint0x80_syscall+0x20

        ino 3370734, on dev da0s1d

0xcb5906e0: tag ufs, type VDIR
    usecount 0, writecount 0, refcount 2 mountedhere 0
    flags ()
    v_object 0xc9b65a2c ref 0 pages 0
     lock type ufs: EXCL (count 1) by thread 0xc6fea8c0 (pid 53)#0 0xc077f6c0 at _lockmgr+0x560
#1 0xc099cb51 at ffs_lock+0x91
#2 0xc0aafcc5 at VOP_LOCK1_APV+0xa5
#3 0xc081f7b8 at _vn_lock+0xf8
#4 0xc07ff5d8 at flushbufqueues+0x568
#5 0xc07ff8f0 at buf_daemon+0xc0
#6 0xc0770ea8 at fork_exit+0xb8
#7 0xc0a7ea90 at fork_trampoline+0x8

        ino 24168915, on dev da0s1d
db> trace 768
Tracing pid 768 tid 100104 td 0xc74c6690
kdb_enter_why(c0b3b0a1,c0b3b0a1,c0b5d0ec,ed2dd85c,1,...) at kdb_enter_why+0x3a
panic(c0b5d0ec,c70365c0,336eee,0,c0b5d164,...) at panic+0x136
ufs_dirbad(c9c40180,0,c0b5d164,0,ed2dd904,...) at ufs_dirbad+0x49
ufs_lookup(ed2dd94c,c0b6f08d,cb090160,cb090160,ed2ddba4,...) at ufs_lookup+0x377
VOP_CACHEDLOOKUP_APV(c0c0c1a0,ed2dd94c,ed2ddba4,ed2ddb90,c75dae00,...) at VOP_CACHEDLOOKUP_APV+0xc5
vfs_cache_lookup(ed2dd9d0,c0b471e4,cb090160,2,cb090160,...) at vfs_cache_lookup+0xd3
VOP_LOOKUP_APV(c0c0c1a0,ed2dd9d0,c74c6690,c0b46172,19b,...) at VOP_LOOKUP_APV+0xe5
lookup(ed2ddb7c,c0b46172,c6,bf,c703a72c,...) at lookup+0x59e
namei(ed2ddb7c,c74c6690,ed2dda74,246,c0c61ab8,...) at namei+0x35e
vn_open_cred(ed2ddb7c,ed2ddc78,1a4,c75dae00,c74871c8,...) at vn_open_cred+0x62
vn_open(ed2ddb7c,ed2ddc78,1a4,c74871c8,0,...) at vn_open+0x33
kern_open(c74c6690,8164100,0,a02,1a4,...) at kern_open+0xe7
open(c74c6690,ed2ddcfc,c,c0b40d50,c0beb278,...) at open+0x30
syscall(ed2ddd38) at syscall+0x2d3
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (5, FreeBSD ELF32, open), eip = 0x281a161b, esp = 0xbfbfe8cc, ebp = 0xbfbfe8f8 ---
db> trace 55 
Tracing pid 55 tid 100054 td 0xc6fea460
sched_switch(c6fea460,0,1,17c,29bc06da,...) at sched_switch+0x49b
mi_switch(1,0,c0b3ec35,1cb,c6fea460,...) at mi_switch+0x223
sleepq_switch(c6fea460,0,c0b3ec35,243,68,...) at sleepq_switch+0x182
sleepq_wait(c0c5666c,0,c0b3ba36,d8,0,...) at sleepq_wait+0x60
_sleep(c0c5666c,c0ca8988,68,c0b47461,0,...) at _sleep+0x369
sched_sync(0,eaff0d38,c0b37584,31c,c6febae0,...) at sched_sync+0x752
fork_exit(c0814220,0,eaff0d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xeaff0d70, ebp = 0 ---
db> trace 4 
Tracing pid 4 tid 100015 td 0xc6797460
sched_switch(c6797460,0,1,17c,5512f4a0,...) at sched_switch+0x49b
mi_switch(1,0,c0b3ec35,1cb,4c,...) at mi_switch+0x223
sleepq_switch(c6797460,0,c0b3ec35,266,4c,...) at sleepq_switch+0x182
sleepq_timedwait(c0c5420c,64,c0b32247,0,0,...) at sleepq_timedwait+0x68
_sleep(c0c5420c,c0c54128,24c,c0b32247,64,...) at _sleep+0x345
g_io_schedule_down(c6797460,0,c0b339ab,74,0,...) at g_io_schedule_down+0x6b
g_down_procbody(0,e6e39d38,c0b37584,31c,c67eb828,...) at g_down_procbody+0x8d
fork_exit(c073d540,0,e6e39d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe6e39d70, ebp = 0 ---
db> continue
::Uptime: 7m21s
  ssttPhysical memory: 3310 MB
Dumping 225 MB:aacckk  bbaacckkttrraaccee::

ddbb__ttrraaccee__sseellff__wwrrappppeerr((cc00bb33ee003388,,eeadf3e4a8ab400c,,cc00881100333333,,00,,eeadf3e4a8ab400c,,......))  aatt  ddbb__ttrraaccee__sseellff__wwrraappppeerr++00xx2266

kkddbb__bbaacckkttrraaccee((00,,ee



                                  4a8ab400c,,cc00aaaacc99bb55,,eeadf3e4a8ab510c,,
                                                                                 c
                                                                                  b4623806dec00,,......))  aatt  kkddbb__bbaacckkttrraaccee++00xx2299

vv
  f


   bbaaddlloocckk((c00cc00cc11aa00,,eeadf3e4a8ab510c,,cc00cc223388aa00,,ccbb4623806dc00,,00))  aatt  vvffss__bbaaddlloocckk++00xx2233

aasssseerrtt__vvoopp__lloocckkeedd((ccbb4623806dec00,,cc
                                                        )  aatt  aasssseerrtt__vvoopp__lloocckkeedd++00xx5500

vvoopp__lloocckk__ppoosstt((eeda3f4e8aba7a8c,,00,,11020,21,21,0c0b242c8b66e300,d.c.0.,). .a.t)  vaotp _vloopc_kl_opcoks_tp+o0sxt3+60
                                                                                                                                    x


P                                                                                                                                    O
 _VLOOPC_KL1O_CAKP1V_(AcP0Vc(0cc01ca00c,1eaa0f,eeada3a48,bc708b,4c709bb427,93b222,,3c2b24,2c8b76b340,e.9.4.,). .a.t)  VaOtP _VLOOPC_KL1O_CAKP1V_x0PxVc+10
nc_1v
 __lvonc_kl(occbk4(2c8b66e300,d1c20,,c160f0e2a,
                                               c
                                                05,cc306b9404,7cc04b,487a93b,2.,.2.4)2 ,a.t. ._)v na_tl o_
+                                                                                                         lxofc8k
h0fxlfu8s
0bvunf_qwureiutees((cc760f5e2c62d48,,e0d,3c40bc46407,cc47,58d0abe,0604,,0.,.c.7)5 ca3t6 9f0l,u.s.h.b)u faqtu uvens_+w0rxi5t6e8+
dxb1u6fa_
 adeomfoinl(e0w,reiatfee(aedd3384,8cc06b03,7f5f8f4f,f3f1fcf,,cf6fffefcf2bf8f,,.0.,.c)7 0a5t2 6bdu4f,_.d.a.e)m oant+ 0dxocf0
                                                                                                                           i
                                                                                                                            efworrikt_ee+x0ixfr843_0w,r0i,teeavf(eca7d53c83)6 9a0t, 4f,oerdk3_4e8xci6t0+,08x1b281
o0f0o,r1k9_at,r.a.m.p)o laitn ek(e)r na_tw rfiotrekv_+t0rxa5m8p
5lwirniet+e0(xc87
4c-3-6-9 0t,readp3 408,c feci,pc ,=c 00b,4 0edscp5 ,=c 00bxeeba2f6e0a,d.7.0.,)  eabtp  w=r i0t e-+-0-x
OfV
tPs_yLsOcCaKl:l (0exdc3b4482d8368e)0  aits  snyostc allolc+k0exd2 db3u
y Xsihnotu0lxd8 0b_es
 sKc
+   aBl:l (e)n taetr :X ilnotc0kx 8v0i_oslyastciaolnl
 [thread pid 53 tid 100052 ]
Stopped at      kdb_enter_why+0x3a:     movl    $0,kdb_why
db> <Enter>
 xD2B0:
 s-t-a-c ks ybsaccakltlr a(c4:,
 Fdrbe_etBrSaDc eE_LsFe3l2f,_ wwrraiptpee)r,( c0ibp3 e=0 308x,2e8a1fbe1a793e3c,, ce0s8p1 0=3 303x,b0f,befaef9e2ac9,e ce,b.p. .=)  0axtb fdbbf_etlr7a8ce-_-s-e
6fV_OwPr_aLpOpCeKr:+ 00xx2c6b
,3k0ddbc_0baicsk tnroatc el(o0c,keeadf ebau9te cs,hco0ualadc 9bbe5
deKaDfBe:a 9efnct,ecrb:4 2l8o6cek ,v.i.o.l)a taito nk
 [thread pid 766 tid 100137 ]
Stopped at      kdb_enter_why+0x3a:     movl    $0,kdb_why
db> <Enter>
r_DbBa:c ksttraaccke +b0axc2k9t
bavcfes:_
 addbl_otcrka(cce0_cs0ecl1f

c                          _w,reaapfpeear9(fcc0,bc30ec023388,ae0d,3c4b84a2d8c6,ec00,801)0 3a3t3 ,vcf7s5_cb3a6d9l0o,cekd+304x82a3d
,,a.s.s.e)r ta_tv odpb__ltorcakceed_(scebl4f2_8w6rea0p,pce0rb+60fx22465
cck0dbb3_9bca4c9k,tdraa9c3e3(0c1705,c43,6.9.0.,)e da3t4 8aasdsce,rct0_avaocp9_bl5o,cekde3d4+80axe5c0,
 bV6O3P0_dScT0R,A.T.E.G)Y _aAtP Vk(dcb0_c
                                         0
,                                         c1kat0r,aecaef+e0axa239c
 cvbf4
)     2_b7abd4l,occ0kc(2c30cca00c,1cab04,2e8d63e408,a.e.c.,)c 0act2 3V8OaP0_,ScTbR6A3T0EdGcY0_,AcP7V5+c03x699b0
  bautf svtfrsa_tbeag
,                    yl(occbk4+20x72e3c
 daas9s3e3r0t1_0v
                 ,

c                 0_0e,lco0cckae8d6(3c0b,6d3a09d3c300,7c00,b.6.f.4)8 0a,te db3u4f8sbt2rca,ce0g8y1+20dx6601,
e0bcu0fcw1rai0t,e.(.d.a)9 3a3t0 1a0s,seearfte_avao9p0_,ecl0oacakfecde+10,xe5a3f
 aV
   OaPc_,W0R,I.T.E._)A PaVt(
                             
c                            0ucf0wcr1iat0e,+e0dx314981b
 4v,fcs7_5bci3o6_9a0w,rci0tbe4(7d9ab923,320420,,.0.,.c)0 ba4t4 7VcO4P,_8WaR4I,T8Ea3A,P.V.+.0)x 9a8t
3vvfns__wbriiot_ea(wcr7i0t5e2+60dx43,2e8d
 48lcu6s0h,bcu7f5qduaeeu0e0s,(0c,6cf7e5cc23b689,00,,.
                                                     .




                                                      4a7tc 4v,n8_0wbr,
t                                                                      6t4e,+.0.x.2)3 1a
f dfolfuislhebwurfiqtuee(ueeds3+408xc5690c,
 fbfufff_fd
           f

b           fonf(f0f,fefaff,e0a,dc3780,5c206bd347,5.8.4.,)3 1act, cd6offeicl2ebw8r,i.t.e.+)0 xa9t5 
5ukfe_rdna_ewmroint+e0vx(c c07
02cf3o6r9k0_,e4x,ietd(3c4087cf6f08,3801,201,4e0a0f,e1a9da3,8.). .a)t  afto rkke_renx_iwtr+i0txebv8+
kxf5o8r
0_wtrriatmep(ocl7i5nce3(6)9 0a,te df3o4r8kc_ftcr,acm,pco0lbi4n0ed+c05x,8c
=b-e-b-2 6t0r,a.p. .0),  aeti pw r=i t0e,+ 0exs4pf 
c s0yxsecaaflela(d1e7d03,04 8edb3p8 )=  a0t  -s-y-s
EaVlOlP+_0SxT2RdA3T
0GXYi:n t00xxc8b04_2s8y6sec0a lils( )n oatt  lXoicnkte0dx 8b0u_ts ysshcoaullld+ 0bxe2
K
LD-B-:-  esnytsecra:l ll o(c4k,  vFiroeleaBtSiDo nE
 [thread pid 53 tid 100052 ]
Stopped at      kdb_enter_why+0x3a:     movl    $0,kdb_why
db> panic
[no answer... let's power cycle the server once more]


-- 
Panagiotis J. Christias    Network Management Center
P.Christias at noc.ntua.gr    National Technical Univ. of Athens, GREECE


More information about the freebsd-stable mailing list