kern/115606: Panic while using mpt controller

Fri Aug 17 09:50:01 PDT 2007

>Number:         115606
>Category:       kern
>Synopsis:       Panic while using mpt controller
>Confidential:   no
>Severity:       serious
>Priority:       low
>Responsible:    freebsd-bugs
>State:          open
>Quarter:        
>Keywords:       
>Date-Required:
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Fri Aug 17 16:50:00 GMT 2007
>Closed-Date:
>Last-Modified:
>Originator:     Paolo Pisati
>Release:        7.0-CURRENT (but the same happens on 6.x)
>Organization:
Tomato Interactive
>Environment:
FreeBSD ferrettolo.tomato.lan 7.0-CURRENT FreeBSD 7.0-CURRENT #1: Fri Aug 17 14:54:35 UTC 2007     piso at ferrettolo.tomato.lan:/usr/obj/usr/src/sys/GENERIC  i386
>Description:
While doing some i/o work on a disk attached to an mpt controller,
FreeBSD 6.x/7.x panic with the following message:

ferrettolo# kgdb /usr/obj/usr/src/sys/GENERIC/kernel.debug /var/crash/vmcore.1
[GDB will not be able to debug user-mode threads: /usr/lib/libthread_db.so: Undefined symbol "ps_pglobal_lookup"]
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "i386-marcel-freebsd".

Unread portion of the kernel message buffer:
mpt0: mpt_cam_event: 0xc0
mpt0: Unhandled Event Notify Frame. Event 0xc0afa2c0 (ACK not required).
mpt0: completing timedout/aborted req 0xc406d450:5896
panic: Bad link elm 0xc406d450 next->prev != elm
cpuid = 0
KDB: enter: panic
Dumping 1023 MB (2 chunks)
  chunk 0: 1MB (157 pages) ... ok
  chunk 1: 1023MB (261868 pages) 1007 991 975 959 943 927 911 895 879 863 847 831 815 799 783 767 751 735 719 703 687 671 655 639 623 607 591 575 559 543 527 511 495 479 463 447 431 415 399 383 367 351 335 319 303 287 271 255 239 223 207 191 175 159 143 127 111 95 79 63 47 31 15

#0  doadump () at pcpu.h:195
195     pcpu.h: No such file or directory.
        in pcpu.h
(kgdb) bt
#0  doadump () at pcpu.h:195
#1  0xc048cb29 in db_fncall (dummy1=-465581460, dummy2=0, dummy3=524358, dummy4=0xe43fc9d8 "ÐíHÀ") at /usr/src/sys/ddb/db_command.c:486
#2  0xc048d095 in db_command_loop () at /usr/src/sys/ddb/db_command.c:401
#3  0xc048e805 in db_trap (type=3, code=0) at /usr/src/sys/ddb/db_main.c:222
#4  0xc0773066 in kdb_trap (type=3, code=0, tf=0xe43fcb80) at /usr/src/sys/kern/subr_kdb.c:502
#5  0xc09fd15b in trap (frame=0xe43fcb80) at /usr/src/sys/i386/i386/trap.c:621
#6  0xc09e2b7b in calltrap () at /usr/src/sys/i386/i386/exception.s:139
#7  0xc07731e2 in kdb_enter (msg=0xc0a93413 "panic") at cpufunc.h:60
#8  0xc074c3e4 in panic (fmt=0xc0a49d68 "Bad link elm %p next->prev != elm") at /usr/src/sys/kern/kern_shutdown.c:547
#9  0xc06065c5 in mpt_scsi_reply_handler (mpt=0xc4066000, req=0xc406d450, reply_desc=65820, reply_frame=0xe43fcc58) at /usr/src/sys/dev/mpt/mpt_cam.c:2661
#10 0xc06033be in mpt_complete_request_chain (mpt=0xc4066000, chain=0xc40663c4, iocstatus=8) at /usr/src/sys/dev/mpt/mpt.c:849
#11 0xc0606734 in mpt_cam_ioc_reset (mpt=0xc4066000, type=0) at /usr/src/sys/dev/mpt/mpt_cam.c:3041
#12 0xc0604acf in mpt_reset (mpt=0xc4066000, reinit=1) at /usr/src/sys/dev/mpt/mpt.c:1145
#13 0xc0607bf2 in mpt_recovery_thread (arg=0xc4066000) at /usr/src/sys/dev/mpt/mpt_cam.c:4176
#14 0xc072e258 in fork_exit (callout=0xc0607920 <mpt_recovery_thread>, arg=0xc4066000, frame=0xe43fcd38) at /usr/src/sys/kern/kern_fork.c:797
#15 0xc09e2bf0 in fork_trampoline () at /usr/src/sys/i386/i386/exception.s:205

dmesg:
ferrettolo# dmesg 
Copyright (c) 1992-2007 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
        The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 7.0-CURRENT #1: Fri Aug 17 14:54:35 UTC 2007
    piso at ferrettolo.tomato.lan:/usr/obj/usr/src/sys/GENERIC
WARNING: WITNESS option enabled, expect reduced performance.
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: Intel(R) Xeon(TM) CPU 2.80GHz (2790.72-MHz 686-class CPU)
  Origin = "GenuineIntel"  Id = 0xf27  Stepping = 7
  Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
  Features2=0x4400<CNXT-ID,xTPR>
  Logical CPUs per core: 2
real memory  = 1073659904 (1023 MB)
avail memory = 1037049856 (989 MB)
ACPI APIC Table: <IBM    SERONYXP>
FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs
 cpu0 (BSP): APIC ID:  0
 cpu1 (AP): APIC ID:  1
MADT: Forcing active-low polarity and level trigger for SCI
ioapic2 <Version 1.1> irqs 32-47 on motherboard
ioapic1 <Version 1.1> irqs 16-31 on motherboard
ioapic0 <Version 1.1> irqs 0-15 on motherboard
kbd1 at kbdmux0
ath_hal: 0.9.20.3 (AR5210, AR5211, AR5212, RF5111, RF5112, RF2413, RF5413)
acpi0: <IBM SERONYXP> on motherboard
acpi0: [ITHREAD]
acpi0: Power Button (fixed)
acpi0: reservation of 460, 2 (4) failed
Timecounter "ACPI-safe" frequency 3579545 Hz quality 850
acpi_timer0: <32-bit timer at 3.579545MHz> port 0x488-0x48b on acpi0
cpu0: <ACPI CPU> on acpi0
p4tcc0: <CPU Frequency Thermal Control> on cpu0
cpu1: <ACPI CPU> on acpi0
p4tcc1: <CPU Frequency Thermal Control> on cpu1
pcib0: <ACPI Host-PCI bridge> on acpi0
pci0: <ACPI PCI bus> on pcib0
vgapci0: <VGA-compatible display> port 0x2200-0x22ff mem 0xfd000000-0xfdffffff,0xfebff000-0xfebfffff irq 26 at device 1.0 on pci0
atapci0: <ServerWorks CSB5 UDMA100 controller> port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0x700-0x70f at device 15.1 on pci0
ata0: <ATA channel 0> on atapci0
ata0: [ITHREAD]
ata1: <ATA channel 1> on atapci0
ata1: [ITHREAD]
ohci0: <OHCI (generic) USB controller> mem 0xfebfe000-0xfebfefff irq 11 at device 15.2 on pci0
ohci0: [GIANT-LOCKED]
ohci0: [ITHREAD]
usb0: OHCI version 1.0, legacy support
usb0: SMM does not respond, resetting
usb0: <OHCI (generic) USB controller> on ohci0
usb0: USB revision 1.0
uhub0: <(0x1166) OHCI root hub, class 9/0, rev 1.00/1.00, addr 1> on usb0
uhub0: 4 ports with 4 removable, self powered
isab0: <PCI-ISA bridge> at device 15.3 on pci0
isa0: <ISA bus> on isab0
pcib1: <ACPI Host-PCI bridge> on acpi0
pci1: <ACPI PCI bus> on pcib1
mpt0: <LSILogic 1030 Ultra4 Adapter> port 0x2300-0x23ff mem 0xfbff0000-0xfbffffff,0xfbfe0000-0xfbfeffff irq 22 at device 1.0 on pci1
mpt0: [ITHREAD]
mpt0: MPI Version=1.2.6.0
mpt0: Capabilities: ( RAID-1 SAFTE )
mpt0: 1 Active Volume (1 Max)
mpt0: 2 Hidden Drive Members (3 Max)
pcib2: <ACPI Host-PCI bridge> on acpi0
pci2: <ACPI PCI bus> on pcib2
pci2:1:0: bad VPD cksum, remain 14
bge0: <Broadcom NetXtreme Gigabit Ethernet Controller, ASIC rev. 0x1002> mem 0xf9ff0000-0xf9ffffff irq 24 at device 1.0 on pci2
miibus0: <MII bus> on bge0
brgphy0: <BCM5703 10/100/1000baseTX PHY> PHY 1 on miibus0
brgphy0:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-FDX, auto
bge0: Ethernet address: 00:11:25:3e:95:1e
bge0: [ITHREAD]
pci2:2:0: bad VPD cksum, remain 14
bge1: <Broadcom NetXtreme Gigabit Ethernet Controller, ASIC rev. 0x1002> mem 0xf9fe0000-0xf9feffff irq 25 at device 2.0 on pci2
miibus1: <MII bus> on bge1
brgphy1: <BCM5703 10/100/1000baseTX PHY> PHY 1 on miibus1
brgphy1:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-FDX, auto
bge1: Ethernet address: 00:11:25:3e:95:1f
bge1: [ITHREAD]
fdc0: <floppy drive controller> port 0x3f0-0x3f5 irq 6 drq 2 on acpi0
fdc0: [FILTER]
fd0: <1440-KB 3.5" drive> on fdc0 drive 0
pmtimer0 on isa0
orm0: <ISA Option ROMs> at iomem 0xc0000-0xc7fff,0xc8000-0xc97ff,0xc9800-0xcd7ff pnpid ORM0000 on isa0
atkbdc0: <Keyboard controller (i8042)> at port 0x60,0x64 on isa0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
atkbd0: [ITHREAD]
psm0: <PS/2 Mouse> irq 12 on atkbdc0
psm0: [GIANT-LOCKED]
psm0: [ITHREAD]
psm0: model IntelliMouse, device ID 3
ppc0: parallel port not found.
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
sio0: configured irq 4 not in bitmap of probed irqs 0
sio0: port may not be enabled
sio0: configured irq 4 not in bitmap of probed irqs 0
sio0: port may not be enabled
sio0 at port 0x3f8-0x3ff irq 4 flags 0x10 on isa0
sio0: type 8250 or not responding
sio0: [FILTER]
sio1: configured irq 3 not in bitmap of probed irqs 0
sio1: port may not be enabled
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
Timecounters tick every 1.000 msec
mpt0:vol0(mpt0:0ad2: 98166MB <IC35L100AVVA07 0 VA5OA52A> at ata1-master UDMA100
:0): Settings ( Hot-Plug-Spares )
mpt0:vol0(mpt0:0:0): Using Spare Pool: 0
mpt0:vol0(mpt0:0:0): 2 Members:
      (mpt0:1:0:0): Primary Online
      (mpt0:1:1:0): Secondary Online
mpt0:vol0(mpt0:0:0): RAID-1 - Optimal
mpt0:vol0(mpt0:0:0): Status ( Enabled )
(mpt0:vol0:0): Physical (mpt0:0:0:0), Pass-thru (mpt0:1:0:0)
(mpt0:vol0:0): Online
(mpt0:vol0:1): Physical (mpt0:0:1:0), Pass-thru (mpt0:1:1:0)
(mpt0:vol0:1): Online
uma_zalloc_arg: zone "1024" with the following non-sleepable locks held:
exclusive sleep mutex mpt r = 0 (0xc4066004) locked @ /usr/src/sys/dev/mpt/mpt_raid.c:671
KDB: stack backtrace:
db_trace_self_wrapper(c0a95e89,e449ac44,c0782a3d,c0a9624c,e449ac58,...) at db_trace_self_wrapper+0x26
kdb_backtrace(c0a9624c,e449ac58,4,1,0,...) at kdb_backtrace+0x29
witness_warn(5,0,c0ab40fa,c0a493c8,142e,...) at witness_warn+0x1cd
uma_zalloc_arg(c146d5a0,0,2,2,c4066000,...) at uma_zalloc_arg+0x34
malloc(260,c0b43880,2,29f,e449acf8,...) at malloc+0xd2
mpt_raid_thread(c4066000,e449ad38,c0a8fda1,315,c4075ab0,...) at mpt_raid_thread+0xfb
fork_exit(c0613610,c4066000,e449ad38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe449ad70, ebp = 0 ---
acd0: DVDROM <IDE DVD-ROM 16X/VER 1005> at ata1-slave UDMA33
Waiting 5 seconds for SCSI devices to settle
(probe0:mpt0:1:0:0): Vendor Specific Command. CDB: 12 0 0 0 a4 0 
(probe0:mpt0:1:0:0): CAM Status: SCSI Status Error
(probe0:mpt0:1:0:0): SCSI Status: Check Condition
(probe0:mpt0:1:0:0): ABORTED COMMAND asc:48,0
(probe0:mpt0:1:0:0): Initiator detected error message received
(probe0:mpt0:1:0:0): Retrying Command (per Sense Data)
(probe0:mpt0:1:0:0): Vendor Specific Command. CDB: 12 0 0 0 a4 0 
(probe0:mpt0:1:0:0): CAM Status: SCSI Status Error
(probe0:mpt0:1:0:0): SCSI Status: Check Condition
(probe0:mpt0:1:0:0): ABORTED COMMAND asc:48,0
(probe0:mpt0:1:0:0): Initiator detected error message received
(probe0:mpt0:1:0:0): Retrying Command (per Sense Data)
(probe0:mpt0:1:0:0): Vendor Specific Command. CDB: 12 0 0 0 a4 0 
(probe0:mpt0:1:0:0): CAM Status: SCSI Status Error
(probe0:mpt0:1:0:0): SCSI Status: Check Condition
(probe0:mpt0:1:0:0): ABORTED COMMAND asc:48,0
(probe0:mpt0:1:0:0): Initiator detected error message received
(probe0:mpt0:1:0:0): Retrying Command (per Sense Data)
ses0 at mpt0 bus 0 target 8 lun 0
ses0: <IBM 25P3495a S320  1 1> Fixed Processor SCSI-2 device 
ses0: 3.300MB/s transfers
ses0: SAF-TE Compliant Device
da0 at mpt0 bus 0 target 0 lun 0
da0: <LSILOGIC 1030 IM 1000> Fixed Direct Access SCSI-2 device 
da0: 320.000MB/s transfers (160.000MHz, offset 80, 16bit)
da0: Command Queueing Enabled
da0: 139898MB (286511105 512 byte sectors: 255H 63S/T 17834C)
SMP: AP CPU #1 Launched!
WARNING: WITNESS option enabled, expect reduced performance.
Trying to mount root from ufs:/dev/ad2s1a
WARNING: / was not properly dismounted
WARNING: /tmp was not properly dismounted
WARNING: /usr was not properly dismounted
WARNING: /var was not properly dismounted
bge0: link state changed to UP

Notice the WITNESS warning:

uma_zalloc_arg: zone "1024" with the following non-sleepable locks held:
exclusive sleep mutex mpt r = 0 (0xc4066004) locked @ /usr/src/sys/dev/mpt/mpt_raid.c:671
KDB: stack backtrace:
db_trace_self_wrapper(c0a95e89,e449ac44,c0782a3d,c0a9624c,e449ac58,...) at db_trace_self_wrapper+0x26
kdb_backtrace(c0a9624c,e449ac58,4,1,0,...) at kdb_backtrace+0x29
witness_warn(5,0,c0ab40fa,c0a493c8,142e,...) at witness_warn+0x1cd
uma_zalloc_arg(c146d5a0,0,2,2,c4066000,...) at uma_zalloc_arg+0x34
malloc(260,c0b43880,2,29f,e449acf8,...) at malloc+0xd2
mpt_raid_thread(c4066000,e449ad38,c0a8fda1,315,c4075ab0,...) at mpt_raid_thread+0xfb
fork_exit(c0613610,c4066000,e449ad38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe449ad70, ebp = 0 ---

Hw:
IBM X335 with SCSI U320
FreeBSD 4.x works flawlessly on the same hw.

You can grab kernel.debug and vmcore.1 this way:

fetch http://people.tomato.it/piso/vmcore.1.gz
fetch http://people.tomato.it/piso/kernel.debug.gz

>How-To-Repeat:
Mount a disk on an mpt controller, copy /usr/src over it and enjoy 
the panic... :)
>Fix:

>Release-Note:
>Audit-Trail:
>Unformatted: