i386/129602: ahd(4) gets confused and wedges SCSI bus

Fri Dec 12 14:20:01 PST 2008

>Number:         129602
>Category:       i386
>Synopsis:       ahd(4) gets confused and wedges SCSI bus
>Confidential:   no
>Severity:       critical
>Priority:       medium
>Responsible:    freebsd-i386
>State:          open
>Quarter:        
>Keywords:       
>Date-Required:
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Fri Dec 12 22:20:00 UTC 2008
>Closed-Date:
>Last-Modified:
>Originator:     Garrett Wollman
>Release:        FreeBSD 7.1-RC1 i386
>Organization:
MIT Computer Science & Artificial Intelligence Laboratory
>Environment:
System: FreeBSD newsswitch.csail.mit.edu 7.1-RC1 FreeBSD 7.1-RC1 #1: Fri Dec 12 11:04:19 EST 2008 root@:/usr/obj/usr/src/sys/NEWSSWITCH i386

>Description:

ahd(4) driver gets confused and eventually wedges SCSI bus.  This is
accompanied by a very long state dump from the driver, and happens during
boot and during periods of heavy disk usage.  Eventually all disks in the
system time out and the machine either panics or freezes depending on what
it was doing.  When it's in this state, one of the drive activity lights
is on solid.

Here's the contents of the message buffer at bootup.  Note that the first
error occurs during the initial probe for disks.

Copyright (c) 1992-2008 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
	The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 7.1-RC1 #1: Fri Dec 12 11:04:19 EST 2008
    root@:/usr/obj/usr/src/sys/NEWSSWITCH
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: Intel(R) Xeon(TM) CPU 2.66GHz (2665.92-MHz 686-class CPU)
  Origin = "GenuineIntel"  Id = 0xf29  Stepping = 9
  Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
  Features2=0x4400<CNXT-ID,xTPR>
  Logical CPUs per core: 2
real memory  = 2146959360 (2047 MB)
avail memory = 2101248000 (2003 MB)
ACPI APIC Table: <PTLTD  	 APIC  >
ioapic0 <Version 2.0> irqs 0-23 on motherboard
ioapic1 <Version 2.0> irqs 24-47 on motherboard
ioapic2 <Version 2.0> irqs 48-71 on motherboard
ioapic3 <Version 2.0> irqs 72-95 on motherboard
ioapic4 <Version 2.0> irqs 96-119 on motherboard
ichwd module loaded
smbios0: <System Management BIOS> at iomem 0xf6810-0xf682e on motherboard
smbios0: Version: 2.31
acpi0: <PTLTD   RSDT> on motherboard
acpi0: [ITHREAD]
acpi0: Power Button (fixed)
Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x1008-0x100b on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
pci0: <unknown> at device 0.1 (no driver attached)
pcib1: <ACPI PCI-PCI bridge> at device 2.0 on pci0
pci1: <ACPI PCI bus> on pcib1
pcib2: <ACPI PCI-PCI bridge> at device 29.0 on pci1
pci2: <ACPI PCI bus> on pcib2
pcib3: <ACPI PCI-PCI bridge> at device 31.0 on pci1
pci3: <ACPI PCI bus> on pcib3
em0: <Intel(R) PRO/1000 Network Connection 6.9.6> port 0x3000-0x303f mem 0xfc200000-0xfc21ffff irq 28 at device 2.0 on pci3
em0: [FILTER]
em0: Ethernet address: 00:30:48:2b:84:16
em1: <Intel(R) PRO/1000 Network Connection 6.9.6> port 0x3040-0x307f mem 0xfc220000-0xfc23ffff irq 29 at device 2.1 on pci3
em1: [FILTER]
em1: Ethernet address: 00:30:48:2b:84:17
pcib4: <ACPI PCI-PCI bridge> at device 3.0 on pci0
pci4: <ACPI PCI bus> on pcib4
pcib5: <ACPI PCI-PCI bridge> at device 29.0 on pci4
pci5: <ACPI PCI bus> on pcib5
pcib6: <ACPI PCI-PCI bridge> at device 31.0 on pci4
pci6: <ACPI PCI bus> on pcib6
ahd0: <Adaptec AIC7902 Ultra320 SCSI adapter> port 0x4400-0x44ff,0x4000-0x40ff mem 0xfc400000-0xfc401fff irq 76 at device 2.0 on pci6
ahd0: [ITHREAD]
aic7902: Ultra320 Wide Channel A, SCSI Id=7, PCI-X 101-133Mhz, 512 SCBs
ahd1: <Adaptec AIC7902 Ultra320 SCSI adapter> port 0x4c00-0x4cff,0x4800-0x48ff mem 0xfc402000-0xfc403fff irq 77 at device 2.1 on pci6
ahd1: [ITHREAD]
aic7902: Ultra320 Wide Channel B, SCSI Id=7, PCI-X 101-133Mhz, 512 SCBs
pci0: <serial bus, USB> at device 29.0 (no driver attached)
pci0: <serial bus, USB> at device 29.1 (no driver attached)
pci0: <serial bus, USB> at device 29.2 (no driver attached)
pcib7: <ACPI PCI-PCI bridge> at device 30.0 on pci0
pci7: <ACPI PCI bus> on pcib7
vgapci0: <VGA-compatible display> port 0x5000-0x50ff mem 0xfd000000-0xfdffffff,0xfc500000-0xfc500fff irq 16 at device 1.0 on pci7
isab0: <PCI-ISA bridge> at device 31.0 on pci0
isa0: <ISA bus> on isab0
pci0: <mass storage, ATA> at device 31.1 (no driver attached)
ichsmb0: <Intel 82801CA (ICH3) SMBus controller> port 0x1100-0x111f at device 31.3 on pci0
ichsmb0: [GIANT-LOCKED]
ichsmb0: [ITHREAD]
smbus0: <System Management Bus> on ichsmb0
smb0: <SMBus generic I/O> on smbus0
acpi_button0: <Power Button> on acpi0
atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
atkbd0: [GIANT-LOCKED]
atkbd0: [ITHREAD]
sio0: <16550A-compatible COM port> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
sio0: type 16550A, console
sio0: [FILTER]
sio1: <16550A-compatible COM port> port 0x2f8-0x2ff irq 3 on acpi0
sio1: type 16550A
sio1: [FILTER]
fdc0: <floppy drive controller> port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on acpi0
fdc0: [FILTER]
fd0: <1440-KB 3.5" drive> on fdc0 drive 0
cpu0: <ACPI CPU> on acpi0
ichwd0: <Intel 82801CA watchdog timer> on isa0
ichwd0: Intel 82801CA watchdog timer (ICH3 or equivalent)
orm0: <ISA Option ROMs> at iomem 0xc0000-0xc7fff,0xc8000-0xc8fff,0xe0000-0xe3fff pnpid ORM0000 on isa0
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
Timecounter "TSC" frequency 2665920656 Hz quality 800
Timecounters tick every 1.000 msec
Waiting 2 seconds for SCSI devices to settle
ahd0: Invalid Sequencer interrupt occurred.
>How-To-Repeat:
Boot a machine with at least 3 SCSI disks on this particular SuperMicro
server platform.  (This issue has been replicated with two similar machines.)

>Fix:

>Release-Note:
>Audit-Trail:
>Unformatted:
 >>>>>>>>>>>>>>>>>> Dump Card State Begins <<<<<<<<<<<<<<<<<
 ahd0: Dumping Card State at program address 0x23b Mode 0x0
 Card was paused
 INTSTAT[0x0] SELOID[0x3] SELID[0x10] HS_MAILBOX[0x0] 
 INTCTL[0x80] SEQINTSTAT[0x0] SAVED_MODE[0x11] DFFSTAT[0x33] 
 SCSISIGI[0x0] SCSIPHASE[0x0] SCSIBUS[0x0] LASTPHASE[0x1] 
 SCSISEQ0[0x0] SCSISEQ1[0x12] SEQCTL0[0x0] SEQINTCTL[0x6] 
 SEQ_FLAGS[0x0] SEQ_FLAGS2[0x0] QFREEZE_COUNT[0x4] 
 KERNEL_QFREEZE_COUNT[0x4] MK_MESSAGE_SCB[0xff00] MK_MESSAGE_SCSIID[0xff] 
 SSTAT0[0x0] SSTAT1[0x0] SSTAT2[0x0] SSTAT3[0x0] PERRDIAG[0x0] 
 SIMODE1[0xa4] LQISTAT0[0x0] LQISTAT1[0x0] LQISTAT2[0x0] 
 LQOSTAT0[0x0] LQOSTAT1[0x0] LQOSTAT2[0x0] 

 SCB Count = 512 CMDS_PENDING = 1 LASTSCB 0xffff CURRSCB 0x1fa NEXTSCB 0xff80
 qinstart = 49 qinfifonext = 51
 QINFIFO: 0x1fa 0x1fd
 WAITING_TID_QUEUES:
        0 ( 0x1fc )
 Pending list:
 509 FIFO_USE[0x0] SCB_CONTROL[0x40] SCB_SCSIID[0x17] 
 506 FIFO_USE[0x0] SCB_CONTROL[0x40] SCB_SCSIID[0x37] 
 508 FIFO_USE[0x0] SCB_CONTROL[0x40] SCB_SCSIID[0x7] 
 Total 3
 Kernel Free SCB lists: 
   Any Device: 504 497 498 499 500 501 502 503 505 507 510 511 496 495 494 493 492 491 490 489 488 487 486 485 484 483 482 481 480 479 478 477 476 475 474 473 472 471 470 469 468 467 466 465 464 463 462 461 460 459 458 457 456 455 454 453 452 451 450 449  448 447 446 445 444 443 442 441 440 439 438 437 436 435 434 433 432 431 430 429 428 427 426 425 424 423 422 421 420 419 418 417 416 415 414 413 412 411 410 409 408 407 406 405 404 403 402 401 400 399 398 397 396 395 394 393 392 391 390 389 388 387 386 38 5 384 383 382 381 380 379 378 377 376 375 374 373 372 371 370 369 368 367 366 365 364 363 362 361 360 359 358 357 356 355 354 353 352 351 350 349 348 347 346 345 344 343 342 341 340 339 338 337 336 335 334 333 332 331 330 329 328 327 326 325 324 323 322  321 320 319 318 317 316 315 314 313 312 311 310 309 308 307 306 305 304 303 302 301 300 299 298 297 296 295 294 293 292 291 290 289 288 287 286 285 284 283 282 281 280 279 278 277 276 275 274 273 272 271 270 269 268 267 266 
 265!
   264 263 262 261 260 259 258 257 256 255 254 253 252 251 250 249 248 247 246 245 244 243 242 241 240 239 238 237 236 235 234 233 232 231 230 229 228 227 226 225 224 223 222 221 220 219 218 217 216 215 214 213 212 211 210 209 208 207 206 205 204 203 202  201 200 199 198 197 196 195 194 193 192 191 190 189 188 187 186 185 184 183 182 181 180 179 178 177 176 175 174 173 172 171 170 169 168 167 166 165 164 163 162 161 160 159 158 157 156 155 154 153 152 151 150 149 148 147 146 145 144 143 142 141 140 139 13 8 137 136 135 134 133 132 131 130 129 128 127 126 125 124 123 122 121 120 119 118 117 116 115 114 113 112 111 110 109 108 107 106 105 104 103 102 101 100 99 98 97 96 95 94 93 92 91 90 89 88 87 86 85 84 83 82 81 80 79 78 77 76 75 74 73 72 71 70 69 68 67 6 6 65 64 63 62 61 60 59 58 57 56 55 54 53 52 51 50 49 48 47 46 45 44 43 42 41 40 39 38 37 36 35 34 33 32 31 30 29 28 27 26 25 24 23 22 21 20 19 18 17 16 15 14 13 12 11 10 9 8 7 6 5 4 3 2 1 0 
 Sequencer Complete DMA-inprog list: 
 Sequencer Complete list: 
 Sequencer DMA-Up and Complete list: 
 Sequencer On QFreeze and Complete list: 

 ahd0: FIFO0 Free, LONGJMP == 0x8000, SCB 0x1fd
 SEQIMODE[0x3f] SEQINTSRC[0x0] DFCNTRL[0x0] DFSTATUS[0x89] 
 SG_CACHE_SHADOW[0x2] SG_STATE[0x0] DFFSXFRCTL[0x0] 
 SOFFCNT[0x0] MDFFSTAT[0x5] SHADDR = 0x00, SHCNT = 0x0 
 HADDR = 0x00, HCNT = 0x0 CCSGCTL[0x10] 

 ahd0: FIFO1 Free, LONGJMP == 0x8063, SCB 0x1fa
 SEQIMODE[0x3f] SEQINTSRC[0x0] DFCNTRL[0x0] DFSTATUS[0x89] 
 SG_CACHE_SHADOW[0x2] SG_STATE[0x0] DFFSXFRCTL[0x0] 
 SOFFCNT[0x0] MDFFSTAT[0x5] SHADDR = 0x00, SHCNT = 0x0 
 HADDR = 0x00, HCNT = 0x0 CCSGCTL[0x10] 
 LQIN: 0x8 0x0 0x1 0xfd 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 
 ahd0: LQISTATE = 0x0, LQOSTATE = 0x0, OPTIONMODE = 0x42
 ahd0: OS_SPACE_CNT = 0x20 MAXCMDCNT = 0x1
 ahd0: SAVED_SCSIID = 0x0 SAVED_LUN = 0x0
 SIMODE0[0xc] 
 CCSCBCTL[0x4] 
 ahd0: REG0 == 0x1c60, SINDEX = 0x10e, DINDEX = 0x108
 ahd0: SCBPTR == 0x1fd, SCB_NEXT == 0xff80, SCB_NEXT2 == 0x1fa
 CDB 12 0 0 80 88 6b
 STACK: 0x236 0x2 0x0 0x0 0x0 0x0 0x0 0x0
 <<<<<<<<<<<<<<<<< Dump Card State Ends >>>>>>>>>>>>>>>>>>
 (probe1:ahd0:0:1:0): inquiry data fails comparison at DV2 step
 (probe3:ahd0:0:3:0): inquiry data fails comparison at DV1 step
 (probe0:ahd0:0:0:0): inquiry data fails comparison at DV1 step
 ses0 at ahd0 bus 0 target 6 lun 0
 ses0: <SUPER GEM318 0> Fixed Processor SCSI-2 device 
 ses0: 3.300MB/s transfers
 ses0: SAF-TE Compliant Device
 da0 at ahd0 bus 0 target 0 lun 0
 da0: <SEAGATE ST336607LC 0007> Fixed Direct Access SCSI-3 device 
 da0: 320.000MB/s transfers (160.000MHz DT, offset 63, 16bit)
 da0: Command Queueing Enabled
 da0: 35003MB (71687372 512 byte sectors: 255H 63S/T 4462C)
 da2 at ahd0 bus 0 target 3 lun 0
 da2: <SEAGATE ST373455LC 0003> Fixed Direct Access SCSI-3 device 
 da2: 320.000MB/s transfers (160.000MHz DT, offset 63, 16bit)
 da2: Command Queueing Enabled
 da2: 70007MB (143374744 512 byte sectors: 255H 63S/T 8924C)
 da1 at ahd0 bus 0 target 1 lun 0
 da1: <SEAGATE ST336607LC 0006> Fixed Direct Access SCSI-3 device 
 da1: 320.000MB/s transfers (160.000MHz DT, offset 63, 16bit)
 da1: Command Queueing Enabled
 da1: 35003MB (71687372 512 byte sectors: 255H 63S/T 4462C)
 GEOM_MIRROR: Device mirror/newnews launched (1/1).
 GEOM_LABEL: Label for provider da2 is ufs/news.
 GEOM_MIRROR: Device mirror/root launched (2/2).
 GEOM_LABEL: Label for provider mirror/roota is ufs/root.
 GEOM_LABEL: Label for provider mirror/rootb is label/swap.
 GEOM_LABEL: Label for provider mirror/rootd is ufs/var.
 GEOM_LABEL: Label for provider mirror/roote is ufs/usr.
 Trying to mount root from ufs:/dev/ufs/root