mfi(4) patch to add MSI-X support, possibly address command timeouts

[Sergey Kandaurov]

On 1 September 2011 15:07, John Baldwin <jhb at freebsd.org> wrote:
> On Wednesday, August 31, 2011 5:27:09 pm Sergey Kandaurov wrote:
>> On 1 September 2011 01:17, John Baldwin <jhb at freebsd.org> wrote:
>> > On Wednesday, August 31, 2011 3:24:12 pm Sergey Kandaurov wrote:
>> >> On 31 August 2011 21:34, John Baldwin <jhb at freebsd.org> wrote:
>> >> > I'd like some folks to test a patch to the mfi(4) driver that may help to
>> >> > address issues several folks have reported.  The patch does two things, first
>> >> > it adds some dummy reads of PCI registers when checking device status in the
>> >> > interrupt handler to "flush" the writes to ACK interrupts.  The Linux
>> >> > megaraid-sas driver uses this approach and some folks have tested a patch from
>> >> > Scott Long which had a somewhat similar effect.  Second, it enables the use of
>> >> > MSI-X interrupts for many newer devices.
>> >> >
>> >> > The patch is available below and at www.freebsd.org/~jhb/patches/mfi.patch
>> >>
>> >> mfi0: <LSI MegaSAS Gen2> port 0x3000-0x30ff mem
>> >> 0x9dd40000-0x9dd43fff,0x9dd00000-0x9dd3ffff irq 26 at device 0.0 on
>> >> pci26
>> >> mfi0: Using MSI-X
>> >> mfi0: Megaraid SAS driver Ver 3.00
>> >>
>> >> However, booting never finishes ending up with:
>> >> mfi0: COMMAND 0xffffff8000b3a550 TIMEOUT AFTER 58 SECONDS
>> >> mfi0: COMMAND 0xffffff8000b3a550 TIMEOUT AFTER 88 SECONDS
>> >> mfi0: COMMAND 0xffffff8000b3a550 TIMEOUT AFTER 118 SECONDS
>> >> mfi0: COMMAND 0xffffff8000b3a550 TIMEOUT AFTER 148 SECONDS
>> >> mfi0: COMMAND 0xffffff8000b3a550 TIMEOUT AFTER 179 SECONDS
>> >> mfi0: COMMAND 0xffffff8000b3a550 TIMEOUT AFTER 209 SECONDS
>> >
>> > Did this work fine without the patch?
>>
>> Yes, like a charm.
>
> Hmm, the Linux driver definitely uses MSI-X for your board.
>
> What chipset do you have,

That is gen2 IBM ServeRAID M5015 SAS/SATA Controller.
So the chipset should be LSI SAS2108, as per its datasheet.


> and does your system use MSI IRQs for any other
> devices?

Yep, 2-port i82576 and 4-port i82576, each port uses 9 vectors.
Also 4 bce (BCM5709) embedded ports register their irq above 255, one irq
per interface, so I suspect they should use some kind of MSI/MSIX, too.

Indeed, verbose dmesg confirms that.

bce0: <Broadcom NetXtreme II BCM5709 1000Base-T (C0)> mem
0x96000000-0x97ffffff irq 28 at device 0.0 on pci11
bce0: Reserved 0x2000000 bytes for rid 0x10 type 3 at 0x96000000
bce0: attempting to allocate 1 MSI vectors (16 supported)
msi: routing MSI IRQ 256 to local APIC 0 vector 64
bce0: using IRQ 256 for MSI
miibus0: <MII bus> on bce0
brgphy0: <BCM5709C 10/100/1000baseTX PHY> PHY 1 on miibus0
brgphy0: OUI 0x0050ef, model 0x003c, rev. 8
brgphy0:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT,
1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto, auto-flow
bce0: [MPSAFE]
bce0: [ITHREAD]
bce0: ASIC (0x57092003); Rev (C0); Bus (PCIe x2, 5Gbps); B/C (5.2.2);
Flags (MSI|MFW); MFW (NCSI 2.0.10)
[...]
igb0: <Intel(R) PRO/1000 Network Connection version - 2.0.7> port
0x4020-0x403f mem
0x9dc20000-0x9dc3ffff,0x9d800000-0x9dbfffff,0x9dc44000-0x9dc47fff irq
24 at device 0.0 on pci21
igb0: Reserved 0x20000 bytes for rid 0x10 type 3 at 0x9dc20000
igb0: Reserved 0x4000 bytes for rid 0x1c type 3 at 0x9dc44000
igb0: attempting to allocate 9 MSI-X vectors (10 supported)
msi: routing MSI-X IRQ 260 to local APIC 0 vector 49
msi: routing MSI-X IRQ 261 to local APIC 0 vector 50
msi: routing MSI-X IRQ 262 to local APIC 0 vector 51
msi: routing MSI-X IRQ 263 to local APIC 0 vector 52
msi: routing MSI-X IRQ 264 to local APIC 0 vector 53
msi: routing MSI-X IRQ 265 to local APIC 0 vector 54
msi: routing MSI-X IRQ 266 to local APIC 0 vector 55
msi: routing MSI-X IRQ 267 to local APIC 0 vector 56
msi: routing MSI-X IRQ 268 to local APIC 0 vector 57
igb0: using IRQs 260-268 for MSI-X
igb0: Using MSIX interrupts with 9 vectors
igb0: [MPSAFE]
igb0: [ITHREAD]
igb0: [MPSAFE]
igb0: [ITHREAD]
igb0: [MPSAFE]
igb0: [ITHREAD]
igb0: [MPSAFE]
igb0: [ITHREAD]
igb0: [MPSAFE]
igb0: [ITHREAD]
igb0: [MPSAFE]
igb0: [ITHREAD]
igb0: [MPSAFE]
igb0: [ITHREAD]
igb0: [MPSAFE]
igb0: [ITHREAD]
igb0: [MPSAFE]
igb0: [ITHREAD]


>
> Also, can you break into ddb and do 'show intrcnt' to see if you are getting
> interrupts for the mfi device?

Captured shortly after boot hangs.

db> show intrcnt
irq4: uart0             1
irq14: ata0             35
irq17: uhci0 uhci2+     16
cpu0: timer             148278
irq278: mfi0            2

And a bit later.

irq4: uart0             3
irq14: ata0             35
irq17: uhci0 uhci2+     16
cpu0: timer             187327
irq278: mfi0            2

-- 
wbr,
pluknet