MSI interrupts.

Wed Apr 20 19:43:34 UTC 2011

    My response, which included dmesg output along with verbose boot
messages was rejected by the mailing list moderator as too large so I'll
reply with some of the interesting bits. Here is the MSI related mesages
grepped out of the verbose boot messages:

ioapic0: routing intpin 9 (ISA IRQ 9) to lapic 0 vector 48
        MSI supports 2 messages, vector masks
        MSI supports 2 messages, vector masks
        MSI supports 2 messages, vector masks
        MSI supports 2 messages, vector masks
        MSI supports 2 messages, vector masks
        MSI supports 2 messages, vector masks
        MSI supports 1 message, 64 bit, vector masks
        MSI supports 1 message, 64 bit, vector masks
igb0: attempting to allocate 5 MSI-X vectors (10 supported)
msi: routing MSI-X IRQ 256 to local APIC 0 vector 49
msi: routing MSI-X IRQ 257 to local APIC 0 vector 50
msi: routing MSI-X IRQ 258 to local APIC 0 vector 51
msi: routing MSI-X IRQ 259 to local APIC 0 vector 52
msi: routing MSI-X IRQ 260 to local APIC 0 vector 53
igb0: Using MSIX interrupts with 5 vectors
igb1: attempting to allocate 5 MSI-X vectors (10 supported)
msi: routing MSI-X IRQ 261 to local APIC 0 vector 54
msi: routing MSI-X IRQ 262 to local APIC 0 vector 55
msi: routing MSI-X IRQ 263 to local APIC 0 vector 56
msi: routing MSI-X IRQ 264 to local APIC 0 vector 57
msi: routing MSI-X IRQ 265 to local APIC 0 vector 58
igb1: Using MSIX interrupts with 5 vectors
cmlpci0: attempting to allocate 1 MSI vectors (32 supported)
msi: routing MSI IRQ 266 to local APIC 0 vector 80
cmlpci1: attempting to allocate 1 MSI vectors (32 supported)
msi: routing MSI IRQ 267 to local APIC 0 vector 112
cmlpci2: attempting to allocate 1 MSI vectors (32 supported)
msi: routing MSI IRQ 268 to local APIC 0 vector 144
cmlpci3: attempting to allocate 1 MSI vectors (32 supported)
msi: routing MSI IRQ 269 to local APIC 0 vector 176
cmlpci4: attempting to allocate 1 MSI vectors (16 supported)
msi: routing MSI IRQ 270 to local APIC 0 vector 64
cmlpci5: attempting to allocate 1 MSI vectors (16 supported)
msi: routing MSI IRQ 271 to local APIC 0 vector 96
cmlpci6: attempting to allocate 1 MSI vectors (16 supported)
msi: routing MSI IRQ 272 to local APIC 0 vector 160
cmlpci7: attempting to allocate 1 MSI vectors (16 supported)
msi: routing MSI IRQ 273 to local APIC 0 vector 192
ioapic0: routing intpin 16 (PCI IRQ 16) to lapic 0 vector 59
ioapic0: routing intpin 21 (PCI IRQ 21) to lapic 0 vector 60
ioapic0: routing intpin 18 (PCI IRQ 18) to lapic 0 vector 61
ioapic0: routing intpin 23 (PCI IRQ 23) to lapic 0 vector 62
ioapic0: routing intpin 19 (PCI IRQ 19) to lapic 0 vector 63
ioapic0: routing intpin 14 (ISA IRQ 14) to lapic 0 vector 65
ioapic0: routing intpin 15 (ISA IRQ 15) to lapic 0 vector 66
        MSI supports 2 messages, vector masks
        MSI supports 2 messages, vector masks
        MSI supports 2 messages, vector masks
        MSI supports 2 messages, vector masks
        MSI supports 2 messages, vector masks
        MSI supports 2 messages, vector masks
cmlpci8: attempting to allocate 1 MSI vectors (2 supported)
msi: routing MSI IRQ 274 to local APIC 0 vector 68
        MSI supports 4 messages, 64 bit, vector masks
        MSI supports 4 messages, 64 bit, vector masks
        MSI supports 4 messages, 64 bit, vector masks
        MSI supports 4 messages, 64 bit, vector masks
        MSI supports 16 messages, 64 bit, vector masks
        MSI supports 16 messages, 64 bit, vector masks
cmlpci9: attempting to allocate 1 MSI vectors (16 supported)
msi: routing MSI IRQ 275 to local APIC 0 vector 208
cmlpci10: attempting to allocate 1 MSI vectors (16 supported)
msi: routing MSI IRQ 276 to local APIC 0 vector 224
        MSI supports 16 messages, 64 bit, vector masks
        MSI supports 16 messages, 64 bit, vector masks
cmlpci11: attempting to allocate 1 MSI vectors (16 supported)
APIC: Couldn't find APIC vectors for 1 IRQs
ioapic2: routing intpin 3 (PCI IRQ 51) to lapic 0 vector 67
cmlpci12: attempting to allocate 1 MSI vectors (16 supported)
APIC: Couldn't find APIC vectors for 1 IRQs
ioapic2: routing intpin 5 (PCI IRQ 53) to lapic 0 vector 69
ioapic0: routing intpin 4 (ISA IRQ 4) to lapic 0 vector 70
ioapic0: routing intpin 3 (ISA IRQ 3) to lapic 0 vector 71
ioapic0: routing intpin 1 (ISA IRQ 1) to lapic 0 vector 72
ioapic0: routing intpin 3 (ISA IRQ 3) to lapic 2 vector 48
ioapic0: routing intpin 4 (ISA IRQ 4) to lapic 4 vector 48
ioapic0: routing intpin 9 (ISA IRQ 9) to lapic 6 vector 48
ioapic0: routing intpin 15 (ISA IRQ 15) to lapic 2 vector 49
ioapic0: routing intpin 16 (PCI IRQ 16) to lapic 4 vector 49
ioapic0: routing intpin 18 (PCI IRQ 18) to lapic 6 vector 49
ioapic0: routing intpin 21 (PCI IRQ 21) to lapic 2 vector 50
ioapic0: routing intpin 23 (PCI IRQ 23) to lapic 4 vector 50
ioapic2: routing intpin 3 (PCI IRQ 51) to lapic 6 vector 50
msi: Assigning MSI-X IRQ 258 to local APIC 2 vector 51
msi: Assigning MSI-X IRQ 260 to local APIC 2 vector 52
msi: Assigning MSI-X IRQ 263 to local APIC 2 vector 53
msi: Assigning MSI-X IRQ 265 to local APIC 4 vector 51
msi: Assigning MSI IRQ 266 to local APIC 6 vector 51
msi: Assigning MSI IRQ 268 to local APIC 2 vector 54
msi: Assigning MSI IRQ 269 to local APIC 4 vector 52
msi: Assigning MSI IRQ 270 to local APIC 6 vector 52
msi: Assigning MSI IRQ 272 to local APIC 2 vector 55
msi: Assigning MSI IRQ 273 to local APIC 4 vector 53
msi: Assigning MSI IRQ 274 to local APIC 6 vector 53
msi: Assigning MSI IRQ 276 to local APIC 2 vector 56

    vectors are allocated starting at 48. The first 48 (beginning at 0) are
reserved IDT I/O interrupts. This is an amd64 build. The maximum that will
be allocated is 192. Note that when requesting a single MSI interrupts
apic_alloc_vectors actually looks for a range of free vectors based upon the
number of vectors the PCI HBA supports. Yet subsequent calls to
apic_alloc_vectors returns vectors within the already allocated range:

msi: routing MSI IRQ 270 to local APIC 0 vector 64
cmlpci5: attempting to allocate 1 MSI vectors (16 supported)
     ....
ioapic0: routing intpin 14 (ISA IRQ 14) to lapic 0 vector 65
ioapic0: routing intpin 15 (ISA IRQ 15) to lapic 0 vector 66

    Question, why limit the allocation of vectors to a range supported by a
PCI device when vectors within that range may be allocated in subsequent
requests?

    In this case enough vectors are allocated that a supported range cannot
be found:

cmlpci11: attempting to allocate 1 MSI vectors (16 supported)
APIC: Couldn't find APIC vectors for 1 IRQs

On Fri, Apr 15, 2011 at 11:11 AM, John Baldwin <jhb at freebsd.org> wrote:

> On Tuesday, April 12, 2011 9:16:16 am Bret Ketchum wrote:
> >     I've a roll-your-own driver for FreeBSD 8.x that uses MSI interrupts
> for
> > PCI-E HBAs where one or more will be installed in a off-the-shelf amd64
> > pizza box. The driver is using bus_setup_intr() and depending upon the
> slots
> > the HBAs are install I see log messages from apic_alloc_vectors(), for
> > example:
> >
> > Apr 12 06:44:15 mfsbsd kernel: xxxpci10: attempting to allocate 1 MSI
> > vectors (16 supported)
> > Apr 12 06:44:15 mfsbsd kernel: APIC: Couldn't find APIC vectors for 1
> IRQs
> > Apr 12 06:44:15 mfsbsd kernel: ioapic1: routing intpin 13 (PCI IRQ 37) to
> > lapic 0 vector 59
> >
> >     Using vmstat -ia:
> >
> > interrupt                          total       rate
> > irq37: xxxpci10                       74          0
> >
> >     The problem appears to be that HBA interrupts are not being delivered
> to
> > the driver. If I swap cards around in slots I can eliminate the message
> and:
> >
> > Apr 12 06:44:15 mfsbsd kernel: msi: routing MSI IRQ 266 to local APIC 0
> > vector 80
> >
> >     And interrupts appear to be delivered properly. Before I dive in, can
> > anyone explain this behavior?
>
> Hmm, can you capture messages with bootverbose enabled?  A full boot -v
> dmesg
> might be useful as well.
>
> --
> John Baldwin
>