MSI interrupts.

Wed Apr 27 13:45:14 UTC 2011

    Do you still want to see the output of 'show lapic'?

    What quirk do you have in mind? I'd be glad to do the testing.

On Mon, Apr 25, 2011 at 1:55 PM, John Baldwin <jhb at freebsd.org> wrote:

> On Wednesday, April 20, 2011 3:43:32 pm Bret Ketchum wrote:
> >     My response, which included dmesg output along with verbose boot
> > messages was rejected by the mailing list moderator as too large so I'll
> > reply with some of the interesting bits. Here is the MSI related mesages
> > grepped out of the verbose boot messages:
> >
> > ioapic0: routing intpin 9 (ISA IRQ 9) to lapic 0 vector 48
> >         MSI supports 2 messages, vector masks
> >         MSI supports 2 messages, vector masks
> >         MSI supports 2 messages, vector masks
> >         MSI supports 2 messages, vector masks
> >         MSI supports 2 messages, vector masks
> >         MSI supports 2 messages, vector masks
> >         MSI supports 1 message, 64 bit, vector masks
> >         MSI supports 1 message, 64 bit, vector masks
> > igb0: attempting to allocate 5 MSI-X vectors (10 supported)
> > msi: routing MSI-X IRQ 256 to local APIC 0 vector 49
> > msi: routing MSI-X IRQ 257 to local APIC 0 vector 50
> > msi: routing MSI-X IRQ 258 to local APIC 0 vector 51
> > msi: routing MSI-X IRQ 259 to local APIC 0 vector 52
> > msi: routing MSI-X IRQ 260 to local APIC 0 vector 53
> > igb0: Using MSIX interrupts with 5 vectors
> > igb1: attempting to allocate 5 MSI-X vectors (10 supported)
> > msi: routing MSI-X IRQ 261 to local APIC 0 vector 54
> > msi: routing MSI-X IRQ 262 to local APIC 0 vector 55
> > msi: routing MSI-X IRQ 263 to local APIC 0 vector 56
> > msi: routing MSI-X IRQ 264 to local APIC 0 vector 57
> > msi: routing MSI-X IRQ 265 to local APIC 0 vector 58
> > igb1: Using MSIX interrupts with 5 vectors
> > cmlpci0: attempting to allocate 1 MSI vectors (32 supported)
> > msi: routing MSI IRQ 266 to local APIC 0 vector 80
> > cmlpci1: attempting to allocate 1 MSI vectors (32 supported)
> > msi: routing MSI IRQ 267 to local APIC 0 vector 112
> > cmlpci2: attempting to allocate 1 MSI vectors (32 supported)
> > msi: routing MSI IRQ 268 to local APIC 0 vector 144
> > cmlpci3: attempting to allocate 1 MSI vectors (32 supported)
> > msi: routing MSI IRQ 269 to local APIC 0 vector 176
> > cmlpci4: attempting to allocate 1 MSI vectors (16 supported)
> > msi: routing MSI IRQ 270 to local APIC 0 vector 64
> > cmlpci5: attempting to allocate 1 MSI vectors (16 supported)
> > msi: routing MSI IRQ 271 to local APIC 0 vector 96
> > cmlpci6: attempting to allocate 1 MSI vectors (16 supported)
> > msi: routing MSI IRQ 272 to local APIC 0 vector 160
> > cmlpci7: attempting to allocate 1 MSI vectors (16 supported)
> > msi: routing MSI IRQ 273 to local APIC 0 vector 192
> > ioapic0: routing intpin 16 (PCI IRQ 16) to lapic 0 vector 59
> > ioapic0: routing intpin 21 (PCI IRQ 21) to lapic 0 vector 60
> > ioapic0: routing intpin 18 (PCI IRQ 18) to lapic 0 vector 61
> > ioapic0: routing intpin 23 (PCI IRQ 23) to lapic 0 vector 62
> > ioapic0: routing intpin 19 (PCI IRQ 19) to lapic 0 vector 63
> > ioapic0: routing intpin 14 (ISA IRQ 14) to lapic 0 vector 65
> > ioapic0: routing intpin 15 (ISA IRQ 15) to lapic 0 vector 66
> >         MSI supports 2 messages, vector masks
> >         MSI supports 2 messages, vector masks
> >         MSI supports 2 messages, vector masks
> >         MSI supports 2 messages, vector masks
> >         MSI supports 2 messages, vector masks
> >         MSI supports 2 messages, vector masks
> > cmlpci8: attempting to allocate 1 MSI vectors (2 supported)
> > msi: routing MSI IRQ 274 to local APIC 0 vector 68
> >         MSI supports 4 messages, 64 bit, vector masks
> >         MSI supports 4 messages, 64 bit, vector masks
> >         MSI supports 4 messages, 64 bit, vector masks
> >         MSI supports 4 messages, 64 bit, vector masks
> >         MSI supports 16 messages, 64 bit, vector masks
> >         MSI supports 16 messages, 64 bit, vector masks
> > cmlpci9: attempting to allocate 1 MSI vectors (16 supported)
> > msi: routing MSI IRQ 275 to local APIC 0 vector 208
> > cmlpci10: attempting to allocate 1 MSI vectors (16 supported)
> > msi: routing MSI IRQ 276 to local APIC 0 vector 224
> >         MSI supports 16 messages, 64 bit, vector masks
> >         MSI supports 16 messages, 64 bit, vector masks
> > cmlpci11: attempting to allocate 1 MSI vectors (16 supported)
> > APIC: Couldn't find APIC vectors for 1 IRQs
> > ioapic2: routing intpin 3 (PCI IRQ 51) to lapic 0 vector 67
> > cmlpci12: attempting to allocate 1 MSI vectors (16 supported)
> > APIC: Couldn't find APIC vectors for 1 IRQs
> > ioapic2: routing intpin 5 (PCI IRQ 53) to lapic 0 vector 69
> > ioapic0: routing intpin 4 (ISA IRQ 4) to lapic 0 vector 70
> > ioapic0: routing intpin 3 (ISA IRQ 3) to lapic 0 vector 71
> > ioapic0: routing intpin 1 (ISA IRQ 1) to lapic 0 vector 72
> > ioapic0: routing intpin 3 (ISA IRQ 3) to lapic 2 vector 48
> > ioapic0: routing intpin 4 (ISA IRQ 4) to lapic 4 vector 48
> > ioapic0: routing intpin 9 (ISA IRQ 9) to lapic 6 vector 48
> > ioapic0: routing intpin 15 (ISA IRQ 15) to lapic 2 vector 49
> > ioapic0: routing intpin 16 (PCI IRQ 16) to lapic 4 vector 49
> > ioapic0: routing intpin 18 (PCI IRQ 18) to lapic 6 vector 49
> > ioapic0: routing intpin 21 (PCI IRQ 21) to lapic 2 vector 50
> > ioapic0: routing intpin 23 (PCI IRQ 23) to lapic 4 vector 50
> > ioapic2: routing intpin 3 (PCI IRQ 51) to lapic 6 vector 50
> > msi: Assigning MSI-X IRQ 258 to local APIC 2 vector 51
> > msi: Assigning MSI-X IRQ 260 to local APIC 2 vector 52
> > msi: Assigning MSI-X IRQ 263 to local APIC 2 vector 53
> > msi: Assigning MSI-X IRQ 265 to local APIC 4 vector 51
> > msi: Assigning MSI IRQ 266 to local APIC 6 vector 51
> > msi: Assigning MSI IRQ 268 to local APIC 2 vector 54
> > msi: Assigning MSI IRQ 269 to local APIC 4 vector 52
> > msi: Assigning MSI IRQ 270 to local APIC 6 vector 52
> > msi: Assigning MSI IRQ 272 to local APIC 2 vector 55
> > msi: Assigning MSI IRQ 273 to local APIC 4 vector 53
> > msi: Assigning MSI IRQ 274 to local APIC 6 vector 53
> > msi: Assigning MSI IRQ 276 to local APIC 2 vector 56
> >
> >     vectors are allocated starting at 48. The first 48 (beginning at 0)
> are
> > reserved IDT I/O interrupts. This is an amd64 build. The maximum that
> will
> > be allocated is 192. Note that when requesting a single MSI interrupts
> > apic_alloc_vectors actually looks for a range of free vectors based upon
> the
> > number of vectors the PCI HBA supports. Yet subsequent calls to
> > apic_alloc_vectors returns vectors within the already allocated range:
> >
> > msi: routing MSI IRQ 270 to local APIC 0 vector 64
> > cmlpci5: attempting to allocate 1 MSI vectors (16 supported)
> >      ....
> > ioapic0: routing intpin 14 (ISA IRQ 14) to lapic 0 vector 65
> > ioapic0: routing intpin 15 (ISA IRQ 15) to lapic 0 vector 66
> >
> >     Question, why limit the allocation of vectors to a range supported by
> a
> > PCI device when vectors within that range may be allocated in subsequent
> > requests?
>
> Ah, I just replied to another e-mail on the list with the reason.
>
> Hmm.  After rechecking this, the spec doesn't require the behavior of fully
> aligning based on the device's size, but IIRC, we encountered some hardware
> that did behave that way.  Perhaps some bge(4) devices (I think the
> hardware
> actually fixes the low N bits of the message data register as 0, so if you
> wrote a value with any low N bits non-zero they were lost).  Perhaps we can
> use a quirk to work around that behavior now though instead.
>
> --
> John Baldwin
>