[Bug 199321] Total MSI-X vector allocation limited to 191 vectors

[bugzilla-noreply at freebsd.org]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=199321

            Bug ID: 199321
           Summary: Total MSI-X vector allocation limited to 191 vectors
           Product: Base System
           Version: 11.0-CURRENT
          Hardware: amd64
                OS: Any
            Status: New
          Severity: Affects Some People
          Priority: ---
         Component: kern
          Assignee: freebsd-bugs at FreeBSD.org
          Reporter: jimharris at FreeBSD.org

System Configuration:
2x 18C E5-v3 CPU (SMT enabled)
2x X520 Intel Ethernet 10Gb Adapter (4 ports total)
11.0-CURRENT (r281283)

ixgbe driver tries to allocate full range of 64 MSI-X vectors per port - first
two ports succeed, but third and fourth ports fail and fall back to a single
MSI vector.

msix_alloc() tries to use intr_next_cpu() to round-robin MSI-X vectors across
all cores.  But driver initialization happens before APs are started, so
intr_next_cpu() just returns the BSP's apic ID 0.  Each local APIC is limited
to APIC_NUM_IOINTS==191 vectors, so a single local APIC cannot handle the 64
vectors for each of the 4 ports.

intr_shuffle_irqs runs in SI_SUB_SMP phase and is intended to shuffle iras off
of core 0 to APs, but at this point it is too late - pci_alloc_msix() already
reported to ixgbe(4) that it could not allocate all 64 vectors.

x86 defines NUM_MSI_INTS as 512, but this bug really restricts number of MSI-X
vectors to APIC_NUM_IOINTS (191).  Impact is that some drivers/devices which
can take advantage of multiple queues and one interrupt per queue must fall
back to a single interrupt.  There may also be inconsistency across platforms
based on device enumeration - whichever devices enumerate first will get its
allotment of MSI-X vectors.

This problem will become exacerbated as NVMe SSDs become more prevalent, which
also try to allocate 32-64 MSIx vectors each.  There are systems available
already with 8 NVMe SSDs in a single system.

-- 
You are receiving this mail because:
You are the assignee for the bug.