[Bug 287492] Unable to detect NIC cards on few PCIe Gen5 slots on Dell R770 servers where x2APIC mode is enabled

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=287492

            Bug ID: 287492
           Summary: Unable to detect NIC cards on few PCIe Gen5 slots on
                    Dell R770 servers where x2APIC mode is enabled
           Product: Base System
           Version: 15.0-CURRENT
          Hardware: Any
                OS: Any
            Status: New
          Severity: Affects Some People
          Priority: ---
         Component: kern
          Assignee: bugs@FreeBSD.org
          Reporter: sreekanth.reddy@broadcom.com

Observing below error messages when NIC card is connected to few PCIe slots of
DELL R770 servers with the FreeBSD 14.2 OS. Same issue is there even with
latest FreeBSD Current kernel. 

bnxt5: <Broadcom BCM57608 NetXtreme-E 25Gb/50Gb/100Gb/200Gb/400Gb Ethernet> mem
0x1eefff010000-0x1eefff01ffff,0x1eeffe000000-0x1eeffeffffff,0x1eefff028000-0x1eefff02ffff
at device 0.0 numa-domain 1 on pci41
bnxt5: kTLS is supported ifp:0xfffff820845e4000 ctx:0xfffff801217bd000 dev:
0xfffff80121821200 softc: 0xfffffe01a9faa000
bnxt5: Using 256 TX descriptors and 256 RX descriptors
bnxt5: Using 64 RX queues 64 TX queues
bnxt5: Using MSI-X interrupts with 81 vectors

msi_map: unsupported destination APIC ID 512
bnxt5: failed to setup interrupt for rid 1, name rxq0: 22
bnxt5: _iflib_irq_alloc failed 22
bnxt5: Failed to register RX completion ring handler
bnxt5: IFDI_MSIX_INTR_ASSIGN failed 22
device_attach: bnxt5 attach returned 22


My Analysis:

msi_map: unsupported destination APIC ID 512

This error message is coming from FreeBSD stack(i.e. from msi_map() in
sys/x86/x86/msi.c file) as shown below,

 if (error == EOPNOTSUPP && msi->msi_cpu > 0xff) {
     printf("%s: unsupported destination APIC ID %u\n", __func__,              
      msi->msi_cpu);
     error = EINVAL;
  }

we will observe this error message when msi->msi_cpu (i.e. cpu's apic id) is
greater then 255. From the error message we can see that apic id is 512 which
is greater than 255. 


Also, I see that on this system x2apic_mode is enabled. 

# sysctl hw.apic.x2apic_mode
hw.apic.x2apic_mode: 1


So, when x2apic_mode is enabled then I think that it can be expected that CPU's
apic id can be greater than 255. So, looks like FreeBSD kernel has bug here in
msi_map(), which works only with xapic mode not for x2apic. Please correct if I
am wrong here.

Thanks,
Sreekanth

-- 
You are receiving this mail because:
You are the assignee for the bug.