Regression in IXGBE(4) on 12.0-RELEASE?

[Sergey Akhmatov]

On 07/08/2019 07:15, Greg Rivers wrote:
> These Intel NICs have been performing very well for me:
> 
> ix0 at pci0:7:0:0:	class=0x020000 card=0x17d3103c chip=0x10fb8086 rev=0x01 hdr=0x00
>      vendor     = 'Intel Corporation'
>      device     = '82599ES 10-Gigabit SFI/SFP+ Network Connection'
>      class      = network
>      subclass   = ethernet
> 
> On 11.2-RELEASE, the cards probe like this:
> 
> ix0: <Intel(R) PRO/10GbE PCI-Express Network Driver, Version - 3.2.12-k> port 0x7000-0x701f mem 0xf7f00000-0xf7ffffff,0xf7ef0000-0xf7ef3fff irq 28 at device 0.0 numa-domain 0 on pci1
> ix0: Using MSI-X interrupts with 9 vectors
> ix0: Ethernet address: 8c:dc:d4:af:8a:e0
> ix0: PCI Express Bus: Speed 5.0GT/s Width x8
> ix0: netmap queues/slots: TX 8/2048, RX 8/2048
> 
> As expected, all 8 queues are enabled on the NIC, with 8 corresponding interrupt threads pinned to 8 different CPUs
> 
> But on 12.0-RELEASE, the cards probe like this:
> 
> ix0: <Intel(R) PRO/10GbE PCI-Express Network Driver> port 0x4020-0x403f mem 0xe2900000-0xe29fffff,0xe2a04000-0xe2a07fff at device 0.0 numa-domain 0 on pci4
> ix0: using 2048 tx descriptors and 2048 rx descriptors
> ix0: msix_init qsets capped at 16
> ix0: Unable to map MSIX table
> ix0: Using an MSI interrupt
> ix0: allocated for 1 queues
> ix0: allocated for 1 rx queues
> ix0: Ethernet address: 48:df:37:62:be:38
> ix0: PCI Express Bus: Speed 5.0GT/s Width x8
> ix0: netmap queues/slots: TX 1/2048, RX 1/2048
> ix0: link state changed to UP
> 
> MSI-X is disabled, and only one queue is enabled. I'm sure this is going to perform poorly under high network loads. Does anyone know why this is?

Don't know if it could help somehow, but I've got two identical boxes 
with similar NICs side by side, one running 11.2 and the other one 12.0
and I don't have such problems.

If someone points me where to look, I could help collecting more data to 
know how my system is different from Greg's.

$ freebsd-version
12.0-RELEASE-p7

$ pciconf -lvc ix0
ix0 at pci0:17:0:0:        class=0x020000 card=0x7a128086 chip=0x10fb8086 
rev=0x01 hdr=0x00
     vendor     = 'Intel Corporation'
     device     = '82599ES 10-Gigabit SFI/SFP+ Network Connection'
     class      = network
     subclass   = ethernet
     cap 01[40] = powerspec 3  supports D0 D3  current D0
     cap 05[50] = MSI supports 1 message, 64 bit, vector masks
     cap 11[70] = MSI-X supports 64 messages, enabled
                  Table in map 0x20[0x0], PBA in map 0x20[0x2000]
     cap 10[a0] = PCI-Express 2 endpoint max data 256(512) FLR RO NS
                  link x8(x8) speed 5.0(5.0) ASPM disabled(L0s)
     ecap 0001[100] = AER 1 0 fatal 0 non-fatal 1 corrected
     ecap 0003[140] = Serial 1 90e2baffff809b08
     ecap 000e[150] = ARI 1
     ecap 0010[160] = SR-IOV 1 IOV disabled, Memory Space disabled, ARI 
disabled
                      0 VFs configured out of 64 supported
                      First VF RID Offset 0x0180, VF RID Stride 0x0002
                      VF Device ID 0x10ed
                      Page Sizes: 4096 (enabled), 8192, 65536, 262144, 
1048576, 4194304

$ dmesg
ix0: <Intel(R) PRO/10GbE PCI-Express Network Driver> port 0x2fc0-0x2fdf 
mem 0x91200000-0x912fffff,0x910fc000-0x910fffff irq 32 at device 0.0 on pci2
ix0: using 2048 tx descriptors and 2048 rx descriptors
ix0: msix_init qsets capped at 16
ix0: pxm cpus: 8 queue msgs: 63 admincnt: 1
ix0: using 8 rx queues 8 tx queues
ix0: Using MSIX interrupts with 9 vectors
ix0: allocated for 8 queues
ix0: allocated for 8 rx queues
ix0: Ethernet address: 90:e2:ba:80:9b:08
ix0: PCI Express Bus: Speed 5.0GT/s Width x8
ix0: netmap queues/slots: TX 8/2048, RX 8/2048

==========
$ freebsd-version
11.2-RELEASE-p3

$ dmesg
ix0: <Intel(R) PRO/10GbE PCI-Express Network Driver, Version - 3.2.12-k> 
port 0x2fc0-0x2fdf mem 0x91200000-0x912fffff,0x910fc000-0x910fffff irq 
32 at device 0.0 numa-domain 0 on pci2
ix0: Using MSI-X interrupts with 9 vectors
ix0: Ethernet address: 90:e2:ba:80:99:d0
ix0: PCI Express Bus: Speed 5.0GT/s Width x8
ix0: netmap queues/slots: TX 8/2048, RX 8/2048