[Bug 248652] [netmap]: pkt-gen tx huge pps difference between 11-STABLE and 12-STABLE/CURRENT on ix & ixl NIC

Fri Aug 14 08:52:04 UTC 2020

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=248652

            Bug ID: 248652
           Summary: [netmap]: pkt-gen tx huge pps difference between
                    11-STABLE and 12-STABLE/CURRENT on ix & ixl NIC
           Product: Base System
           Version: CURRENT
          Hardware: amd64
                OS: Any
            Status: New
          Severity: Affects Only Me
          Priority: ---
         Component: kern
          Assignee: bugs at FreeBSD.org
          Reporter: sg at efficientip.com

I'm testing netmap tx performance between 11-STABLE and CURRENT (same results
as 12-STABLE) with 2 NICs:
Intel X520 (10G) and Intel IXL710 (40G)
Here are my tests and the results using differents OS version/NIC & number of
queues

*******************************************

Testing NIC Intel X520, 1 queue configured
pkt-gen -i ix1 -f tx -S a0:36:9f:3e:57:1a -D 3c:fd:fe:a2:22:91 -s 192.168.0.1
-d 192.168.0.2

11-STABLE:
ix1: <Intel(R) PRO/10GbE PCI-Express Network Driver, Version - 3.2.12-k> port
0xece0-0xecff mem 0xdb600000-0xdb6fffff,0xdb7fc000-0xdb7fffff irq 53 at device
0.1 numa-domain 0 on pci5
ix1: Using MSI-X interrupts with 2 vectors
ix1: Ethernet address: a0:36:9f:51:c9:66
ix1: PCI Express Bus: Speed 5.0GT/s Width x8
ix1: netmap queues/slots: TX 1/2048, RX 1/2048

pkt-gen result:
297.988718 main_thread [2639] 14.151 Mpps (15.049 Mpkts 6.792 Gbps in 1063439
usec) 510.11 avg_batch 0 min_space
14Mpps

CURRENT:
ix1: <Intel(R) PRO/10GbE PCI-Express Network Driver> port 0xece0-0xecff mem
0xdb600000-0xdb6fffff,0xdb7fc000-0xdb7fffff irq 53 at device 0.1 numa-domain 0
on pci5
ix1: Using 2048 TX descriptors and 2048 RX descriptors
ix1: Using 1 RX queues 1 TX queues
ix1: Using MSI-X interrupts with 2 vectors
ix1: allocated for 1 queues
ix1: allocated for 1 rx queues
ix1: Ethernet address: a0:36:9f:51:c9:66
ix1: PCI Express Bus: Speed 5.0GT/s Width x8
ix1: netmap queues/slots: TX 1/2048, RX 1/2048

pkt-gen result:
198.445241 main_thread [2639] 2.615 Mpps (2.620 Mpkts 1.255 Gbps in 1001871
usec) 466.26 avg_batch 99999 min_space

2.6Mpps: much slower than 11-STABLE (14Mpps)

*******************************************

Testing NIC Intel IX710, 6 queues configured
pkt-gen -i ixl0 -f tx -S 9c:69:b4:60:ef:44 -D 9c:69:b4:60:35:ac -s 192.168.2.1
-d 192.168.2.2

11-STABLE:
ixl0: <Intel(R) Ethernet Connection 700 Series PF Driver, Version - 1.11.9-k>
mem 0xd5000000-0xd57fffff,0xd6ff0000-0xd6ff7fff irq 40 at device 0.0
numa-domain 0 on pci2
ixl0: using 2048 tx descriptors and 2048 rx descriptors
ixl0: fw 6.0.48442 api 1.7 nvm 6.01 etid 800034a4 oem 1.262.0
ixl0: PF-ID[0]: VFs 64, MSIX 129, VF MSIX 5, QPs 768, I2C
ixl0: Using MSIX interrupts with 7 vectors
ixl0: Allocating 8 queues for PF LAN VSI; 6 queues active
ixl0: Ethernet address: 9c:69:b4:60:ef:44
ixl0: PCI Express Bus: Speed 8.0GT/s Width x8
ixl0: SR-IOV ready
ixl0: netmap queues/slots: TX 6/2048, RX 6/2048
ixl0: TSO4 requires txcsum, disabling both...

pkt-gen result:
515.210701 main_thread [2639] 42.566 Mpps (45.248 Mpkts 20.432 Gbps in 1062998
usec) 395.17 avg_batch 99999 min_space

42Mpps

CURRENT:
ixl0: <Intel(R) Ethernet Controller XL710 for 40GbE QSFP+ - 2.2.0-k> mem
0xd5000000-0xd57fffff,0xd6ff0000-0xd6ff7fff irq 40 at device 0.0 numa-domain 0
on pci2
ixl0: fw 6.0.48442 api 1.7 nvm 6.01 etid 800034a4 oem 1.262.0
ixl0: PF-ID[0]: VFs 64, MSI-X 129, VF MSI-X 5, QPs 768, I2C
ixl0: Using 2048 TX descriptors and 2048 RX descriptors
ixl0: Using 6 RX queues 6 TX queues
ixl0: Using MSI-X interrupts with 7 vectors
ixl0: Ethernet address: 9c:69:b4:60:ef:44
ixl0: Allocating 8 queues for PF LAN VSI; 6 queues active
ixl0: PCI Express Bus: Speed 8.0GT/s Width x8
ixl0: SR-IOV ready
ixl0: netmap queues/slots: TX 6/2048, RX 6/2048
ixl0: Media change is not supported.
ixl0: Link is up, 40 Gbps Full Duplex, Requested FEC: None, Negotiated FEC:
None, Autoneg: True, Flow Control: None

pkt-gen result:
941.463329 main_thread [2639] 13.564 Mpps (13.741 Mpkts 6.511 Gbps in 1013001
usec) 16.04 avg_batch 99999 min_space

13Mpps: much slower than 11-STABLE (42Mpps)

*******************************************
And a last test, this one showing better performance in CURRENT vs 11-STABLE :)

Testing NIC Intel IX710, 1 queue configured
pkt-gen -i ixl0 -f tx -S 9c:69:b4:60:ef:44 -D 9c:69:b4:60:35:ac -s 192.168.2.1
-d 192.168.2.2

11-STABLE:
ixl0: <Intel(R) Ethernet Connection 700 Series PF Driver, Version - 1.11.9-k>
mem 0xd5000000-0xd57fffff,0xd6ff0000-0xd6ff7fff irq 40 at device 0.0
numa-domain 0 on pci2
ixl0: using 2048 tx descriptors and 2048 rx descriptors
ixl0: fw 6.0.48442 api 1.7 nvm 6.01 etid 800034a4 oem 1.262.0
ixl0: PF-ID[0]: VFs 64, MSIX 129, VF MSIX 5, QPs 768, I2C
ixl0: Using MSIX interrupts with 2 vectors
ixl0: Allocating 1 queues for PF LAN VSI; 1 queues active
ixl0: Ethernet address: 9c:69:b4:60:ef:44
ixl0: PCI Express Bus: Speed 8.0GT/s Width x8
ixl0: SR-IOV ready
ixl0: netmap queues/slots: TX 1/2048, RX 1/2048
ixl0: TSO4 requires txcsum, disabling both...

pkt-gen result:
609.889550 main_thread [2639] 8.413 Mpps (8.617 Mpkts 4.038 Gbps in 1024294
usec) 511.42 avg_batch 0 min_space

8Mpps

CURRENT:
ixl0: <Intel(R) Ethernet Controller XL710 for 40GbE QSFP+ - 2.2.0-k> mem
0xd5000000-0xd57fffff,0xd6ff0000-0xd6ff7fff irq 40 at device 0.0 numa-domain 0
on pci2
ixl0: fw 6.0.48442 api 1.7 nvm 6.01 etid 800034a4 oem 1.262.0
ixl0: PF-ID[0]: VFs 64, MSI-X 129, VF MSI-X 5, QPs 768, I2C
ixl0: Using 2048 TX descriptors and 2048 RX descriptors
ixl0: Using 1 RX queues 1 TX queues
ixl0: Using MSI-X interrupts with 2 vectors
ixl0: Ethernet address: 9c:69:b4:60:ef:44
ixl0: Allocating 1 queues for PF LAN VSI; 1 queues active
ixl0: PCI Express Bus: Speed 8.0GT/s Width x8
ixl0: SR-IOV ready
ixl0: netmap queues/slots: TX 1/2048, RX 1/2048
ixl0: Media change is not supported.
ixl0: Link is up, 40 Gbps Full Duplex, Requested FEC: None, Negotiated FEC:
None, Autoneg: True, Flow Control: None

pkt-gen result:
526.299416 main_thread [2639] 12.228 Mpps (12.240 Mpkts 5.870 Gbps in 1001000
usec) 14.37 avg_batch 99999 min_space

12Mpps: much better than 11-STABLE (8Mpps)

-- 
You are receiving this mail because:
You are the assignee for the bug.