[Bug 258755] vmxnet3: low throughput

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=258755

            Bug ID: 258755
           Summary: vmxnet3: low throughput
           Product: Base System
           Version: 13.0-RELEASE
          Hardware: amd64
                OS: Any
            Status: New
          Severity: Affects Some People
          Priority: ---
         Component: kern
          Assignee: bugs@FreeBSD.org
          Reporter: michal@maximale.sk

I've setup a simple throughput test on my ESX 6.7:

   10.6.0.10     vmx1      vmx2    10.6.10.10
[ debian 1 ] <==> [ freebsd ] <==> [ debian 2 ]
iperf server        router         iperf client

freebsd is version 13-RELEASE-p4 and both debians are of version 11.

When I run iperf test I get suspiciously low throughput:

root@debian2:~# iperf3 -t 30 -i 10 -c 10.6.0.10 
Connecting to host 10.6.0.10, port 5201
[  5] local 10.6.10.10 port 56006 connected to 10.6.0.10 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-10.00  sec  2.24 GBytes  1.93 Gbits/sec   79   2.36 MBytes       
[  5]  10.00-20.00  sec  2.24 GBytes  1.93 Gbits/sec    2   2.30 MBytes       
[  5]  20.00-30.00  sec  2.27 GBytes  1.95 Gbits/sec    1   2.21 MBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-30.00  sec  6.76 GBytes  1.94 Gbits/sec   82             sender
[  5]   0.00-30.00  sec  6.76 GBytes  1.93 Gbits/sec                  receiver

iperf Done.

While test is running, kernel{if_io_tqg_X} is very high (hitting 100% on one
CPU). If I run the test with more parallel sessions (iperf -P X), I get higher
throughput with more kernel{if_io_tqg_X} threads utilizing the CPUs (this is
expected as parallel sessions are spread across CPUs). 

Now If I replace router with some linux (tested with centos 7), I get more than
10Gbps with single session (all three VMs are running on the same ESX host)
with CPU on the linux router doing almost nothing.

LRO and TSO are disabled on both vmx adapters.

root@freebsd:~ # dmesg | grep vmx
vmx0: <VMware VMXNET3 Ethernet Adapter> port 0x4000-0x400f mem
0xfe903000-0xfe903fff,0xfe902000-0xfe902fff,0xfe900000-0xfe901fff at device 0.0
on pci3
vmx0: Using 4096 TX descriptors and 2048 RX descriptors
vmx0: Using 4 RX queues 4 TX queues
vmx0: Using MSI-X interrupts with 5 vectors
vmx0: Ethernet address: 00:50:56:a7:85:e4
vmx0: netmap queues/slots: TX 4/4096, RX 4/2048
vmx1: <VMware VMXNET3 Ethernet Adapter> port 0x3000-0x300f mem
0xfe103000-0xfe103fff,0xfe102000-0xfe102fff,0xfe100000-0xfe101fff at device 0.0
on pci4
vmx1: Using 4096 TX descriptors and 2048 RX descriptors
vmx1: Using 4 RX queues 4 TX queues
vmx1: Using MSI-X interrupts with 5 vectors
vmx1: Ethernet address: 00:50:56:a7:1f:01
vmx1: netmap queues/slots: TX 4/4096, RX 4/2048
vmx2: <VMware VMXNET3 Ethernet Adapter> port 0x2000-0x200f mem
0xfd903000-0xfd903fff,0xfd902000-0xfd902fff,0xfd900000-0xfd901fff at device 0.0
on pci5
vmx2: Using 4096 TX descriptors and 2048 RX descriptors
vmx2: Using 4 RX queues 4 TX queues
vmx2: Using MSI-X interrupts with 5 vectors
vmx2: Ethernet address: 00:50:56:a7:39:49
vmx2: netmap queues/slots: TX 4/4096, RX 4/2048
vmx0: link state changed to UP
vmx1: link state changed to UP
vmx2: link state changed to UP

Are there any more tweaks I can try to improve this?

regards
Michal

-- 
You are receiving this mail because:
You are the assignee for the bug.