From nobody Tue Sep 28 07:12:42 2021 X-Original-To: bugs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 8618F17D0E8B for ; Tue, 28 Sep 2021 07:12:42 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4HJW163HLnz4XGr for ; Tue, 28 Sep 2021 07:12:42 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 50DF920623 for ; Tue, 28 Sep 2021 07:12:42 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 18S7CgLg007347 for ; Tue, 28 Sep 2021 07:12:42 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 18S7Cg7M007346 for bugs@FreeBSD.org; Tue, 28 Sep 2021 07:12:42 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: bugs@FreeBSD.org Subject: [Bug 258755] vmxnet3: low throughput Date: Tue, 28 Sep 2021 07:12:42 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: new X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 13.0-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: michal@maximale.sk X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: bugs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: bug_id short_desc product version rep_platform op_sys bug_status bug_severity priority component assigned_to reporter Message-ID: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Bug reports List-Archive: https://lists.freebsd.org/archives/freebsd-bugs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-bugs@freebsd.org MIME-Version: 1.0 X-ThisMailContainsUnwantedMimeParts: N https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D258755 Bug ID: 258755 Summary: vmxnet3: low throughput Product: Base System Version: 13.0-RELEASE Hardware: amd64 OS: Any Status: New Severity: Affects Some People Priority: --- Component: kern Assignee: bugs@FreeBSD.org Reporter: michal@maximale.sk I've setup a simple throughput test on my ESX 6.7: 10.6.0.10 vmx1 vmx2 10.6.10.10 [ debian 1 ] <=3D=3D> [ freebsd ] <=3D=3D> [ debian 2 ] iperf server router iperf client freebsd is version 13-RELEASE-p4 and both debians are of version 11. When I run iperf test I get suspiciously low throughput: root@debian2:~# iperf3 -t 30 -i 10 -c 10.6.0.10=20 Connecting to host 10.6.0.10, port 5201 [ 5] local 10.6.10.10 port 56006 connected to 10.6.0.10 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-10.00 sec 2.24 GBytes 1.93 Gbits/sec 79 2.36 MBytes=20= =20=20=20=20=20=20 [ 5] 10.00-20.00 sec 2.24 GBytes 1.93 Gbits/sec 2 2.30 MBytes=20= =20=20=20=20=20=20 [ 5] 20.00-30.00 sec 2.27 GBytes 1.95 Gbits/sec 1 2.21 MBytes=20= =20=20=20=20=20=20 - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-30.00 sec 6.76 GBytes 1.94 Gbits/sec 82 sender [ 5] 0.00-30.00 sec 6.76 GBytes 1.93 Gbits/sec recei= ver iperf Done. While test is running, kernel{if_io_tqg_X} is very high (hitting 100% on one CPU). If I run the test with more parallel sessions (iperf -P X), I get hig= her throughput with more kernel{if_io_tqg_X} threads utilizing the CPUs (this is expected as parallel sessions are spread across CPUs).=20 Now If I replace router with some linux (tested with centos 7), I get more = than 10Gbps with single session (all three VMs are running on the same ESX host) with CPU on the linux router doing almost nothing. LRO and TSO are disabled on both vmx adapters. root@freebsd:~ # dmesg | grep vmx vmx0: port 0x4000-0x400f mem 0xfe903000-0xfe903fff,0xfe902000-0xfe902fff,0xfe900000-0xfe901fff at device= 0.0 on pci3 vmx0: Using 4096 TX descriptors and 2048 RX descriptors vmx0: Using 4 RX queues 4 TX queues vmx0: Using MSI-X interrupts with 5 vectors vmx0: Ethernet address: 00:50:56:a7:85:e4 vmx0: netmap queues/slots: TX 4/4096, RX 4/2048 vmx1: port 0x3000-0x300f mem 0xfe103000-0xfe103fff,0xfe102000-0xfe102fff,0xfe100000-0xfe101fff at device= 0.0 on pci4 vmx1: Using 4096 TX descriptors and 2048 RX descriptors vmx1: Using 4 RX queues 4 TX queues vmx1: Using MSI-X interrupts with 5 vectors vmx1: Ethernet address: 00:50:56:a7:1f:01 vmx1: netmap queues/slots: TX 4/4096, RX 4/2048 vmx2: port 0x2000-0x200f mem 0xfd903000-0xfd903fff,0xfd902000-0xfd902fff,0xfd900000-0xfd901fff at device= 0.0 on pci5 vmx2: Using 4096 TX descriptors and 2048 RX descriptors vmx2: Using 4 RX queues 4 TX queues vmx2: Using MSI-X interrupts with 5 vectors vmx2: Ethernet address: 00:50:56:a7:39:49 vmx2: netmap queues/slots: TX 4/4096, RX 4/2048 vmx0: link state changed to UP vmx1: link state changed to UP vmx2: link state changed to UP Are there any more tweaks I can try to improve this? regards Michal --=20 You are receiving this mail because: You are the assignee for the bug.=