From nobody Fri Jul 01 02:47:08 2022 X-Original-To: freebsd-virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id E977887186D for ; Fri, 1 Jul 2022 02:47:18 +0000 (UTC) (envelope-from jmg@gold.funkthat.com) Received: from gold.funkthat.com (gold.funkthat.com [IPv6:2001:470:800b::2]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "gate2.funkthat.com", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4LZ03T5Y5mz4WRb for ; Fri, 1 Jul 2022 02:47:17 +0000 (UTC) (envelope-from jmg@gold.funkthat.com) Received: from gold.funkthat.com (localhost [127.0.0.1]) by gold.funkthat.com (8.15.2/8.15.2) with ESMTPS id 2612l9Tj080384 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Thu, 30 Jun 2022 19:47:09 -0700 (PDT) (envelope-from jmg@gold.funkthat.com) Received: (from jmg@localhost) by gold.funkthat.com (8.15.2/8.15.2/Submit) id 2612l8GC080381; Thu, 30 Jun 2022 19:47:08 -0700 (PDT) (envelope-from jmg) Date: Thu, 30 Jun 2022 19:47:08 -0700 From: John-Mark Gurney To: Nils Beyer Cc: freebsd-virtualization@FreeBSD.org Subject: Re: bhyve: slow network throughput between guest VM and host (and vice versa)? Message-ID: <20220701024708.GC88842@funkthat.com> References: List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-virtualization@freebsd.org X-BeenThere: freebsd-virtualization@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Operating-System: FreeBSD 11.3-STABLE amd64 X-PGP-Fingerprint: D87A 235F FB71 1F3F 55B7 ED9B D5FF 5A51 C0AC 3D65 X-Files: The truth is out there X-URL: https://www.funkthat.com/ X-Resume: https://www.funkthat.com/~jmg/resume.html X-TipJar: bitcoin:13Qmb6AeTgQecazTWph4XasEsP7nGRbAPE X-to-the-FBI-CIA-and-NSA: HI! HOW YA DOIN? can i haz chizburger? User-Agent: Mutt/1.6.1 (2016-04-27) X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.4.3 (gold.funkthat.com [127.0.0.1]); Thu, 30 Jun 2022 19:47:09 -0700 (PDT) X-Rspamd-Queue-Id: 4LZ03T5Y5mz4WRb X-Spamd-Bar: + Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=none (mx1.freebsd.org: domain of jmg@gold.funkthat.com has no SPF policy when checking 2001:470:800b::2) smtp.mailfrom=jmg@gold.funkthat.com X-Spamd-Result: default: False [1.23 / 15.00]; RCVD_TLS_ALL(0.00)[]; ARC_NA(0.00)[]; FREEFALL_USER(0.00)[jmg]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; MID_RHS_MATCH_FROM(0.00)[]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[funkthat.com]; AUTH_NA(1.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; NEURAL_HAM_SHORT(-0.97)[-0.967]; RCPT_COUNT_TWO(0.00)[2]; NEURAL_SPAM_LONG(1.00)[0.998]; MLMMJ_DEST(0.00)[freebsd-virtualization]; FORGED_SENDER(0.30)[jmg@funkthat.com,jmg@gold.funkthat.com]; R_SPF_NA(0.00)[no SPF record]; SUBJECT_ENDS_QUESTION(1.00)[]; R_DKIM_NA(0.00)[]; ASN(0.00)[asn:6939, ipnet:2001:470::/32, country:US]; MIME_TRACE(0.00)[0:+]; FROM_NEQ_ENVFROM(0.00)[jmg@funkthat.com,jmg@gold.funkthat.com]; RCVD_COUNT_TWO(0.00)[2] X-ThisMailContainsUnwantedMimeParts: N Nils Beyer wrote this message on Thu, Jun 30, 2022 at 12:45 +0200: > Hi, > > I've setup a FreeBSD VM and am using a VirtIO network interface within. Then I've setup > an ip address 192.168.0.2/30 to that VirtIO NIC of the guest using: > > ifconfig vtnet0 192.168.0.2/30 up > > and on the TAP interface of the host an ip address 192.168.0.1/30 using > > ifconfig tap0 192.168.0.1/30 up > > Trying an iperf3-transfer (iperf3-server on the host, iperf3-client on the guest) with a TCP window size of > 128k I only get around 2.45Gbit/s: > > > # env LD_LIBRARY_PATH=. ./iperf3 -c 192.168.0.1 -w 128k > > Connecting to host 192.168.0.1, port 5201 > > [ 5] local 192.168.0.2 port 25651 connected to 192.168.0.1 port 5201 > > [ ID] Interval Transfer Bitrate Retr Cwnd > > [ 5] 0.00-1.00 sec 314 MBytes 2.63 Gbits/sec 0 1.43 MBytes > > [ 5] 1.00-2.00 sec 301 MBytes 2.52 Gbits/sec 0 1.43 MBytes > > [ 5] 2.00-3.00 sec 264 MBytes 2.21 Gbits/sec 0 1.43 MBytes > > [ 5] 3.00-4.00 sec 284 MBytes 2.38 Gbits/sec 0 1.43 MBytes > > [ 5] 4.00-5.00 sec 296 MBytes 2.48 Gbits/sec 0 1.43 MBytes > > [ 5] 5.00-6.00 sec 279 MBytes 2.34 Gbits/sec 0 1.43 MBytes > > [ 5] 6.00-7.00 sec 280 MBytes 2.35 Gbits/sec 0 1.43 MBytes > > [ 5] 7.00-8.00 sec 310 MBytes 2.60 Gbits/sec 0 1.43 MBytes > > [ 5] 8.00-9.00 sec 302 MBytes 2.53 Gbits/sec 0 1.43 MBytes > > [ 5] 9.00-10.00 sec 333 MBytes 2.79 Gbits/sec 0 1.43 MBytes > > - - - - - - - - - - - - - - - - - - - - - - - - - > > [ ID] Interval Transfer Bitrate Retr > > [ 5] 0.00-10.00 sec 2.89 GBytes 2.49 Gbits/sec 0 sender > > [ 5] 0.00-10.00 sec 2.89 GBytes 2.49 Gbits/sec receiver > > > Switching the roles (iperf3-server on the guest, iperf3-client on the host) with a TCP windows size of 128k, > I get 4.04Gbit/s: > > > #iperf3 -c 192.168.0.2 -w 128k > > Connecting to host 192.168.0.2, port 5201 > > [ 5] local 192.168.0.1 port 56892 connected to 192.168.0.2 port 5201 > > [ ID] Interval Transfer Bitrate Retr Cwnd > > [ 5] 0.00-1.00 sec 411 MBytes 3.44 Gbits/sec 40 973 KBytes > > [ 5] 1.00-2.00 sec 483 MBytes 4.05 Gbits/sec 5 1.03 MBytes > > [ 5] 2.00-3.00 sec 507 MBytes 4.26 Gbits/sec 1 1.21 MBytes > > [ 5] 3.00-4.00 sec 514 MBytes 4.31 Gbits/sec 15 561 KBytes > > [ 5] 4.00-5.00 sec 498 MBytes 4.18 Gbits/sec 10 966 KBytes > > [ 5] 5.00-6.00 sec 491 MBytes 4.12 Gbits/sec 19 841 KBytes > > [ 5] 6.00-7.00 sec 513 MBytes 4.31 Gbits/sec 0 1.43 MBytes > > [ 5] 7.00-8.00 sec 504 MBytes 4.23 Gbits/sec 0 1.43 MBytes > > [ 5] 8.00-9.00 sec 459 MBytes 3.85 Gbits/sec 0 1.43 MBytes > > [ 5] 9.00-10.00 sec 435 MBytes 3.65 Gbits/sec 0 1.43 MBytes > > - - - - - - - - - - - - - - - - - - - - - - - - - > > [ ID] Interval Transfer Bitrate Retr > > [ 5] 0.00-10.00 sec 4.70 GBytes 4.04 Gbits/sec 90 sender > > [ 5] 0.00-10.00 sec 4.70 GBytes 4.04 Gbits/sec receiver > > > Increasing MTU on the Virtio interface and on the TAP interface to 9000 helps a little bit: > getting 8.38Gbit/s guest->host and 10.3Gbit/s host->guest. > > Increasing TCP windows size to 1024k only produces more retries and does nothing on the > throughput. > > Is that expected that I'm not able to get more throughput within the bhyve network > stack (guest <-> host)? I was expecting way more then 10Gbit/s... What does CPU usage look like on the host and in the VM? This sounds like hitting a CPU bandwidth limit in that you're limited by pps... -- John-Mark Gurney Voice: +1 415 225 5579 "All that I will do, has been done, All that I have, has not."