From nobody Mon Oct 31 03:59:53 2022 X-Original-To: freebsd-virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4N0zvB1n7lz4gW1v for ; Mon, 31 Oct 2022 04:00:06 +0000 (UTC) (envelope-from bsdlists@jld3.net) Received: from mail.jld3.net (mail.jld3.net [45.55.236.93]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4N0zv93WgZz3fsS for ; Mon, 31 Oct 2022 04:00:05 +0000 (UTC) (envelope-from bsdlists@jld3.net) Received: from localhost (localhost [127.0.0.1]) by mail.jld3.net (Postfix) with ESMTP id 96FAD4162B for ; Sun, 30 Oct 2022 21:59:59 -0600 (MDT) X-Virus-Scanned: amavisd-new at jld3.net Received: from mail.jld3.net ([127.0.0.1]) by localhost (mail.jld3.net [127.0.0.1]) (amavisd-new, port 10024) with LMTP id qcoujMtK6E1K for ; Sun, 30 Oct 2022 21:59:58 -0600 (MDT) Received: from [172.21.35.250] (c-24-9-144-115.hsd1.co.comcast.net [24.9.144.115]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) (Authenticated sender: jld) by mail.jld3.net (Postfix) with ESMTPSA id 73E9940455 for ; Sun, 30 Oct 2022 21:59:58 -0600 (MDT) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.jld3.net 73E9940455 DKIM-Filter: OpenDKIM Filter v2.11.0 mail.jld3.net 73E9940455 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=jld3.net; s=8d052f02dde2; t=1667188798; bh=We5zcfHJzxDCL5Tu8gdUfg8J8PhUbaBVFtkWbTZPzAs=; h=From:To:Subject:Date:Message-ID:MIME-Version:From; b=yc0h605TTax6bIHr+kC1zT/E3EiE/g1zjydaku9qOFH5zRhScDGJXsMgAoN2TMwiU FcH4IXn5NoUktM3NN5NP3ShcHlSP+G1q4S1qYN3jxOatUq2FBrc40o3AX0rFjDqSFU 5oGIbx0TyNToDY8OSqyLYlXfiYBHkmmJyZyII0Gg= From: "John Doherty" To: "FreeBSD virtualization" Subject: NFS in bhyve VM mounted via bridge interface Date: Sun, 30 Oct 2022 21:59:53 -0600 X-Mailer: MailMate (1.13.2r5673) Message-ID: List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-virtualization@freebsd.org X-BeenThere: freebsd-virtualization@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; format=flowed X-Rspamd-Queue-Id: 4N0zv93WgZz3fsS X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=jld3.net header.s=8d052f02dde2 header.b=yc0h605T; dmarc=pass (policy=quarantine) header.from=jld3.net; spf=pass (mx1.freebsd.org: domain of bsdlists@jld3.net designates 45.55.236.93 as permitted sender) smtp.mailfrom=bsdlists@jld3.net X-Spamd-Result: default: False [-4.00 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-0.999]; DMARC_POLICY_ALLOW(-0.50)[jld3.net,quarantine]; R_SPF_ALLOW(-0.20)[+ip4:45.55.236.93]; R_DKIM_ALLOW(-0.20)[jld3.net:s=8d052f02dde2]; MIME_GOOD(-0.10)[text/plain]; MLMMJ_DEST(0.00)[freebsd-virtualization@freebsd.org]; RCVD_VIA_SMTP_AUTH(0.00)[]; MIME_TRACE(0.00)[0:+]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; DKIM_TRACE(0.00)[jld3.net:+]; RCPT_COUNT_ONE(0.00)[1]; RCVD_COUNT_THREE(0.00)[4]; FROM_HAS_DN(0.00)[]; ARC_NA(0.00)[]; TO_DN_ALL(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; ASN(0.00)[asn:14061, ipnet:45.55.192.0/18, country:US]; PREVIOUSLY_DELIVERED(0.00)[freebsd-virtualization@freebsd.org]; MID_RHS_MATCH_FROM(0.00)[] X-ThisMailContainsUnwantedMimeParts: N I have a machine running FreeBSD 12.3-RELEASE with a zpool that consists of 12 mirrored pairs of 14 TB disks. I'll call this the "storage server." On that machine, I can write to ZFS file systems at around 950 MB/s and read from them at around 1450 MB/s. I'm happy with that. I have another machine running Alma linux 8.6 that mounts file systems from the storage server via NFS over a 10 GbE network. On this machine, I can write to and read from an NFS file system at around 450 MB/s. I wish that this were better but it's OK. I created a bhyve VM on the storage server that also runs Alma linux 8.6. It has a vNIC that is bridged with the 10 GbE physical NIC and a tap interface: [root@ss3] # ifconfig vm-storage vm-storage: flags=8843 metric 0 mtu 1500 ether 82:d3:46:17:4e:ee id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15 maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200 root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0 member: tap1 flags=143 ifmaxaddr 0 port 10 priority 128 path cost 2000000 member: ixl0 flags=143 ifmaxaddr 0 port 5 priority 128 path cost 2000 groups: bridge vm-switch viid-ddece@ nd6 options=1 I mount file systems from the storage server on this VM via NFS. I can write to those file systems at around 250 MB/s and read from them at around 280 MB/s. This surprised me a little: I thought that this might perform better than or at least as well as the physical 10 GbE network but find that it performs significantly worse. All my read and write tests here are stupidly simple, using dd to read from /dev/zero and write to a file or to read from a file and write to /dev/null. Is anyone else either surprised or unsurprised by these results? I have not yet tried passing a physical interface on the storage server through to the VM with PCI passthrough, but the machine does have another 10 GbE interface I could use for this. This stuff is all about 3,200 miles away from me so I need to get someone to plug a cable in for me. I'll be interested to see how that works out, though. Any comments much appreciated. Thanks.