slow network performance in bhyve with freebsd guests compared with any other guest os

From: void <void_at_f-m.fm>
Date: Sun, 29 Sep 2024 06:58:43 UTC
Surprisingly, freebsd guest performance is about 1/3rd of the line speed.
Do some sysctls need to be tuned in freebsd specifically for when it is in a 
guest context?

The host is 15.0-CURRENT (GENERIC-NODEBUG) #1 n271832-04262ed78d23
Xeon E5-2690 @ 2.90GHz with 128GB RAM and the guests are all on a ssd zpool.

iperf3 from the guests to a freebsd 14-stable workstation via a 1G switch.
The network is all 1GB

iperf3 --time 30 -c desktop

freebsd15-testvm guest
======================
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-30.05 sec 1.36 GBytes 389 Mbits/sec 0 sender
[ 5] 0.00-30.05 sec 1.36 GBytes 389 Mbits/sec receiver

then, with kldload tcp_rack and then sysctl net.inet.tcp.functions_default=rack

[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-30.01 sec 1.47 GBytes 420 Mbits/sec 0 sender
[ 5] 0.00-30.02 sec 1.47 GBytes 420 Mbits/sec receiver

freebsd-14.1-p5 vm guest
========================
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-30.07 sec 1.19 GBytes 340 Mbits/sec 0 sender
[ 5] 0.00-30.07 sec 1.19 GBytes 340 Mbits/sec receiver

then, with kldload tcp_rack and then sysctl net.inet.tcp.functions_default=rack

[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-30.03 sec 1.09 GBytes 313 Mbits/sec 0 sender
[ 5] 0.00-30.04 sec 1.09 GBytes 313 Mbits/sec receiver

tcp_rack seems to have made it slower in 14-p5, quicker in -current

from bhyve host, baremetal
==========================
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-30.01 sec 3.29 GBytes 941 Mbits/sec 3 sender
[ 5] 0.00-30.01 sec 3.29 GBytes 941 Mbits/sec receiver

compare with (from) Ubuntu 24.04.1 LTS guest:
=============================================
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-30.00 sec 3.27 GBytes 935 Mbits/sec 0 sender
[ 5] 0.00-30.01 sec 3.27 GBytes 935 Mbits/sec receiver

alpine linux guest:
===================
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-30.01  sec  3.29 GBytes   941 Mbits/sec  0 sender
[  5]   0.00-30.01  sec  3.29 GBytes   941 Mbits/sec receiver

--