bhyve: slow network throughput between guest VM and host (and vice versa)?

Hi,

I've setup a FreeBSD VM and am using a VirtIO network interface within. Then I've setup
an ip address 192.168.0.2/30 to that VirtIO NIC of the guest using:

	ifconfig vtnet0 192.168.0.2/30 up

and on the TAP interface of the host an ip address 192.168.0.1/30 using

	ifconfig tap0 192.168.0.1/30 up

Trying an iperf3-transfer (iperf3-server on the host, iperf3-client on the guest) with a TCP window size of
128k I only get around 2.45Gbit/s:

> # env LD_LIBRARY_PATH=. ./iperf3 -c 192.168.0.1 -w 128k
> Connecting to host 192.168.0.1, port 5201
> [  5] local 192.168.0.2 port 25651 connected to 192.168.0.1 port 5201
> [ ID] Interval           Transfer     Bitrate         Retr  Cwnd
> [  5]   0.00-1.00   sec   314 MBytes  2.63 Gbits/sec    0   1.43 MBytes       
> [  5]   1.00-2.00   sec   301 MBytes  2.52 Gbits/sec    0   1.43 MBytes       
> [  5]   2.00-3.00   sec   264 MBytes  2.21 Gbits/sec    0   1.43 MBytes       
> [  5]   3.00-4.00   sec   284 MBytes  2.38 Gbits/sec    0   1.43 MBytes       
> [  5]   4.00-5.00   sec   296 MBytes  2.48 Gbits/sec    0   1.43 MBytes       
> [  5]   5.00-6.00   sec   279 MBytes  2.34 Gbits/sec    0   1.43 MBytes       
> [  5]   6.00-7.00   sec   280 MBytes  2.35 Gbits/sec    0   1.43 MBytes       
> [  5]   7.00-8.00   sec   310 MBytes  2.60 Gbits/sec    0   1.43 MBytes       
> [  5]   8.00-9.00   sec   302 MBytes  2.53 Gbits/sec    0   1.43 MBytes       
> [  5]   9.00-10.00  sec   333 MBytes  2.79 Gbits/sec    0   1.43 MBytes       
> - - - - - - - - - - - - - - - - - - - - - - - - -
> [ ID] Interval           Transfer     Bitrate         Retr
> [  5]   0.00-10.00  sec  2.89 GBytes  2.49 Gbits/sec    0             sender
> [  5]   0.00-10.00  sec  2.89 GBytes  2.49 Gbits/sec                  receiver


Switching the roles (iperf3-server on the guest, iperf3-client on the host) with a TCP windows size of 128k,
I get 4.04Gbit/s:

> #iperf3 -c 192.168.0.2 -w 128k
> Connecting to host 192.168.0.2, port 5201
> [  5] local 192.168.0.1 port 56892 connected to 192.168.0.2 port 5201
> [ ID] Interval           Transfer     Bitrate         Retr  Cwnd
> [  5]   0.00-1.00   sec   411 MBytes  3.44 Gbits/sec   40    973 KBytes       
> [  5]   1.00-2.00   sec   483 MBytes  4.05 Gbits/sec    5   1.03 MBytes       
> [  5]   2.00-3.00   sec   507 MBytes  4.26 Gbits/sec    1   1.21 MBytes       
> [  5]   3.00-4.00   sec   514 MBytes  4.31 Gbits/sec   15    561 KBytes       
> [  5]   4.00-5.00   sec   498 MBytes  4.18 Gbits/sec   10    966 KBytes       
> [  5]   5.00-6.00   sec   491 MBytes  4.12 Gbits/sec   19    841 KBytes       
> [  5]   6.00-7.00   sec   513 MBytes  4.31 Gbits/sec    0   1.43 MBytes       
> [  5]   7.00-8.00   sec   504 MBytes  4.23 Gbits/sec    0   1.43 MBytes       
> [  5]   8.00-9.00   sec   459 MBytes  3.85 Gbits/sec    0   1.43 MBytes       
> [  5]   9.00-10.00  sec   435 MBytes  3.65 Gbits/sec    0   1.43 MBytes       
> - - - - - - - - - - - - - - - - - - - - - - - - -
> [ ID] Interval           Transfer     Bitrate         Retr
> [  5]   0.00-10.00  sec  4.70 GBytes  4.04 Gbits/sec   90             sender
> [  5]   0.00-10.00  sec  4.70 GBytes  4.04 Gbits/sec                  receiver


Increasing MTU on the Virtio interface and on the TAP interface to 9000 helps a little bit:
getting 8.38Gbit/s guest->host and 10.3Gbit/s host->guest.

Increasing TCP windows size to 1024k only produces more retries and does nothing on the
throughput.

Is that expected that I'm not able to get more throughput within the bhyve network
stack (guest <-> host)? I was expecting way more then 10Gbit/s...



TIA and KR,
Nils