Very slow and inconsistent internal network speed (between VM's on the same host) for FreeBSD 11.0+ as guest on XCP-ng/XenServer

[Christian M]

I've noticed very slow networking speed between VM's with FreeBSD on the
same host (XCP-ng 7.6.0) for more recent FreeBSD-versions.


I've made some tests that show me that something happened from 10.4-RELEASE
to 11.0-RELEASE that had a huge impact on network performance, and
12.0-RELEASE is even slower.


My test setup:

Host: XCP-ng 7.6.0, managed with XenOrchestra. Open source.

Network: Internal Private Network on the host (not connected to a PIF).
Each VM has only one VIF connected to this network.

VM's:

2 x 12.0-RELEASE

2 x 11.0-RELEASE

2 x 10.4-RELEASE


All clean identical installs from XenOrchestra, only installed iperf on
each VM for testing. (xe-guest-utilities makes no difference in my tests,
I've tried with and without).


iperf -s on first server listed below, and iperf -c <ip> -r on the second
to test speed back and forth:


12.0 <-> 12.0: 50Mbit as client and server

12.0 <-> 11.0: 800Mbit/s (11.0 as client), and 140Mbit/s (11.0 as server)

12.0 <-> 10.4: 2.76Gbit (10.4 as client), and 1.25Gbit (10.4 as server).

11.0 <-> 11.0: 219Mbit as client, 99Mbit as server

10.4 <-> 10.4: 11.2Gbit as client, 10.9Gbit as server


As I side note, not sure if related, but I've noticed that I can't run
iperf with -r flag on 10.4-RELEASE. I get this error message:


iperf -c 172.31.16.122 -r

------------------------------------------------------------

Server listening on TCP port 5001

TCP window size: 64.0 KByte (default)

------------------------------------------------------------

write failed: Broken pipe

------------------------------------------------------------

Client connecting to 172.31.16.122, TCP port 5001

TCP window size: 32.5 KByte (default)

------------------------------------------------------------

[  5] local 172.31.16.121 port 19231 connected with 172.31.16.122 port 5001

[ ID] Interval       Transfer     Bandwidth

[  5]  0.0- 0.0 sec  0.00 Bytes  0.00 bits/sec



I can run iperf -s fine, and iperf -c <ip> from the other 10.4 VM though:



iperf -c 172.31.16.122

------------------------------------------------------------

Client connecting to 172.31.16.122, TCP port 5001

TCP window size: 32.5 KByte (default)

------------------------------------------------------------

[  3] local 172.31.16.121 port 22055 connected with 172.31.16.122 port 5001

[ ID] Interval       Transfer     Bandwidth

[  3]  0.0-10.0 sec  12.9 GBytes  11.1 Gbits/sec



What have I tried to solve this?

I've tried to disable checksum offloading for the 12.0-RELEASE VIF's via
XCP-ng. Disabled basically everything without any difference in iperf
results: other-config (MRW): ethtool-sg: off; ethtool-tso: off;
ethtool-ufo: off; ethtool-gso: off; ethtool-rx: off; ethtool-tx: off


Also tried disabling offloading in FreeBSD with ifconfig xn0 -txcsum
-rxcsum -tso -lro and no difference here either.


Any ideas of how to proceed now to find a solution for this?


Regards,

Christian