Very slow and inconsistent internal network speed (between, VM's on the same host) for FreeBSD 11.0+ as guest on, XCP-ng/XenServer
Bjarne
bjb at darco.dk
Mon Jun 24 15:58:19 UTC 2019
Well, to add some datapoints, I just did a quick test om my private homeserver, testing with iperf3
-s / iperf3 -c <server-ip>
I have not much knowledge of iperf3 network test and it's validity, but there is definitely some
differences to be seen.
Summary:
FBSD <-> FBSD 3,5Gbit/sec
FBSD -> Centos: 3GBit/Sec
Centos -> FBSD: 12 Gbit/sec
Centos <-> Centos: 12 Gbit/sec
OpenSuse <-> OpenSuse 13 GBit/sec
So it ssems there is some fishy when Freebsd is used as client
xen running on opensuse, 11 vm's, freebsd , openSuse and Centos. No vm has been optimized in any
way, everything is out of the box.
===========================
Test 1, Freebsd 12 -> freebsd 12
===========================
Server Freebsd 12:
ccepted connection from 192.168.1.100, port 39518
[ 5] local 192.168.1.99 port 5201 connected to 192.168.1.100 port 48518
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 374 MBytes 3.14 Gbits/sec
[ 5] 1.00-2.00 sec 444 MBytes 3.72 Gbits/sec
[ 5] 2.00-3.00 sec 411 MBytes 3.45 Gbits/sec
[ 5] 3.00-4.00 sec 438 MBytes 3.67 Gbits/sec
[ 5] 4.00-5.00 sec 413 MBytes 3.46 Gbits/sec
[ 5] 5.00-6.00 sec 417 MBytes 3.50 Gbits/sec
[ 5] 6.00-7.00 sec 412 MBytes 3.45 Gbits/sec
[ 5] 7.00-8.00 sec 432 MBytes 3.63 Gbits/sec
[ 5] 8.00-9.00 sec 418 MBytes 3.51 Gbits/sec
[ 5] 9.00-10.00 sec 375 MBytes 3.15 Gbits/sec
[ 5] 10.00-10.00 sec 332 KBytes 2.17 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.00 sec 4.04 GBytes 3.47 Gbits/sec receiver
Client, freebsd 12:
Connecting to host 10.192.168.99, port 5201
[ 5] local 192.168.1.100 port 48518 connected to 192.168.1.99 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 375 MBytes 3.14 Gbits/sec 239 178 KBytes
[ 5] 1.00-2.00 sec 445 MBytes 3.73 Gbits/sec 834 74.1 KBytes
[ 5] 2.00-3.00 sec 411 MBytes 3.45 Gbits/sec 794 38.5 KBytes
[ 5] 3.00-4.00 sec 438 MBytes 3.68 Gbits/sec 808 198 KBytes
[ 5] 4.00-5.00 sec 413 MBytes 3.46 Gbits/sec 777 67.0 KBytes
[ 5] 5.00-6.00 sec 417 MBytes 3.50 Gbits/sec 795 120 KBytes
[ 5] 6.00-7.00 sec 412 MBytes 3.45 Gbits/sec 763 72.7 KBytes
[ 5] 7.00-8.00 sec 432 MBytes 3.62 Gbits/sec 810 329 KBytes
[ 5] 8.00-9.00 sec 418 MBytes 3.51 Gbits/sec 777 219 KBytes
[ 5] 9.00-10.00 sec 375 MBytes 3.15 Gbits/sec 688 304 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 4.04 GBytes 3.47 Gbits/sec 7285 sender
[ 5] 0.00-10.00 sec 4.04 GBytes 3.47 Gbits/sec receiver
==========================
test 2, centos 7.6 -> Freebsd 12
==========================
Server Freebsd 12:
Accepted connection from 192.168.1.162, port 49180
[ 5] local 192.168.1.99 port 5201 connected to 192.168.1.162 port 49182
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 1.17 GBytes 10.0 Gbits/sec
[ 5] 1.00-2.00 sec 1.47 GBytes 12.6 Gbits/sec
[ 5] 2.00-3.00 sec 1.30 GBytes 11.2 Gbits/sec
[ 5] 3.00-4.00 sec 1.32 GBytes 11.3 Gbits/sec
[ 5] 4.00-5.00 sec 1.44 GBytes 12.4 Gbits/sec
[ 5] 5.00-6.00 sec 1.29 GBytes 11.1 Gbits/sec
[ 5] 6.00-7.00 sec 1.32 GBytes 11.3 Gbits/sec
[ 5] 7.00-8.00 sec 1.36 GBytes 11.7 Gbits/sec
[ 5] 8.00-9.00 sec 1.32 GBytes 11.3 Gbits/sec
[ 5] 9.00-10.00 sec 1.41 GBytes 12.1 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.00 sec 13.4 GBytes 11.5 Gbits/sec receiver
Client, centos 7.6:
Connecting to host 192.168.1.99, port 5201
[ 4] local 192.168.1.162 port 49182 connected to 192.168.1.99 port 5201
[ ID] Interval Transfer Bandwidth Retr Cwnd
[ 4] 0.00-1.00 sec 1.17 GBytes 10.1 Gbits/sec 0 822 KBytes
[ 4] 1.00-2.00 sec 1.47 GBytes 12.6 Gbits/sec 0 1.53 MBytes
[ 4] 2.00-3.00 sec 1.30 GBytes 11.2 Gbits/sec 0 1.57 MBytes
[ 4] 3.00-4.00 sec 1.32 GBytes 11.3 Gbits/sec 0 1.64 MBytes
[ 4] 4.00-5.00 sec 1.44 GBytes 12.4 Gbits/sec 0 1.70 MBytes
[ 4] 5.00-6.00 sec 1.29 GBytes 11.1 Gbits/sec 0 1.70 MBytes
[ 4] 6.00-7.00 sec 1.32 GBytes 11.3 Gbits/sec 0 1.70 MBytes
[ 4] 7.00-8.00 sec 1.36 GBytes 11.7 Gbits/sec 0 1.70 MBytes
[ 4] 8.00-9.00 sec 1.32 GBytes 11.3 Gbits/sec 0 1.70 MBytes
[ 4] 9.00-10.00 sec 1.41 GBytes 12.1 Gbits/sec 0 1.70 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth Retr
[ 4] 0.00-10.00 sec 13.4 GBytes 11.5 Gbits/sec 0 sender
[ 4] 0.00-10.00 sec 13.4 GBytes 11.5 Gbits/sec receiver
==========================
test 3, Freebsd 12 -> Centos 7.6
==========================
Server, Centos 7.6:
Accepted connection from 192.168.1.100, port 20309
[ 5] local 192.168.1.155 port 5201 connected to 192.168.1.100 port 17917
[ ID] Interval Transfer Bandwidth
[ 5] 0.00-1.00 sec 335 MBytes 2.81 Gbits/sec
[ 5] 1.00-2.00 sec 379 MBytes 3.18 Gbits/sec
[ 5] 2.00-3.00 sec 368 MBytes 3.08 Gbits/sec
[ 5] 3.00-4.00 sec 371 MBytes 3.11 Gbits/sec
[ 5] 4.00-5.00 sec 374 MBytes 3.14 Gbits/sec
[ 5] 5.00-6.00 sec 362 MBytes 3.04 Gbits/sec
[ 5] 6.00-7.00 sec 381 MBytes 3.19 Gbits/sec
[ 5] 7.00-8.00 sec 384 MBytes 3.22 Gbits/sec
[ 5] 8.00-9.00 sec 374 MBytes 3.14 Gbits/sec
[ 5] 9.00-10.00 sec 366 MBytes 3.07 Gbits/sec
[ 5] 10.00-10.13 sec 56.4 MBytes 3.52 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth
[ 5] 0.00-10.13 sec 0.00 Bytes 0.00 bits/sec sender
[ 5] 0.00-10.13 sec 3.66 GBytes 3.10 Gbits/sec receiver
Client, Freebsd 12:
5] local 192.168.1.100 port 17917 connected to 192.168.1.155 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 387 MBytes 3.25 Gbits/sec 83 309 KBytes
[ 5] 1.00-2.00 sec 380 MBytes 3.19 Gbits/sec 88 11.4 KBytes
[ 5] 2.00-3.00 sec 365 MBytes 3.06 Gbits/sec 83 54.2 KBytes
[ 5] 3.00-4.00 sec 372 MBytes 3.12 Gbits/sec 86 322 KBytes
[ 5] 4.00-5.00 sec 370 MBytes 3.11 Gbits/sec 83 2.85 KBytes
[ 5] 5.00-6.00 sec 366 MBytes 3.07 Gbits/sec 84 298 KBytes
[ 5] 6.00-7.00 sec 382 MBytes 3.20 Gbits/sec 87 356 KBytes
[ 5] 7.00-8.00 sec 380 MBytes 3.19 Gbits/sec 87 369 KBytes
[ 5] 8.00-9.00 sec 374 MBytes 3.14 Gbits/sec 87 29.9 KBytes
[ 5] 9.00-10.00 sec 375 MBytes 3.15 Gbits/sec 86 419 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 3.66 GBytes 3.15 Gbits/sec 854 sender
[ 5] 0.00-10.00 sec 3.66 GBytes 3.15 Gbits/sec receiver
==========================
test 4, Centos 7.6 -> Centos 7.6
==========================
Server:
Accepted connection from 192.168.1.162, port 47606
[ 5] local 192.168.1.155 port 5201 connected to 192.168.1.162 port 47608
[ ID] Interval Transfer Bandwidth
[ 5] 0.00-1.00 sec 1.60 GBytes 13.7 Gbits/sec
[ 5] 1.00-2.00 sec 1.59 GBytes 13.7 Gbits/sec
[ 5] 2.00-3.00 sec 1.60 GBytes 13.8 Gbits/sec
[ 5] 3.00-4.00 sec 1.71 GBytes 14.7 Gbits/sec
[ 5] 4.00-5.00 sec 1.49 GBytes 12.8 Gbits/sec
[ 5] 5.00-6.00 sec 1.38 GBytes 11.9 Gbits/sec
[ 5] 6.00-7.00 sec 1.52 GBytes 13.1 Gbits/sec
[ 5] 7.00-8.00 sec 1.51 GBytes 13.0 Gbits/sec
[ 5] 8.00-9.00 sec 1.51 GBytes 13.0 Gbits/sec
[ 5] 9.00-10.00 sec 1.51 GBytes 12.9 Gbits/sec
[ 5] 10.00-10.03 sec 48.1 MBytes 11.9 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth
[ 5] 0.00-10.03 sec 0.00 Bytes 0.00 bits/sec sender
[ 5] 0.00-10.03 sec 15.5 GBytes 13.2 Gbits/sec receiver
Client, Centos 7.6
Connecting to host 192.168.1.155, port 5201
[ 4] local 192.168.1.162 port 47608 connected to 192.168.1.155 port 5201
[ ID] Interval Transfer Bandwidth Retr Cwnd
[ 4] 0.00-1.00 sec 1.66 GBytes 14.3 Gbits/sec 0 1.41 MBytes
[ 4] 1.00-2.00 sec 1.58 GBytes 13.5 Gbits/sec 0 1.46 MBytes
[ 4] 2.00-3.00 sec 1.61 GBytes 13.9 Gbits/sec 0 1.91 MBytes
[ 4] 3.00-4.00 sec 1.71 GBytes 14.7 Gbits/sec 0 2.13 MBytes
[ 4] 4.00-5.00 sec 1.47 GBytes 12.6 Gbits/sec 0 2.49 MBytes
[ 4] 5.00-6.00 sec 1.39 GBytes 11.9 Gbits/sec 0 2.56 MBytes
[ 4] 6.00-7.00 sec 1.53 GBytes 13.1 Gbits/sec 0 2.71 MBytes
[ 4] 7.00-8.00 sec 1.50 GBytes 12.9 Gbits/sec 0 2.73 MBytes
[ 4] 8.00-9.00 sec 1.50 GBytes 12.9 Gbits/sec 0 2.81 MBytes
[ 4] 9.00-10.00 sec 1.51 GBytes 12.9 Gbits/sec 0 2.84 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth Retr
[ 4] 0.00-10.00 sec 15.5 GBytes 13.3 Gbits/sec 0 sender
[ 4] 0.00-10.00 sec 15.5 GBytes 13.3 Gbits/sec receiver
===============================
test4, Opensuse 15.1 -> OpenSuse 15.1
================================
Server, Opensuse 15.1:
[ 5] local 192.168.1.158 port 5201 connected to 192.168.1.163 port 39334
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 1.51 GBytes 13.0 Gbits/sec
[ 5] 1.00-2.00 sec 1.61 GBytes 13.8 Gbits/sec
[ 5] 2.00-3.00 sec 1.58 GBytes 13.6 Gbits/sec
[ 5] 3.00-4.00 sec 1.58 GBytes 13.6 Gbits/sec
[ 5] 4.00-5.00 sec 1.56 GBytes 13.4 Gbits/sec
[ 5] 5.00-6.00 sec 1.60 GBytes 13.7 Gbits/sec
[ 5] 6.00-7.00 sec 1.58 GBytes 13.5 Gbits/sec
[ 5] 7.00-8.00 sec 1.60 GBytes 13.8 Gbits/sec
[ 5] 8.00-9.00 sec 1.60 GBytes 13.7 Gbits/sec
[ 5] 9.00-10.00 sec 1.61 GBytes 13.8 Gbits/sec
[ 5] 10.00-10.04 sec 68.0 MBytes 14.2 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.04 sec 15.9 GBytes 13.6 Gbits/sec receiver
Client, OpenSuse 15.1:
[ 5] local 192.168.1.163 port 39334 connected to 192.168.1.158 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 1.58 GBytes 13.6 Gbits/sec 0 723 KBytes
[ 5] 1.00-2.00 sec 1.61 GBytes 13.8 Gbits/sec 0 805 KBytes
[ 5] 2.00-3.00 sec 1.57 GBytes 13.5 Gbits/sec 0 926 KBytes
[ 5] 3.00-4.00 sec 1.58 GBytes 13.6 Gbits/sec 0 926 KBytes
[ 5] 4.00-5.00 sec 1.56 GBytes 13.4 Gbits/sec 0 970 KBytes
[ 5] 5.00-6.00 sec 1.60 GBytes 13.7 Gbits/sec 0 1.05 MBytes
[ 5] 6.00-7.00 sec 1.57 GBytes 13.5 Gbits/sec 0 1.05 MBytes
[ 5] 7.00-8.00 sec 1.61 GBytes 13.8 Gbits/sec 0 1.05 MBytes
[ 5] 8.00-9.00 sec 1.59 GBytes 13.7 Gbits/sec 0 1.05 MBytes
[ 5] 9.00-10.00 sec 1.62 GBytes 13.9 Gbits/sec 0 1.05 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 15.9 GBytes 13.7 Gbits/sec 0 sender
[ 5] 0.00-10.04 sec 15.9 GBytes 13.6 Gbits/sec receiver
> I've noticed very slow networking speed between VM's with FreeBSD on the
> same host (XCP-ng 7.6.0) for more recent FreeBSD-versions.
>
>
> I've made some tests that show me that something happened from 10.4-RELEASE
> to 11.0-RELEASE that had a huge impact on network performance, and
> 12.0-RELEASE is even slower.
>
>
> My test setup:
>
> Host: XCP-ng 7.6.0, managed with XenOrchestra. Open source.
>
> Network: Internal Private Network on the host (not connected to a PIF).
> Each VM has only one VIF connected to this network.
>
> VM's:
>
> 2 x 12.0-RELEASE
>
> 2 x 11.0-RELEASE
>
> 2 x 10.4-RELEASE
>
>
> All clean identical installs from XenOrchestra, only installed iperf on
> each VM for testing. (xe-guest-utilities makes no difference in my tests,
> I've tried with and without).
>
>
> iperf -s on first server listed below, and iperf -c <ip> -r on the second
> to test speed back and forth:
>
>
> 12.0 <-> 12.0: 50Mbit as client and server
>
> 12.0 <-> 11.0: 800Mbit/s (11.0 as client), and 140Mbit/s (11.0 as server)
>
> 12.0 <-> 10.4: 2.76Gbit (10.4 as client), and 1.25Gbit (10.4 as server).
>
> 11.0 <-> 11.0: 219Mbit as client, 99Mbit as server
>
> 10.4 <-> 10.4: 11.2Gbit as client, 10.9Gbit as server
>
>
> As I side note, not sure if related, but I've noticed that I can't run
> iperf with -r flag on 10.4-RELEASE. I get this error message:
>
>
> iperf -c 172.31.16.122 -r
>
> ------------------------------------------------------------
>
> Server listening on TCP port 5001
>
> TCP window size: 64.0 KByte (default)
>
> ------------------------------------------------------------
>
> write failed: Broken pipe
>
> ------------------------------------------------------------
>
> Client connecting to 172.31.16.122, TCP port 5001
>
> TCP window size: 32.5 KByte (default)
>
> ------------------------------------------------------------
>
> [ 5] local 172.31.16.121 port 19231 connected with 172.31.16.122 port 5001
>
> [ ID] Interval Transfer Bandwidth
>
> [ 5] 0.0- 0.0 sec 0.00 Bytes 0.00 bits/sec
>
>
>
> I can run iperf -s fine, and iperf -c <ip> from the other 10.4 VM though:
>
>
>
> iperf -c 172.31.16.122
>
> ------------------------------------------------------------
>
> Client connecting to 172.31.16.122, TCP port 5001
>
> TCP window size: 32.5 KByte (default)
>
> ------------------------------------------------------------
>
> [ 3] local 172.31.16.121 port 22055 connected with 172.31.16.122 port 5001
>
> [ ID] Interval Transfer Bandwidth
>
> [ 3] 0.0-10.0 sec 12.9 GBytes 11.1 Gbits/sec
>
>
>
> What have I tried to solve this?
>
> I've tried to disable checksum offloading for the 12.0-RELEASE VIF's via
> XCP-ng. Disabled basically everything without any difference in iperf
> results: other-config (MRW): ethtool-sg: off; ethtool-tso: off;
> ethtool-ufo: off; ethtool-gso: off; ethtool-rx: off; ethtool-tx: off
>
>
> Also tried disabling offloading in FreeBSD with ifconfig xn0 -txcsum
> -rxcsum -tso -lro and no difference here either.
>
>
> Any ideas of how to proceed now to find a solution for this?
>
>
> Regards,
>
> Christian
>
--
mvh,
Bjarne
More information about the freebsd-xen
mailing list