Very slow and inconsistent internal network speed (between, VM's on the same host) for FreeBSD 11.0+ as guest on, XCP-ng/XenServer
Christian M
christian.marcos at gmail.com
Mon Jun 24 20:24:24 UTC 2019
Thanks for your input Bjarne.
Previously I tested with iperf2, but I have made som tests with iperf3 now
also and noticed there are a lot of "Retr" (TCP retries) in some cases.
Went back here to your post and saw that your results also showed a lot of
retries in some cases. My new tests showed similar results to yours
(although not nearly as hi throughput as you have). When Linux is client
the retries are 0, with FreeBSD as client the number is > 0. I'm not sure
if our numbers are considered high though, and something that actually is a
problem? Although, on a internal network with no external factors like
interference, I feel this should always be 0 no matter what?
What hypervisor were you running in your tests? Version?
Regards,
Christian
Den mån 24 juni 2019 kl 17:56 skrev Bjarne <bjb at darco.dk>:
> Well, to add some datapoints, I just did a quick test om my private
> homeserver, testing with iperf3
> -s / iperf3 -c <server-ip>
>
> I have not much knowledge of iperf3 network test and it's validity, but
> there is definitely some
> differences to be seen.
>
> Summary:
>
> FBSD <-> FBSD 3,5Gbit/sec
> FBSD -> Centos: 3GBit/Sec
> Centos -> FBSD: 12 Gbit/sec
>
> Centos <-> Centos: 12 Gbit/sec
> OpenSuse <-> OpenSuse 13 GBit/sec
>
> So it ssems there is some fishy when Freebsd is used as client
>
> xen running on opensuse, 11 vm's, freebsd , openSuse and Centos. No vm
> has been optimized in any
> way, everything is out of the box.
>
>
> ===========================
> Test 1, Freebsd 12 -> freebsd 12
> ===========================
>
> Server Freebsd 12:
> ccepted connection from 192.168.1.100, port 39518
> [ 5] local 192.168.1.99 port 5201 connected to 192.168.1.100 port 48518
> [ ID] Interval Transfer Bitrate
> [ 5] 0.00-1.00 sec 374 MBytes 3.14 Gbits/sec
> [ 5] 1.00-2.00 sec 444 MBytes 3.72 Gbits/sec
> [ 5] 2.00-3.00 sec 411 MBytes 3.45 Gbits/sec
> [ 5] 3.00-4.00 sec 438 MBytes 3.67 Gbits/sec
> [ 5] 4.00-5.00 sec 413 MBytes 3.46 Gbits/sec
> [ 5] 5.00-6.00 sec 417 MBytes 3.50 Gbits/sec
> [ 5] 6.00-7.00 sec 412 MBytes 3.45 Gbits/sec
> [ 5] 7.00-8.00 sec 432 MBytes 3.63 Gbits/sec
> [ 5] 8.00-9.00 sec 418 MBytes 3.51 Gbits/sec
> [ 5] 9.00-10.00 sec 375 MBytes 3.15 Gbits/sec
> [ 5] 10.00-10.00 sec 332 KBytes 2.17 Gbits/sec
> - - - - - - - - - - - - - - - - - - - - - - - - -
> [ ID] Interval Transfer Bitrate
> [ 5] 0.00-10.00 sec 4.04 GBytes 3.47 Gbits/sec
> receiver
>
> Client, freebsd 12:
> Connecting to host 10.192.168.99, port 5201
> [ 5] local 192.168.1.100 port 48518 connected to 192.168.1.99 port 5201
> [ ID] Interval Transfer Bitrate Retr Cwnd
> [ 5] 0.00-1.00 sec 375 MBytes 3.14 Gbits/sec 239 178 KBytes
> [ 5] 1.00-2.00 sec 445 MBytes 3.73 Gbits/sec 834 74.1 KBytes
> [ 5] 2.00-3.00 sec 411 MBytes 3.45 Gbits/sec 794 38.5 KBytes
> [ 5] 3.00-4.00 sec 438 MBytes 3.68 Gbits/sec 808 198 KBytes
> [ 5] 4.00-5.00 sec 413 MBytes 3.46 Gbits/sec 777 67.0 KBytes
> [ 5] 5.00-6.00 sec 417 MBytes 3.50 Gbits/sec 795 120 KBytes
> [ 5] 6.00-7.00 sec 412 MBytes 3.45 Gbits/sec 763 72.7 KBytes
> [ 5] 7.00-8.00 sec 432 MBytes 3.62 Gbits/sec 810 329 KBytes
> [ 5] 8.00-9.00 sec 418 MBytes 3.51 Gbits/sec 777 219 KBytes
> [ 5] 9.00-10.00 sec 375 MBytes 3.15 Gbits/sec 688 304 KBytes
> - - - - - - - - - - - - - - - - - - - - - - - - -
> [ ID] Interval Transfer Bitrate Retr
> [ 5] 0.00-10.00 sec 4.04 GBytes 3.47 Gbits/sec 7285
> sender
> [ 5] 0.00-10.00 sec 4.04 GBytes 3.47 Gbits/sec
> receiver
>
>
> ==========================
> test 2, centos 7.6 -> Freebsd 12
> ==========================
>
> Server Freebsd 12:
> Accepted connection from 192.168.1.162, port 49180
> [ 5] local 192.168.1.99 port 5201 connected to 192.168.1.162 port 49182
> [ ID] Interval Transfer Bitrate
> [ 5] 0.00-1.00 sec 1.17 GBytes 10.0 Gbits/sec
> [ 5] 1.00-2.00 sec 1.47 GBytes 12.6 Gbits/sec
> [ 5] 2.00-3.00 sec 1.30 GBytes 11.2 Gbits/sec
> [ 5] 3.00-4.00 sec 1.32 GBytes 11.3 Gbits/sec
> [ 5] 4.00-5.00 sec 1.44 GBytes 12.4 Gbits/sec
> [ 5] 5.00-6.00 sec 1.29 GBytes 11.1 Gbits/sec
> [ 5] 6.00-7.00 sec 1.32 GBytes 11.3 Gbits/sec
> [ 5] 7.00-8.00 sec 1.36 GBytes 11.7 Gbits/sec
> [ 5] 8.00-9.00 sec 1.32 GBytes 11.3 Gbits/sec
> [ 5] 9.00-10.00 sec 1.41 GBytes 12.1 Gbits/sec
> - - - - - - - - - - - - - - - - - - - - - - - - -
> [ ID] Interval Transfer Bitrate
> [ 5] 0.00-10.00 sec 13.4 GBytes 11.5 Gbits/sec
> receiver
>
> Client, centos 7.6:
> Connecting to host 192.168.1.99, port 5201
> [ 4] local 192.168.1.162 port 49182 connected to 192.168.1.99 port 5201
> [ ID] Interval Transfer Bandwidth Retr Cwnd
> [ 4] 0.00-1.00 sec 1.17 GBytes 10.1 Gbits/sec 0 822 KBytes
> [ 4] 1.00-2.00 sec 1.47 GBytes 12.6 Gbits/sec 0 1.53 MBytes
> [ 4] 2.00-3.00 sec 1.30 GBytes 11.2 Gbits/sec 0 1.57 MBytes
> [ 4] 3.00-4.00 sec 1.32 GBytes 11.3 Gbits/sec 0 1.64 MBytes
> [ 4] 4.00-5.00 sec 1.44 GBytes 12.4 Gbits/sec 0 1.70 MBytes
> [ 4] 5.00-6.00 sec 1.29 GBytes 11.1 Gbits/sec 0 1.70 MBytes
> [ 4] 6.00-7.00 sec 1.32 GBytes 11.3 Gbits/sec 0 1.70 MBytes
> [ 4] 7.00-8.00 sec 1.36 GBytes 11.7 Gbits/sec 0 1.70 MBytes
> [ 4] 8.00-9.00 sec 1.32 GBytes 11.3 Gbits/sec 0 1.70 MBytes
> [ 4] 9.00-10.00 sec 1.41 GBytes 12.1 Gbits/sec 0 1.70 MBytes
> - - - - - - - - - - - - - - - - - - - - - - - - -
> [ ID] Interval Transfer Bandwidth Retr
> [ 4] 0.00-10.00 sec 13.4 GBytes 11.5 Gbits/sec 0 sender
> [ 4] 0.00-10.00 sec 13.4 GBytes 11.5 Gbits/sec
> receiver
>
>
> ==========================
> test 3, Freebsd 12 -> Centos 7.6
> ==========================
>
> Server, Centos 7.6:
> Accepted connection from 192.168.1.100, port 20309
> [ 5] local 192.168.1.155 port 5201 connected to 192.168.1.100 port 17917
> [ ID] Interval Transfer Bandwidth
> [ 5] 0.00-1.00 sec 335 MBytes 2.81 Gbits/sec
> [ 5] 1.00-2.00 sec 379 MBytes 3.18 Gbits/sec
> [ 5] 2.00-3.00 sec 368 MBytes 3.08 Gbits/sec
> [ 5] 3.00-4.00 sec 371 MBytes 3.11 Gbits/sec
> [ 5] 4.00-5.00 sec 374 MBytes 3.14 Gbits/sec
> [ 5] 5.00-6.00 sec 362 MBytes 3.04 Gbits/sec
> [ 5] 6.00-7.00 sec 381 MBytes 3.19 Gbits/sec
> [ 5] 7.00-8.00 sec 384 MBytes 3.22 Gbits/sec
> [ 5] 8.00-9.00 sec 374 MBytes 3.14 Gbits/sec
> [ 5] 9.00-10.00 sec 366 MBytes 3.07 Gbits/sec
> [ 5] 10.00-10.13 sec 56.4 MBytes 3.52 Gbits/sec
> - - - - - - - - - - - - - - - - - - - - - - - - -
> [ ID] Interval Transfer Bandwidth
> [ 5] 0.00-10.13 sec 0.00 Bytes 0.00 bits/sec sender
> [ 5] 0.00-10.13 sec 3.66 GBytes 3.10 Gbits/sec
> receiver
>
> Client, Freebsd 12:
> 5] local 192.168.1.100 port 17917 connected to 192.168.1.155 port 5201
> [ ID] Interval Transfer Bitrate Retr Cwnd
> [ 5] 0.00-1.00 sec 387 MBytes 3.25 Gbits/sec 83 309 KBytes
> [ 5] 1.00-2.00 sec 380 MBytes 3.19 Gbits/sec 88 11.4 KBytes
> [ 5] 2.00-3.00 sec 365 MBytes 3.06 Gbits/sec 83 54.2 KBytes
> [ 5] 3.00-4.00 sec 372 MBytes 3.12 Gbits/sec 86 322 KBytes
> [ 5] 4.00-5.00 sec 370 MBytes 3.11 Gbits/sec 83 2.85 KBytes
> [ 5] 5.00-6.00 sec 366 MBytes 3.07 Gbits/sec 84 298 KBytes
> [ 5] 6.00-7.00 sec 382 MBytes 3.20 Gbits/sec 87 356 KBytes
> [ 5] 7.00-8.00 sec 380 MBytes 3.19 Gbits/sec 87 369 KBytes
> [ 5] 8.00-9.00 sec 374 MBytes 3.14 Gbits/sec 87 29.9 KBytes
> [ 5] 9.00-10.00 sec 375 MBytes 3.15 Gbits/sec 86 419 KBytes
> - - - - - - - - - - - - - - - - - - - - - - - - -
> [ ID] Interval Transfer Bitrate Retr
> [ 5] 0.00-10.00 sec 3.66 GBytes 3.15 Gbits/sec 854 sender
> [ 5] 0.00-10.00 sec 3.66 GBytes 3.15 Gbits/sec
> receiver
>
>
> ==========================
> test 4, Centos 7.6 -> Centos 7.6
> ==========================
>
> Server:
> Accepted connection from 192.168.1.162, port 47606
> [ 5] local 192.168.1.155 port 5201 connected to 192.168.1.162 port 47608
> [ ID] Interval Transfer Bandwidth
> [ 5] 0.00-1.00 sec 1.60 GBytes 13.7 Gbits/sec
> [ 5] 1.00-2.00 sec 1.59 GBytes 13.7 Gbits/sec
> [ 5] 2.00-3.00 sec 1.60 GBytes 13.8 Gbits/sec
> [ 5] 3.00-4.00 sec 1.71 GBytes 14.7 Gbits/sec
> [ 5] 4.00-5.00 sec 1.49 GBytes 12.8 Gbits/sec
> [ 5] 5.00-6.00 sec 1.38 GBytes 11.9 Gbits/sec
> [ 5] 6.00-7.00 sec 1.52 GBytes 13.1 Gbits/sec
> [ 5] 7.00-8.00 sec 1.51 GBytes 13.0 Gbits/sec
> [ 5] 8.00-9.00 sec 1.51 GBytes 13.0 Gbits/sec
> [ 5] 9.00-10.00 sec 1.51 GBytes 12.9 Gbits/sec
> [ 5] 10.00-10.03 sec 48.1 MBytes 11.9 Gbits/sec
> - - - - - - - - - - - - - - - - - - - - - - - - -
> [ ID] Interval Transfer Bandwidth
> [ 5] 0.00-10.03 sec 0.00 Bytes 0.00 bits/sec sender
> [ 5] 0.00-10.03 sec 15.5 GBytes 13.2 Gbits/sec
> receiver
>
> Client, Centos 7.6
> Connecting to host 192.168.1.155, port 5201
> [ 4] local 192.168.1.162 port 47608 connected to 192.168.1.155 port 5201
> [ ID] Interval Transfer Bandwidth Retr Cwnd
> [ 4] 0.00-1.00 sec 1.66 GBytes 14.3 Gbits/sec 0 1.41 MBytes
> [ 4] 1.00-2.00 sec 1.58 GBytes 13.5 Gbits/sec 0 1.46 MBytes
> [ 4] 2.00-3.00 sec 1.61 GBytes 13.9 Gbits/sec 0 1.91 MBytes
> [ 4] 3.00-4.00 sec 1.71 GBytes 14.7 Gbits/sec 0 2.13 MBytes
> [ 4] 4.00-5.00 sec 1.47 GBytes 12.6 Gbits/sec 0 2.49 MBytes
> [ 4] 5.00-6.00 sec 1.39 GBytes 11.9 Gbits/sec 0 2.56 MBytes
> [ 4] 6.00-7.00 sec 1.53 GBytes 13.1 Gbits/sec 0 2.71 MBytes
> [ 4] 7.00-8.00 sec 1.50 GBytes 12.9 Gbits/sec 0 2.73 MBytes
> [ 4] 8.00-9.00 sec 1.50 GBytes 12.9 Gbits/sec 0 2.81 MBytes
> [ 4] 9.00-10.00 sec 1.51 GBytes 12.9 Gbits/sec 0 2.84 MBytes
> - - - - - - - - - - - - - - - - - - - - - - - - -
> [ ID] Interval Transfer Bandwidth Retr
> [ 4] 0.00-10.00 sec 15.5 GBytes 13.3 Gbits/sec 0 sender
> [ 4] 0.00-10.00 sec 15.5 GBytes 13.3 Gbits/sec
> receiver
>
>
> ===============================
> test4, Opensuse 15.1 -> OpenSuse 15.1
> ================================
>
> Server, Opensuse 15.1:
> [ 5] local 192.168.1.158 port 5201 connected to 192.168.1.163 port 39334
> [ ID] Interval Transfer Bitrate
> [ 5] 0.00-1.00 sec 1.51 GBytes 13.0 Gbits/sec
> [ 5] 1.00-2.00 sec 1.61 GBytes 13.8 Gbits/sec
> [ 5] 2.00-3.00 sec 1.58 GBytes 13.6 Gbits/sec
> [ 5] 3.00-4.00 sec 1.58 GBytes 13.6 Gbits/sec
> [ 5] 4.00-5.00 sec 1.56 GBytes 13.4 Gbits/sec
> [ 5] 5.00-6.00 sec 1.60 GBytes 13.7 Gbits/sec
> [ 5] 6.00-7.00 sec 1.58 GBytes 13.5 Gbits/sec
> [ 5] 7.00-8.00 sec 1.60 GBytes 13.8 Gbits/sec
> [ 5] 8.00-9.00 sec 1.60 GBytes 13.7 Gbits/sec
> [ 5] 9.00-10.00 sec 1.61 GBytes 13.8 Gbits/sec
> [ 5] 10.00-10.04 sec 68.0 MBytes 14.2 Gbits/sec
> - - - - - - - - - - - - - - - - - - - - - - - - -
> [ ID] Interval Transfer Bitrate
> [ 5] 0.00-10.04 sec 15.9 GBytes 13.6 Gbits/sec
> receiver
>
> Client, OpenSuse 15.1:
> [ 5] local 192.168.1.163 port 39334 connected to 192.168.1.158 port 5201
> [ ID] Interval Transfer Bitrate Retr Cwnd
> [ 5] 0.00-1.00 sec 1.58 GBytes 13.6 Gbits/sec 0 723 KBytes
> [ 5] 1.00-2.00 sec 1.61 GBytes 13.8 Gbits/sec 0 805 KBytes
> [ 5] 2.00-3.00 sec 1.57 GBytes 13.5 Gbits/sec 0 926 KBytes
> [ 5] 3.00-4.00 sec 1.58 GBytes 13.6 Gbits/sec 0 926 KBytes
> [ 5] 4.00-5.00 sec 1.56 GBytes 13.4 Gbits/sec 0 970 KBytes
> [ 5] 5.00-6.00 sec 1.60 GBytes 13.7 Gbits/sec 0 1.05 MBytes
> [ 5] 6.00-7.00 sec 1.57 GBytes 13.5 Gbits/sec 0 1.05 MBytes
> [ 5] 7.00-8.00 sec 1.61 GBytes 13.8 Gbits/sec 0 1.05 MBytes
> [ 5] 8.00-9.00 sec 1.59 GBytes 13.7 Gbits/sec 0 1.05 MBytes
> [ 5] 9.00-10.00 sec 1.62 GBytes 13.9 Gbits/sec 0 1.05 MBytes
> - - - - - - - - - - - - - - - - - - - - - - - - -
> [ ID] Interval Transfer Bitrate Retr
> [ 5] 0.00-10.00 sec 15.9 GBytes 13.7 Gbits/sec 0 sender
> [ 5] 0.00-10.04 sec 15.9 GBytes 13.6 Gbits/sec
> receiver
>
>
>
>
>
>
>
More information about the freebsd-xen
mailing list