Major performance/stability regression in virtio network drivers between 9.2-RELEASE and 10.0-RC5
Rick Macklem
rmacklem at uoguelph.ca
Mon Feb 3 23:34:46 UTC 2014
Alfred Perlstein wrote:
> What does vmstat say about memory zones? I think vmstat -m/vmstat -z
> and also netstat -m?
>
> Are you set with the same mbufs on both 9 and 10?
>
> -Alfred
>
>
> On 1/18/14 1:32 PM, Eric Dombroski wrote:
> > Adrian:
> >
> > Yes, no change.
> >
> > -Eric
> >
> >
> >
> >
> >
> > On Sat, Jan 18, 2014 at 4:06 PM, Adrian Chadd
> > <adrian.chadd at gmail.com>wrote:
> >
> >> Hi,
> >>
> >> Have you tried disabling tso?
> >>
> >> Adrian
> >> On Jan 18, 2014 1:52 PM, "Eric Dombroski" <eric at edombroski.com>
> >> wrote:
> >>
> >>> Hello:
> >>>
> >>> I believe there is a major performance regression between FreeBSD
> >>> 9.2-RELEASE and 10.0-RC5 involving the virtio network drivers
> >>> (vtnet) and
> >>> handling incoming traffic. Below are the results of some iperf
> >>> tests and
> >>> large dd operations over NFS. Write throughput goes from ~40Gbps
> >>> to
> >>> ~2.4Gbps from 9.2 to 10.0RC5, and over time the connection
> >>> becomes
> >>> unstable
> >>> ("no buffer space available"), requiring the interface to be
> >>> taken
> >>> down/up.
> >>>
> >>>
> >>> These results are on fresh installs of 9.2 and 10.0RC5, no sysctl
> >>> tweaks
> >>> on
> >>> either system.
> >>>
> >>> I can't reproduce this using an Intel 1Gbps ethernet through PCIe
> >>> passthrough, although I suspect the problem manifests itself over
> >>> 1Gbps
> >>> speeds anyway.
> >>>
> >>> Tests:
> >>>
> >>> Client (host):
> >>> root at gogo:~# uname -a
> >>> Linux gogo 3.2.0-4-amd64 #1 SMP Debian 3.2.51-1 x86_64
> >>> GNU/Linux
> >>> root at gogo:~# kvm -version
> >>> QEMU emulator version 1.1.2 (qemu-kvm-1.1.2+dfsg-6, Debian),
> >>> Copyright
> >>> (c) 2003-2008 Fabrice Bellard
> >>> root at gogo:~# lsmod | grep vhost
> >>> vhost_net 27436 3
> >>> tun 18337 8 vhost_net
> >>> macvtap 17633 1 vhost_net
> >>>
> >>>
> >>> Command: iperf -c 192.168.100.x -t 60
> >>>
> >>>
> >>> Server (FreeBSD 9.2 VM):
> >>>
> >>> root at umarotest:~ # uname -a
> >>> FreeBSD umarotest 9.2-RELEASE-p3 FreeBSD 9.2-RELEASE-p3
> >>> #0: Sat Jan
> >>> 11 03:25:02 UTC 2014
> >>> root at amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC
> >>> amd64
> >>> root at umarotest:~ # iperf -s
> >>> ------------------------------------------------------------
> >>> Server listening on TCP port 5001
> >>> TCP window size: 64.0 KByte (default)
> >>> ------------------------------------------------------------
> >>> [ 4] local 192.168.100.44 port 5001 connected with
> >>> 192.168.100.1
> >>> port 58996
> >>> [ ID] Interval Transfer Bandwidth
> >>> [ 4] 0.0-60.0 sec 293 GBytes 41.9 Gbits/sec
> >>> [ 5] local 192.168.100.44 port 5001 connected with
> >>> 192.168.100.1
> >>> port 58997
> >>> [ 5] 0.0-60.0 sec 297 GBytes 42.5 Gbits/sec
> >>> [ 4] local 192.168.100.44 port 5001 connected with
> >>> 192.168.100.1
> >>> port 58998
> >>> [ 4] 0.0-60.0 sec 291 GBytes 41.6 Gbits/sec
> >>> [ 5] local 192.168.100.44 port 5001 connected with
> >>> 192.168.100.1
> >>> port 58999
> >>> [ 5] 0.0-60.0 sec 297 GBytes 42.6 Gbits/sec
> >>> [ 4] local 192.168.100.44 port 5001 connected with
> >>> 192.168.100.1
> >>> port 59000
> >>> [ 4] 0.0-60.0 sec 297 GBytes 42.5 Gbits/sec
> >>>
> >>> While pinging out from the server to the client, I do not
> >>> get any
> >>> errors.
> >>>
> >>>
> >>> root at umaro:~ # uname -a FreeBSD umaro 10.0-RC5 FreeBSD
> >>> 10.0-RC5 #0
> >>> r260430: Wed Jan 8 05:10:04 UTC 2014
> >>> root at snap.freebsd.org:/usr/obj/usr/src/sys/GENERIC
> >>> amd64
> >>> root at umaro:~ # iperf -s
> >>> ------------------------------------------------------------
> >>> Server listening on TCP port 5001
> >>> TCP window size: 64.0 KByte (default)
> >>> ------------------------------------------------------------
> >>> [ 4] local 192.168.100.5 port 5001 connected with
> >>> 192.168.100.1
> >>> port
> >>> 50264
> >>> [ ID] Interval Transfer Bandwidth
> >>> [ 4] 0.0-60.0 sec 16.7 GBytes 2.39 Gbits/sec
> >>> [ 5] local 192.168.100.5 port 5001 connected with
> >>> 192.168.100.1
> >>> port
> >>> 50265
> >>> [ 5] 0.0-60.0 sec 18.3 GBytes 2.62 Gbits/sec
> >>> [ 4] local 192.168.100.5 port 5001 connected with
> >>> 192.168.100.1
> >>> port
> >>> 50266
> >>> [ 4] 0.0-60.0 sec 16.8 GBytes 2.40 Gbits/sec
> >>> [ 5] local 192.168.100.5 port 5001 connected with
> >>> 192.168.100.1
> >>> port
> >>> 50267
> >>> [ 5] 0.0-60.0 sec 16.8 GBytes 2.40 Gbits/sec
> >>> [ 4] local 192.168.100.5 port 5001 connected with
> >>> 192.168.100.1
> >>> port
> >>> 50268
> >>> [ 4] 0.0-60.0 sec 16.8 GBytes 2.41 Gbits/sec
> >>>
> >>> *** While pinging out from the server to client, frequent
> >>> "ping:
> >>> sendto: No space left on device" errors ***
> >>>
> >>>
> >>> After a while, I can also reliably re-produce more
> >>> egregious "ping:
> >>> sendto: No buffer space available" errors after doing a large
> >>> sequential
> >>> write over NFS:
> >>>
> >>> mount -t nfs -o rsize=65536,wsize=65536 192.168.100.5:
> >>> /storage/shared
> >>> /mnt/nfs
> >>> dd if=/dev/zero of=/mnt/nfs/testfile bs=1M count=30000
> >>>
> >>> I am going to file a freebsd bug report as well.
> >>>
Are you able to test the driver from an up to date head?
Bryan Venteicher just committed some changes to the driver
that increase the size of the segments list (# of mbufs in chain)
and also switches it from using m_collapse() to m_defrag().
I have no idea if these might affect what you are seeing?
rick
> >>> Thanks,
> >>> Eric
> >>> _______________________________________________
> >>> freebsd-stable at freebsd.org mailing list
> >>> http://lists.freebsd.org/mailman/listinfo/freebsd-stable
> >>> To unsubscribe, send any mail to
> >>> "freebsd-stable-unsubscribe at freebsd.org"
> >>>
> > _______________________________________________
> > freebsd-stable at freebsd.org mailing list
> > http://lists.freebsd.org/mailman/listinfo/freebsd-stable
> > To unsubscribe, send any mail to
> > "freebsd-stable-unsubscribe at freebsd.org"
> >
>
> _______________________________________________
> freebsd-stable at freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-stable
> To unsubscribe, send any mail to
> "freebsd-stable-unsubscribe at freebsd.org"
>
More information about the freebsd-stable
mailing list