Major performance/stability regression in virtio network drivers between 9.2-RELEASE and 10.0-RC5
Bryan Venteicher
bryanv at freebsd.org
Tue Feb 4 06:23:43 UTC 2014
On Mon, Feb 3, 2014 at 5:34 PM, Rick Macklem <rmacklem at uoguelph.ca> wrote:
> Alfred Perlstein wrote:
> > What does vmstat say about memory zones? I think vmstat -m/vmstat -z
> > and also netstat -m?
> >
> > Are you set with the same mbufs on both 9 and 10?
> >
> > -Alfred
> >
> >
> > On 1/18/14 1:32 PM, Eric Dombroski wrote:
> > > Adrian:
> > >
> > > Yes, no change.
> > >
> > > -Eric
> > >
> > >
> > >
> > >
> > >
> > > On Sat, Jan 18, 2014 at 4:06 PM, Adrian Chadd
> > > <adrian.chadd at gmail.com>wrote:
> > >
> > >> Hi,
> > >>
> > >> Have you tried disabling tso?
> > >>
> > >> Adrian
> > >> On Jan 18, 2014 1:52 PM, "Eric Dombroski" <eric at edombroski.com>
> > >> wrote:
> > >>
> > >>> Hello:
> > >>>
> > >>> I believe there is a major performance regression between FreeBSD
> > >>> 9.2-RELEASE and 10.0-RC5 involving the virtio network drivers
> > >>> (vtnet) and
> > >>> handling incoming traffic. Below are the results of some iperf
> > >>> tests and
> > >>> large dd operations over NFS. Write throughput goes from ~40Gbps
> > >>> to
> > >>> ~2.4Gbps from 9.2 to 10.0RC5, and over time the connection
> > >>> becomes
> > >>> unstable
> > >>> ("no buffer space available"), requiring the interface to be
> > >>> taken
> > >>> down/up.
> > >>>
> > >>>
> > >>> These results are on fresh installs of 9.2 and 10.0RC5, no sysctl
> > >>> tweaks
> > >>> on
> > >>> either system.
> > >>>
> > >>> I can't reproduce this using an Intel 1Gbps ethernet through PCIe
> > >>> passthrough, although I suspect the problem manifests itself over
> > >>> 1Gbps
> > >>> speeds anyway.
> > >>>
> > >>> Tests:
> > >>>
> > >>> Client (host):
> > >>> root at gogo:~# uname -a
> > >>> Linux gogo 3.2.0-4-amd64 #1 SMP Debian 3.2.51-1 x86_64
> > >>> GNU/Linux
> > >>> root at gogo:~# kvm -version
> > >>> QEMU emulator version 1.1.2 (qemu-kvm-1.1.2+dfsg-6, Debian),
> > >>> Copyright
> > >>> (c) 2003-2008 Fabrice Bellard
> > >>> root at gogo:~# lsmod | grep vhost
> > >>> vhost_net 27436 3
> > >>> tun 18337 8 vhost_net
> > >>> macvtap 17633 1 vhost_net
> > >>>
> > >>>
> > >>> Command: iperf -c 192.168.100.x -t 60
> > >>>
> > >>>
> > >>> Server (FreeBSD 9.2 VM):
> > >>>
> > >>> root at umarotest:~ # uname -a
> > >>> FreeBSD umarotest 9.2-RELEASE-p3 FreeBSD 9.2-RELEASE-p3
> > >>> #0: Sat Jan
> > >>> 11 03:25:02 UTC 2014
> > >>> root at amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC
> > >>> amd64
> > >>> root at umarotest:~ # iperf -s
> > >>> ------------------------------------------------------------
> > >>> Server listening on TCP port 5001
> > >>> TCP window size: 64.0 KByte (default)
> > >>> ------------------------------------------------------------
> > >>> [ 4] local 192.168.100.44 port 5001 connected with
> > >>> 192.168.100.1
> > >>> port 58996
> > >>> [ ID] Interval Transfer Bandwidth
> > >>> [ 4] 0.0-60.0 sec 293 GBytes 41.9 Gbits/sec
> > >>> [ 5] local 192.168.100.44 port 5001 connected with
> > >>> 192.168.100.1
> > >>> port 58997
> > >>> [ 5] 0.0-60.0 sec 297 GBytes 42.5 Gbits/sec
> > >>> [ 4] local 192.168.100.44 port 5001 connected with
> > >>> 192.168.100.1
> > >>> port 58998
> > >>> [ 4] 0.0-60.0 sec 291 GBytes 41.6 Gbits/sec
> > >>> [ 5] local 192.168.100.44 port 5001 connected with
> > >>> 192.168.100.1
> > >>> port 58999
> > >>> [ 5] 0.0-60.0 sec 297 GBytes 42.6 Gbits/sec
> > >>> [ 4] local 192.168.100.44 port 5001 connected with
> > >>> 192.168.100.1
> > >>> port 59000
> > >>> [ 4] 0.0-60.0 sec 297 GBytes 42.5 Gbits/sec
> > >>>
> > >>> While pinging out from the server to the client, I do not
> > >>> get any
> > >>> errors.
> > >>>
> > >>>
> > >>> root at umaro:~ # uname -a FreeBSD umaro 10.0-RC5 FreeBSD
> > >>> 10.0-RC5 #0
> > >>> r260430: Wed Jan 8 05:10:04 UTC 2014
> > >>> root at snap.freebsd.org:/usr/obj/usr/src/sys/GENERIC
> > >>> amd64
> > >>> root at umaro:~ # iperf -s
> > >>> ------------------------------------------------------------
> > >>> Server listening on TCP port 5001
> > >>> TCP window size: 64.0 KByte (default)
> > >>> ------------------------------------------------------------
> > >>> [ 4] local 192.168.100.5 port 5001 connected with
> > >>> 192.168.100.1
> > >>> port
> > >>> 50264
> > >>> [ ID] Interval Transfer Bandwidth
> > >>> [ 4] 0.0-60.0 sec 16.7 GBytes 2.39 Gbits/sec
> > >>> [ 5] local 192.168.100.5 port 5001 connected with
> > >>> 192.168.100.1
> > >>> port
> > >>> 50265
> > >>> [ 5] 0.0-60.0 sec 18.3 GBytes 2.62 Gbits/sec
> > >>> [ 4] local 192.168.100.5 port 5001 connected with
> > >>> 192.168.100.1
> > >>> port
> > >>> 50266
> > >>> [ 4] 0.0-60.0 sec 16.8 GBytes 2.40 Gbits/sec
> > >>> [ 5] local 192.168.100.5 port 5001 connected with
> > >>> 192.168.100.1
> > >>> port
> > >>> 50267
> > >>> [ 5] 0.0-60.0 sec 16.8 GBytes 2.40 Gbits/sec
> > >>> [ 4] local 192.168.100.5 port 5001 connected with
> > >>> 192.168.100.1
> > >>> port
> > >>> 50268
> > >>> [ 4] 0.0-60.0 sec 16.8 GBytes 2.41 Gbits/sec
> > >>>
> > >>> *** While pinging out from the server to client, frequent
> > >>> "ping:
> > >>> sendto: No space left on device" errors ***
> > >>>
> > >>>
> > >>> After a while, I can also reliably re-produce more
> > >>> egregious "ping:
> > >>> sendto: No buffer space available" errors after doing a large
> > >>> sequential
> > >>> write over NFS:
> > >>>
> > >>> mount -t nfs -o rsize=65536,wsize=65536 192.168.100.5:
> > >>> /storage/shared
> > >>> /mnt/nfs
> > >>> dd if=/dev/zero of=/mnt/nfs/testfile bs=1M count=30000
> > >>>
> > >>> I am going to file a freebsd bug report as well.
> > >>>
> Are you able to test the driver from an up to date head?
> Bryan Venteicher just committed some changes to the driver
> that increase the size of the segments list (# of mbufs in chain)
> and also switches it from using m_collapse() to m_defrag().
> I have no idea if these might affect what you are seeing?
>
>
Yes, please try HEAD if you can, but I think you can take the driver from
HEAD on 10 without too much work. I'll MFC my recent changes to 10 in a
week or so.
As for the "No buffer space available" error, can you try the attached
patch [1]? It is from HEAD, but should apply to 10 as well. I have not been
able to recreate this but I suspect it might fix it.
[1] - http://people.freebsd.org/~bryanv/patches/vtnet_watchdog.patch
rick
>
> > >>> Thanks,
> > >>> Eric
> > >>> _______________________________________________
> > >>> freebsd-stable at freebsd.org mailing list
> > >>> http://lists.freebsd.org/mailman/listinfo/freebsd-stable
> > >>> To unsubscribe, send any mail to
> > >>> "freebsd-stable-unsubscribe at freebsd.org"
> > >>>
> > > _______________________________________________
> > > freebsd-stable at freebsd.org mailing list
> > > http://lists.freebsd.org/mailman/listinfo/freebsd-stable
> > > To unsubscribe, send any mail to
> > > "freebsd-stable-unsubscribe at freebsd.org"
> > >
> >
> > _______________________________________________
> > freebsd-stable at freebsd.org mailing list
> > http://lists.freebsd.org/mailman/listinfo/freebsd-stable
> > To unsubscribe, send any mail to
> > "freebsd-stable-unsubscribe at freebsd.org"
> >
>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: vtnet_watchdog.patch
Type: text/x-patch
Size: 1921 bytes
Desc: not available
URL: <http://lists.freebsd.org/pipermail/freebsd-stable/attachments/20140204/92523b0b/attachment.bin>
More information about the freebsd-stable
mailing list