Major performance/stability regression in virtio network drivers between 9.2-RELEASE and 10.0-RC5

Rick Macklem rmacklem at uoguelph.ca
Mon Feb 3 23:34:46 UTC 2014


Alfred Perlstein wrote:
> What does vmstat say about memory zones?  I think vmstat -m/vmstat -z
> and also netstat -m?
> 
> Are you set with the same mbufs on both 9 and 10?
> 
> -Alfred
> 
> 
> On 1/18/14 1:32 PM, Eric Dombroski wrote:
> > Adrian:
> >
> > Yes, no change.
> >
> > -Eric
> >
> >
> >
> >
> >
> > On Sat, Jan 18, 2014 at 4:06 PM, Adrian Chadd
> > <adrian.chadd at gmail.com>wrote:
> >
> >> Hi,
> >>
> >> Have you tried disabling tso?
> >>
> >> Adrian
> >> On Jan 18, 2014 1:52 PM, "Eric Dombroski" <eric at edombroski.com>
> >> wrote:
> >>
> >>> Hello:
> >>>
> >>> I believe there is a major performance regression between FreeBSD
> >>> 9.2-RELEASE and 10.0-RC5 involving the virtio network drivers
> >>> (vtnet) and
> >>> handling incoming traffic.  Below are the results of some iperf
> >>> tests and
> >>> large dd operations over NFS.  Write throughput goes from ~40Gbps
> >>> to
> >>> ~2.4Gbps from 9.2 to 10.0RC5, and over time the connection
> >>> becomes
> >>> unstable
> >>> ("no buffer space available"), requiring the interface to be
> >>> taken
> >>> down/up.
> >>>
> >>>
> >>> These results are on fresh installs of 9.2 and 10.0RC5, no sysctl
> >>> tweaks
> >>> on
> >>> either system.
> >>>
> >>> I can't reproduce this using an Intel 1Gbps ethernet through PCIe
> >>> passthrough, although I suspect the problem manifests itself over
> >>> 1Gbps
> >>> speeds anyway.
> >>>
> >>> Tests:
> >>>
> >>> Client (host):
> >>>    root at gogo:~# uname -a
> >>>    Linux gogo 3.2.0-4-amd64 #1 SMP Debian 3.2.51-1 x86_64
> >>>    GNU/Linux
> >>>    root at gogo:~# kvm -version
> >>>    QEMU emulator version 1.1.2 (qemu-kvm-1.1.2+dfsg-6, Debian),
> >>>    Copyright
> >>> (c) 2003-2008 Fabrice Bellard
> >>>    root at gogo:~# lsmod | grep vhost
> >>>    vhost_net              27436  3
> >>>    tun                    18337  8 vhost_net
> >>>    macvtap                17633  1 vhost_net
> >>>
> >>>
> >>>    Command: iperf -c 192.168.100.x -t 60
> >>>
> >>>
> >>> Server (FreeBSD 9.2 VM):
> >>>
> >>>        root at umarotest:~ # uname -a
> >>>        FreeBSD umarotest 9.2-RELEASE-p3 FreeBSD 9.2-RELEASE-p3
> >>>        #0: Sat Jan
> >>> 11 03:25:02 UTC 2014
> >>> root at amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC
> >>>   amd64
> >>>        root at umarotest:~ # iperf -s
> >>>        ------------------------------------------------------------
> >>>        Server listening on TCP port 5001
> >>>        TCP window size: 64.0 KByte (default)
> >>>        ------------------------------------------------------------
> >>>        [  4] local 192.168.100.44 port 5001 connected with
> >>>        192.168.100.1
> >>> port 58996
> >>>        [ ID] Interval       Transfer     Bandwidth
> >>>        [  4]  0.0-60.0 sec   293 GBytes  41.9 Gbits/sec
> >>>        [  5] local 192.168.100.44 port 5001 connected with
> >>>        192.168.100.1
> >>> port 58997
> >>>        [  5]  0.0-60.0 sec   297 GBytes  42.5 Gbits/sec
> >>>        [  4] local 192.168.100.44 port 5001 connected with
> >>>        192.168.100.1
> >>> port 58998
> >>>        [  4]  0.0-60.0 sec   291 GBytes  41.6 Gbits/sec
> >>>        [  5] local 192.168.100.44 port 5001 connected with
> >>>        192.168.100.1
> >>> port 58999
> >>>        [  5]  0.0-60.0 sec   297 GBytes  42.6 Gbits/sec
> >>>        [  4] local 192.168.100.44 port 5001 connected with
> >>>        192.168.100.1
> >>> port 59000
> >>>        [  4]  0.0-60.0 sec   297 GBytes  42.5 Gbits/sec
> >>>
> >>>        While pinging out from the server to the client, I do not
> >>>        get any
> >>> errors.
> >>>
> >>>
> >>>        root at umaro:~ # uname -a FreeBSD umaro 10.0-RC5 FreeBSD
> >>>        10.0-RC5 #0
> >>> r260430: Wed Jan  8 05:10:04 UTC 2014
> >>> root at snap.freebsd.org:/usr/obj/usr/src/sys/GENERIC
> >>>   amd64
> >>>        root at umaro:~ # iperf -s
> >>>        ------------------------------------------------------------
> >>>        Server listening on TCP port 5001
> >>>        TCP window size: 64.0 KByte (default)
> >>>        ------------------------------------------------------------
> >>>        [  4] local 192.168.100.5 port 5001 connected with
> >>>        192.168.100.1
> >>> port
> >>> 50264
> >>>        [ ID] Interval       Transfer     Bandwidth
> >>>        [  4]  0.0-60.0 sec  16.7 GBytes  2.39 Gbits/sec
> >>>        [  5] local 192.168.100.5 port 5001 connected with
> >>>        192.168.100.1
> >>> port
> >>> 50265
> >>>        [  5]  0.0-60.0 sec  18.3 GBytes  2.62 Gbits/sec
> >>>        [  4] local 192.168.100.5 port 5001 connected with
> >>>        192.168.100.1
> >>> port
> >>> 50266
> >>>        [  4]  0.0-60.0 sec  16.8 GBytes  2.40 Gbits/sec
> >>>        [  5] local 192.168.100.5 port 5001 connected with
> >>>        192.168.100.1
> >>> port
> >>> 50267
> >>>        [  5]  0.0-60.0 sec  16.8 GBytes  2.40 Gbits/sec
> >>>        [  4] local 192.168.100.5 port 5001 connected with
> >>>        192.168.100.1
> >>> port
> >>> 50268
> >>>        [  4]  0.0-60.0 sec  16.8 GBytes  2.41 Gbits/sec
> >>>
> >>>        *** While pinging out from the server to client, frequent
> >>>        "ping:
> >>> sendto: No space left on device" errors ***
> >>>
> >>>
> >>>        After a while, I can also reliably re-produce more
> >>>        egregious "ping:
> >>> sendto: No buffer space available" errors after doing a large
> >>> sequential
> >>> write over NFS:
> >>>
> >>>        mount -t nfs -o rsize=65536,wsize=65536 192.168.100.5:
> >>> /storage/shared
> >>> /mnt/nfs
> >>>        dd if=/dev/zero of=/mnt/nfs/testfile bs=1M count=30000
> >>>
> >>> I am going to file a freebsd bug report as well.
> >>>
Are you able to test the driver from an up to date head?
Bryan Venteicher just committed some changes to the driver
that increase the size of the segments list (# of mbufs in chain)
and also switches it from using m_collapse() to m_defrag().
I have no idea if these might affect what you are seeing?

rick

> >>> Thanks,
> >>> Eric
> >>> _______________________________________________
> >>> freebsd-stable at freebsd.org mailing list
> >>> http://lists.freebsd.org/mailman/listinfo/freebsd-stable
> >>> To unsubscribe, send any mail to
> >>> "freebsd-stable-unsubscribe at freebsd.org"
> >>>
> > _______________________________________________
> > freebsd-stable at freebsd.org mailing list
> > http://lists.freebsd.org/mailman/listinfo/freebsd-stable
> > To unsubscribe, send any mail to
> > "freebsd-stable-unsubscribe at freebsd.org"
> >
> 
> _______________________________________________
> freebsd-stable at freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-stable
> To unsubscribe, send any mail to
> "freebsd-stable-unsubscribe at freebsd.org"
> 


More information about the freebsd-stable mailing list