Major performance/stability regression in virtio network drivers between 9.2-RELEASE and 10.0-RC5

[Eric Dombroski]

Hello:

I believe there is a major performance regression between FreeBSD
9.2-RELEASE and 10.0-RC5 involving the virtio network drivers (vtnet) and
handling incoming traffic.  Below are the results of some iperf tests and
large dd operations over NFS.  Write throughput goes from ~40Gbps to
~2.4Gbps from 9.2 to 10.0RC5, and over time the connection becomes unstable
("no buffer space available"), requiring the interface to be taken down/up.


These results are on fresh installs of 9.2 and 10.0RC5, no sysctl tweaks on
either system.

I can't reproduce this using an Intel 1Gbps ethernet through PCIe
passthrough, although I suspect the problem manifests itself over 1Gbps
speeds anyway.

Tests:

Client (host):
  root at gogo:~# uname -a
  Linux gogo 3.2.0-4-amd64 #1 SMP Debian 3.2.51-1 x86_64 GNU/Linux
  root at gogo:~# kvm -version
  QEMU emulator version 1.1.2 (qemu-kvm-1.1.2+dfsg-6, Debian), Copyright
(c) 2003-2008 Fabrice Bellard
  root at gogo:~# lsmod | grep vhost
  vhost_net              27436  3
  tun                    18337  8 vhost_net
  macvtap                17633  1 vhost_net


  Command: iperf -c 192.168.100.x -t 60


Server (FreeBSD 9.2 VM):

      root at umarotest:~ # uname -a
      FreeBSD umarotest 9.2-RELEASE-p3 FreeBSD 9.2-RELEASE-p3 #0: Sat Jan
11 03:25:02 UTC 2014
root at amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC
 amd64
      root at umarotest:~ # iperf -s
      ------------------------------------------------------------
      Server listening on TCP port 5001
      TCP window size: 64.0 KByte (default)
      ------------------------------------------------------------
      [  4] local 192.168.100.44 port 5001 connected with 192.168.100.1
port 58996
      [ ID] Interval       Transfer     Bandwidth
      [  4]  0.0-60.0 sec   293 GBytes  41.9 Gbits/sec
      [  5] local 192.168.100.44 port 5001 connected with 192.168.100.1
port 58997
      [  5]  0.0-60.0 sec   297 GBytes  42.5 Gbits/sec
      [  4] local 192.168.100.44 port 5001 connected with 192.168.100.1
port 58998
      [  4]  0.0-60.0 sec   291 GBytes  41.6 Gbits/sec
      [  5] local 192.168.100.44 port 5001 connected with 192.168.100.1
port 58999
      [  5]  0.0-60.0 sec   297 GBytes  42.6 Gbits/sec
      [  4] local 192.168.100.44 port 5001 connected with 192.168.100.1
port 59000
      [  4]  0.0-60.0 sec   297 GBytes  42.5 Gbits/sec

      While pinging out from the server to the client, I do not get any
errors.


      root at umaro:~ # uname -a FreeBSD umaro 10.0-RC5 FreeBSD 10.0-RC5 #0
r260430: Wed Jan  8 05:10:04 UTC 2014
root at snap.freebsd.org:/usr/obj/usr/src/sys/GENERIC
 amd64
      root at umaro:~ # iperf -s
      ------------------------------------------------------------
      Server listening on TCP port 5001
      TCP window size: 64.0 KByte (default)
      ------------------------------------------------------------
      [  4] local 192.168.100.5 port 5001 connected with 192.168.100.1 port
50264
      [ ID] Interval       Transfer     Bandwidth
      [  4]  0.0-60.0 sec  16.7 GBytes  2.39 Gbits/sec
      [  5] local 192.168.100.5 port 5001 connected with 192.168.100.1 port
50265
      [  5]  0.0-60.0 sec  18.3 GBytes  2.62 Gbits/sec
      [  4] local 192.168.100.5 port 5001 connected with 192.168.100.1 port
50266
      [  4]  0.0-60.0 sec  16.8 GBytes  2.40 Gbits/sec
      [  5] local 192.168.100.5 port 5001 connected with 192.168.100.1 port
50267
      [  5]  0.0-60.0 sec  16.8 GBytes  2.40 Gbits/sec
      [  4] local 192.168.100.5 port 5001 connected with 192.168.100.1 port
50268
      [  4]  0.0-60.0 sec  16.8 GBytes  2.41 Gbits/sec

      *** While pinging out from the server to client, frequent "ping:
sendto: No space left on device" errors ***


      After a while, I can also reliably re-produce more egregious "ping:
sendto: No buffer space available" errors after doing a large sequential
write over NFS:

      mount -t nfs -o rsize=65536,wsize=65536 192.168.100.5:/storage/shared
/mnt/nfs
      dd if=/dev/zero of=/mnt/nfs/testfile bs=1M count=30000

I am going to file a freebsd bug report as well.

Thanks,
Eric