amd64/185864: Major performance/stability regression in virtio network drivers between 9.2-RELEASE and 10.0-RC5

Sat Jan 18 19:00:01 UTC 2014

>Number:         185864
>Category:       amd64
>Synopsis:       Major performance/stability regression in virtio network drivers between 9.2-RELEASE and 10.0-RC5
>Confidential:   no
>Severity:       non-critical
>Priority:       low
>Responsible:    freebsd-amd64
>State:          open
>Quarter:        
>Keywords:       
>Date-Required:
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Sat Jan 18 19:00:00 UTC 2014
>Closed-Date:
>Last-Modified:
>Originator:     Eric Dombroski
>Release:        10.0RC5
>Organization:
>Environment:
FreeBSD umaro 10.0-RC5 FreeBSD 10.0-RC5 #0 r260430: Wed Jan  8 05:10:04 UTC 2014     root at snap.freebsd.org:/usr/obj/usr/src/sys/GENERIC  amd64
>Description:
 believe there is a major performance regression between FreeBSD 9.2-RELEASE and 10.0-RC5 involving the virtio network drivers (vtnet) and handling incoming traffic.  Below are the results of some iperf tests and large dd operations over NFS.  Write throughput goes from ~40Gbps to ~2.4Gbps from 9.2 to 10.0RC5, and over time the connection becomes unstable ("no buffer space available"), requiring the interface to be taken down/up.  

These results are on fresh installs of 9.2 and 10.0RC5, no sysctl tweaks on either system.

I can't reproduce this using an Intel 1Gbps ethernet through PCIe passthrough, although I suspect the problem manifests itself over 1Gbps speeds anyway.

Tests: 

Client (host):
  root at gogo:~# uname -a
  Linux gogo 3.2.0-4-amd64 #1 SMP Debian 3.2.51-1 x86_64 GNU/Linux
  root at gogo:~# kvm -version
  QEMU emulator version 1.1.2 (qemu-kvm-1.1.2+dfsg-6, Debian), Copyright (c) 2003-2008 Fabrice Bellard
  root at gogo:~# lsmod | grep vhost
  vhost_net              27436  3
  tun                    18337  8 vhost_net
  macvtap                17633  1 vhost_net

  Command: iperf -c 192.168.100.x -t 60

Server (FreeBSD 9.2 VM):

      root at umarotest:~ # uname -a
      FreeBSD umarotest 9.2-RELEASE-p3 FreeBSD 9.2-RELEASE-p3 #0: Sat Jan 11 03:25:02 UTC 2014     root at amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC  amd64
      root at umarotest:~ # iperf -s
      ------------------------------------------------------------
      Server listening on TCP port 5001
      TCP window size: 64.0 KByte (default)
      ------------------------------------------------------------
      [  4] local 192.168.100.44 port 5001 connected with 192.168.100.1 port 58996
      [ ID] Interval       Transfer     Bandwidth
      [  4]  0.0-60.0 sec   293 GBytes  41.9 Gbits/sec
      [  5] local 192.168.100.44 port 5001 connected with 192.168.100.1 port 58997
      [  5]  0.0-60.0 sec   297 GBytes  42.5 Gbits/sec
      [  4] local 192.168.100.44 port 5001 connected with 192.168.100.1 port 58998
      [  4]  0.0-60.0 sec   291 GBytes  41.6 Gbits/sec
      [  5] local 192.168.100.44 port 5001 connected with 192.168.100.1 port 58999
      [  5]  0.0-60.0 sec   297 GBytes  42.6 Gbits/sec
      [  4] local 192.168.100.44 port 5001 connected with 192.168.100.1 port 59000
      [  4]  0.0-60.0 sec   297 GBytes  42.5 Gbits/sec

      While pinging out from the server to the client, I do not get any errors.

      root at umaro:~ # uname -a FreeBSD umaro 10.0-RC5 FreeBSD 10.0-RC5 #0 r260430: Wed Jan  8 05:10:04 UTC 2014     root at snap.freebsd.org:/usr/obj/usr/src/sys/GENERIC  amd64
      root at umaro:~ # iperf -s
      ------------------------------------------------------------
      Server listening on TCP port 5001
      TCP window size: 64.0 KByte (default)
      ------------------------------------------------------------
      [  4] local 192.168.100.5 port 5001 connected with 192.168.100.1 port 50264
      [ ID] Interval       Transfer     Bandwidth
      [  4]  0.0-60.0 sec  16.7 GBytes  2.39 Gbits/sec
      [  5] local 192.168.100.5 port 5001 connected with 192.168.100.1 port 50265
      [  5]  0.0-60.0 sec  18.3 GBytes  2.62 Gbits/sec
      [  4] local 192.168.100.5 port 5001 connected with 192.168.100.1 port 50266
      [  4]  0.0-60.0 sec  16.8 GBytes  2.40 Gbits/sec
      [  5] local 192.168.100.5 port 5001 connected with 192.168.100.1 port 50267
      [  5]  0.0-60.0 sec  16.8 GBytes  2.40 Gbits/sec
      [  4] local 192.168.100.5 port 5001 connected with 192.168.100.1 port 50268
      [  4]  0.0-60.0 sec  16.8 GBytes  2.41 Gbits/sec

      *** While pinging out from the server to client, frequent "ping: sendto: No space left on device" errors ***

      After a while, I can also reliably re-produce more egregious "ping: sendto: No buffer space available" errors after doing a large sequential write over NFS:

      mount -t nfs -o rsize=65536,wsize=65536 192.168.100.5:/storage/shared /mnt/nfs
      dd if=/dev/zero of=/mnt/nfs/testfile bs=1M count=30000

>How-To-Repeat:
Perform iperf test between a KVM host and the FreeBSD 9.2 VM as a server using virtio network drivers while pinging out from FreeBSD system; compare to 10.0RC5...results go from ~40Gbps to ~2.4Gbps and the 10.0RC5 system shows many "no space left on device" errors during pings.

Perform large sequential write (dd, mysqldump, etc.) operation over NFS, eventually interface goes down with "no buffer space available" errors during ping.  Only remediation at that point is to take down the interface and take it up again.
>Fix:

>Release-Note:
>Audit-Trail:
>Unformatted: