Major performance/stability regression in virtio network drivers between 9.2-RELEASE and 10.0-RC5

Sun Jan 19 14:30:50 UTC 2014

What does vmstat say about memory zones?  I think vmstat -m/vmstat -z 
and also netstat -m?

Are you set with the same mbufs on both 9 and 10?

-Alfred

On 1/18/14 1:32 PM, Eric Dombroski wrote:
> Adrian:
>
> Yes, no change.
>
> -Eric
>
>
>
>
>
> On Sat, Jan 18, 2014 at 4:06 PM, Adrian Chadd <adrian.chadd at gmail.com>wrote:
>
>> Hi,
>>
>> Have you tried disabling tso?
>>
>> Adrian
>> On Jan 18, 2014 1:52 PM, "Eric Dombroski" <eric at edombroski.com> wrote:
>>
>>> Hello:
>>>
>>> I believe there is a major performance regression between FreeBSD
>>> 9.2-RELEASE and 10.0-RC5 involving the virtio network drivers (vtnet) and
>>> handling incoming traffic.  Below are the results of some iperf tests and
>>> large dd operations over NFS.  Write throughput goes from ~40Gbps to
>>> ~2.4Gbps from 9.2 to 10.0RC5, and over time the connection becomes
>>> unstable
>>> ("no buffer space available"), requiring the interface to be taken
>>> down/up.
>>>
>>>
>>> These results are on fresh installs of 9.2 and 10.0RC5, no sysctl tweaks
>>> on
>>> either system.
>>>
>>> I can't reproduce this using an Intel 1Gbps ethernet through PCIe
>>> passthrough, although I suspect the problem manifests itself over 1Gbps
>>> speeds anyway.
>>>
>>> Tests:
>>>
>>> Client (host):
>>>    root at gogo:~# uname -a
>>>    Linux gogo 3.2.0-4-amd64 #1 SMP Debian 3.2.51-1 x86_64 GNU/Linux
>>>    root at gogo:~# kvm -version
>>>    QEMU emulator version 1.1.2 (qemu-kvm-1.1.2+dfsg-6, Debian), Copyright
>>> (c) 2003-2008 Fabrice Bellard
>>>    root at gogo:~# lsmod | grep vhost
>>>    vhost_net              27436  3
>>>    tun                    18337  8 vhost_net
>>>    macvtap                17633  1 vhost_net
>>>
>>>
>>>    Command: iperf -c 192.168.100.x -t 60
>>>
>>>
>>> Server (FreeBSD 9.2 VM):
>>>
>>>        root at umarotest:~ # uname -a
>>>        FreeBSD umarotest 9.2-RELEASE-p3 FreeBSD 9.2-RELEASE-p3 #0: Sat Jan
>>> 11 03:25:02 UTC 2014
>>> root at amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC
>>>   amd64
>>>        root at umarotest:~ # iperf -s
>>>        ------------------------------------------------------------
>>>        Server listening on TCP port 5001
>>>        TCP window size: 64.0 KByte (default)
>>>        ------------------------------------------------------------
>>>        [  4] local 192.168.100.44 port 5001 connected with 192.168.100.1
>>> port 58996
>>>        [ ID] Interval       Transfer     Bandwidth
>>>        [  4]  0.0-60.0 sec   293 GBytes  41.9 Gbits/sec
>>>        [  5] local 192.168.100.44 port 5001 connected with 192.168.100.1
>>> port 58997
>>>        [  5]  0.0-60.0 sec   297 GBytes  42.5 Gbits/sec
>>>        [  4] local 192.168.100.44 port 5001 connected with 192.168.100.1
>>> port 58998
>>>        [  4]  0.0-60.0 sec   291 GBytes  41.6 Gbits/sec
>>>        [  5] local 192.168.100.44 port 5001 connected with 192.168.100.1
>>> port 58999
>>>        [  5]  0.0-60.0 sec   297 GBytes  42.6 Gbits/sec
>>>        [  4] local 192.168.100.44 port 5001 connected with 192.168.100.1
>>> port 59000
>>>        [  4]  0.0-60.0 sec   297 GBytes  42.5 Gbits/sec
>>>
>>>        While pinging out from the server to the client, I do not get any
>>> errors.
>>>
>>>
>>>        root at umaro:~ # uname -a FreeBSD umaro 10.0-RC5 FreeBSD 10.0-RC5 #0
>>> r260430: Wed Jan  8 05:10:04 UTC 2014
>>> root at snap.freebsd.org:/usr/obj/usr/src/sys/GENERIC
>>>   amd64
>>>        root at umaro:~ # iperf -s
>>>        ------------------------------------------------------------
>>>        Server listening on TCP port 5001
>>>        TCP window size: 64.0 KByte (default)
>>>        ------------------------------------------------------------
>>>        [  4] local 192.168.100.5 port 5001 connected with 192.168.100.1
>>> port
>>> 50264
>>>        [ ID] Interval       Transfer     Bandwidth
>>>        [  4]  0.0-60.0 sec  16.7 GBytes  2.39 Gbits/sec
>>>        [  5] local 192.168.100.5 port 5001 connected with 192.168.100.1
>>> port
>>> 50265
>>>        [  5]  0.0-60.0 sec  18.3 GBytes  2.62 Gbits/sec
>>>        [  4] local 192.168.100.5 port 5001 connected with 192.168.100.1
>>> port
>>> 50266
>>>        [  4]  0.0-60.0 sec  16.8 GBytes  2.40 Gbits/sec
>>>        [  5] local 192.168.100.5 port 5001 connected with 192.168.100.1
>>> port
>>> 50267
>>>        [  5]  0.0-60.0 sec  16.8 GBytes  2.40 Gbits/sec
>>>        [  4] local 192.168.100.5 port 5001 connected with 192.168.100.1
>>> port
>>> 50268
>>>        [  4]  0.0-60.0 sec  16.8 GBytes  2.41 Gbits/sec
>>>
>>>        *** While pinging out from the server to client, frequent "ping:
>>> sendto: No space left on device" errors ***
>>>
>>>
>>>        After a while, I can also reliably re-produce more egregious "ping:
>>> sendto: No buffer space available" errors after doing a large sequential
>>> write over NFS:
>>>
>>>        mount -t nfs -o rsize=65536,wsize=65536 192.168.100.5:
>>> /storage/shared
>>> /mnt/nfs
>>>        dd if=/dev/zero of=/mnt/nfs/testfile bs=1M count=30000
>>>
>>> I am going to file a freebsd bug report as well.
>>>
>>> Thanks,
>>> Eric
>>> _______________________________________________
>>> freebsd-stable at freebsd.org mailing list
>>> http://lists.freebsd.org/mailman/listinfo/freebsd-stable
>>> To unsubscribe, send any mail to "freebsd-stable-unsubscribe at freebsd.org"
>>>
> _______________________________________________
> freebsd-stable at freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-stable
> To unsubscribe, send any mail to "freebsd-stable-unsubscribe at freebsd.org"
>