NFS on 10G interfaces still painfully slow

[Valeri Galtsev]

On Wed, August 3, 2016 12:59 am, Gerrit Kühn wrote:
> On Tue, 2 Aug 2016 08:45:07 -0600 Alan Somers <asomers at freebsd.org> wrote
> about Re: NFS on 10G interfaces still painfully slow:
>
>> > Is there anyone around here who can confirm that nfs can go faster
>> > over 10G links?
>> > Any hints for further tuning/debugging are greatly appreciated.
>
> AS> I can get 1GB/s over NFS on a 10G link, so it's not always slow.
> AS> There's probably something about your setup that's slowing it down.
> AS> What is your NFS client?
>
> This time, FreeBSD 10.3 on both client and server (to make debugging
> easier).
>
> AS>  What benchmark are you using to measure that speed?
>
> Right now only very simple things like using dd from /dev/zero or copying
> large files.

When I had trouble (too slow data throughput, never mind on what medium)
using dd - it was: not specifying bs=[big number] in dd command. It turned
out without that dd sends stuff down the pipe in very small chunks and
wouldn't send next chunk till acknowledgement on previous is received.
Imagine hard drive wrighting 4 kb at a time send 1 byte at a time, or RAID
expected to write 8 x 64 kB striped in parallel sent 1 byte at a time. In
your case this [big number] ideally should be equal or slightly smaller
than size of jumbo packet of you net connection.

Just speculating (remembered really trivial thing _I_ hit myself ;-)

Thanks.
Valeri

> In my experience it is useless to go for more sophisticated
> benchmarks, if these simple things already don't work as expected.
>
> AS>  Did you remember to start lockd and statd?
>
> Yes.
>
> AS> If you post your /etc/exports and the client's /etc/fstab, that might
> AS> reveal something.
>
> exports on the server side:
>
> V4: /mt-rear    -sec=sys 192.168.1.11
> /mt-right       192.168.1.11 -maproot=root
> /mt-rear        -maproot=root 192.168.1.11
> /mt-left        192.168.1.11 -maproot=root
>
>
> fstab on the client does not tell you anything, I still use commandline
> mounts during testing. This is what nfsstat -m will tell (V4 is not
> mounted right now):
>
> tom:/mt-rear on /net/mt-rear
> nfsv3,tcp,resvport,hard,cto,lockd,rdirplus,sec=sys,acdirmin=3,acdirmax=60,acregmin=5,acregmax=60,nametimeo=60,negnametimeo=60,rsize=65536,wsize=65536,readdirsize=65536,readahead=4,wcommitsize=50000000,timeout=120,retrans=2
> tom:/mt-right on /net/mt-right
> nfsv3,tcp,resvport,hard,cto,lockd,rdirplus,sec=sys,acdirmin=3,acdirmax=60,acregmin=5,acregmax=60,nametimeo=60,negnametimeo=60,rsize=65536,wsize=65536,readdirsize=65536,readahead=1,wcommitsize=16777216,timeout=120,retrans=2
> tom:/mt-left on /net/mt-left
> nfsv3,tcp,resvport,hard,cto,lockd,rdirplus,sec=sys,acdirmin=3,acdirmax=60,acregmin=5,acregmax=60,nametimeo=60,negnametimeo=60,rsize=65536,wsize=65536,readdirsize=65536,readahead=1,wcommitsize=16777216,timeout=120,retrans=2
>
>
> This is what, e.g., dd gives:
>
> root at crest:~ # dd if=/dev/zero of=/net/mt-rear/Z bs=1024k count=1000
> 1000+0 records in
> 1000+0 records out
> 1048576000 bytes transferred in 1.403620 secs (747051194 bytes/sec)
>
> root at crest:~ # dd if=/dev/zero of=/net/mt-right/Z bs=1024k
> count=1000 1000+0 records in
> 1000+0 records out
> 1048576000 bytes transferred in 1.380546 secs (759537249 bytes/sec)
>
>
> And yes (before that question pops up :-), I'm using zfs on the server
> side, but I disabled syncing for testing purposes.
>
>
> cu
>   Gerrit
> _______________________________________________
> freebsd-net at freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-net
> To unsubscribe, send any mail to "freebsd-net-unsubscribe at freebsd.org"
>


++++++++++++++++++++++++++++++++++++++++
Valeri Galtsev
Sr System Administrator
Department of Astronomy and Astrophysics
Kavli Institute for Cosmological Physics
University of Chicago
Phone: 773-702-4247
++++++++++++++++++++++++++++++++++++++++