NFS client READ performance on -current

[Rick Macklem]

Russell L. Carter wrote:
> Greetings!
> It's been 14 years.  OMG do I love poudriere and zfs.
> 
> But apropos of this post from last January:
> 
> http://lists.freebsd.org/pipermail/freebsd-net/2014-January/037547.html
> 
> I am going to capitalize READ to emphasize that all I am looking at
> here is reading on the client.  In some of posts I have read people
> have been confusing read vs. write performance and losing the thread.
> 
> I have fresh current server and client, each with otherwise quiescent
> em NICs.  I also have debian kernel 3.14-1 server and client.  The
> freebsd hardware is large enough, 16GB RAM + vishera 8 cores.  (The
> debian systems are much smaller/slower)
> 
> Upshot is that with a -current NFSV4 (4.1) serving a -current client
> and a linux client, I see today client READ performance of:
> 
> -current -> -current:  2.7MB/s READ nice and steady
> -current -> linux:    80MB/s  READ very bursty
> 
> And for the weird (to me) result:
> 
> linux -> -current: 55.74MB/s READ
> 
> So to be very clear, the freebsd -current server seems capable of of
> supporting excellent client READ bandwidth.  But something is
> dramatically awry with the freebsd client.  (scp, rsync over ssh, all
> are near wire speed when I get rid of disk overhead).  The -current
> server has got an 11TB raidz2 behind it but I'm only READing, not
> writing, so all of the discussion about ZFS + NFS sync write problems
> seems immaterial, possibly.
> 
> Here's the mount info:
> 
> -current client's output from nfsstat -m (terp is current server, ari
>  is linux server):
> 
> terp:/raid/library on /mnt/terp/library
> nfsv4,minorversion=1,tcp,resvport,hard,cto,sec=sys,acdirmin=3,acdirmax=60,acregmin=5,acregmax=60,nametimeo=60,negnametimeo=60,rsize=65536,wsize=65536,readdirsize=65536,readahead=1,wcommitsize=2798255,timeout=120,retrans=2147483647
> 
> ari:/d1/library on /mnt/ari
> nfsv4,minorversion=0,tcp,resvport,hard,cto,sec=sys,acdirmin=3,acdirmax=60,acregmin=5,acregmax=60,nametimeo=60,negnametimeo=60,rsize=65536,wsize=65536,readdirsize=65536,readahead=1,wcommitsize=2798255,timeout=120,retrans=2147483647
> 
> Linux client's excerpt from mount output:
> 
> terp:/raid/library on /mnt/terp/library type nfs4
> (rw,relatime,vers=4.0,rsize=65536,wsize=65536,namlen=255,hard,proto=tcp,port=0,timeo=600,retrans=2,sec=sys,clientaddr=10.0.10.3,local_lock=none,addr=10.0.10.4)
> 
> BTW, linux -> linux is about 80MB/s
> 
> I tried nfsv3 -current -> -current and still got 2.7MB/s READ.
> 
> I have spent about 5 hours strolling through the archives of the last
> several years of discussion about this issue in various forums, but I
> didn't see a reference once since Jaunary's thread on freebsd-net,
> and
> I didn't find a bug entry, so I thought I'd keep it alive, so to
> speak.
> 
> I've tried all of the various sysctl tweaks that have been suggested
> over time, that would make sense for reads, to no effect.
> 
Please try disabling TSO on the FreeBSD systems and also
try an rsize=32768 to see if either of those have any effect.

There was also a recent case where disabling msix interrupt
handling for the network interface fixed a slow NFS I/O rate
issue. (I don't know what the loader conf variable was, but
look for something with msix in it for the net device driver.
It might be hw.em.enable_msix=0 or something like that?)

rick

> Russell
> 
> 
> _______________________________________________
> freebsd-net at freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-net
> To unsubscribe, send any mail to
> "freebsd-net-unsubscribe at freebsd.org"
>