NFS on 10G interface terribly slow

[Rick Macklem]

Scott Larson wrote:
> We've got 10.0 and 10.1 servers accessing Isilon and Nexenta via NFS
> with Intel 10G gear and bursting to near wire speed with the stock
> MTU/rsize/wsize works as expected. TSO definitely needs to be enabled for
> that performance.
Btw, can you tell us what Intel chip(s) you're using?

For example, from the "ix" driver:
#define IXGBE_82598_SCATTER		100
#define IXGBE_82599_SCATTER		32

This implies that the 82598 won't have problems with 64K TSO segments, but
the 82599 will end up doing calls to m_defrag() which copies the entire
list of mbufs into 32 new mbuf clusters for each of them.
--> Even for one driver, different chips may result in different NFS perf.

Btw, it appears that the driver in head/current now sets if_hw_tsomaxsegcount,
but the driver in stable/10 does not. This means that the 82599 chip will end
up doing the m_defrag() calls for 10.x.

rick

> The fact iperf gives you the expected throughput but NFS
> does not would have me looking at tuning for the NFS platform. Other things
> to look at: Are all the servers involved negotiating the correct speed and
> duplex, with TSO? Does it need to have the network stack tuned with
> whatever it's equivalent of maxsockbuf and send/recvbuf are? Do the switch
> ports and NIC counters show any drops or errors? On the FBSD servers you
> could also run 'netstat -i -w 1' under load to see if drops are occurring
> locally, or 'systat -vmstat' for resource contention problems. But again, a
> similar setup here and no such issues have appeared.
> 
> 
> *[image: userimage]Scott Larson[image: los angeles]
> <https://www.google.com/maps/place/4216+Glencoe+Ave,+Marina+Del+Rey,+CA+90292/@33.9892151,-118.4421334,17z/data=!3m1!4b1!4m2!3m1!1s0x80c2ba88ffae914d:0x14e1d00084d4d09c>Lead
> Systems Administrator[image: wdlogo] <https://www.wiredrive.com/> [image:
> linkedin] <https://www.linkedin.com/company/wiredrive> [image: facebook]
> <https://www.twitter.com/wiredrive> [image: twitter]
> <https://www.facebook.com/wiredrive> [image: instagram]
> <https://www.instagram.com/wiredrive>T 310 823 8238 x1106
> <310%20823%208238%20x1106>  |  M 310 904 8818 <310%20904%208818>*
> 
> On Thu, Jun 25, 2015 at 5:52 AM, Gerrit Kühn <gerrit.kuehn at aei.mpg.de>
> wrote:
> 
> > Hi all,
> >
> > We have a recent FreeBSD 10.1 installation here that is supposed to act as
> > nfs (v3) client to an Oracle x4-2l server running Soalris 11.2.
> > We have Intel 10-Gigabit X540-AT2 NICs on both ends, iperf is showing
> > plenty of bandwidth (9.xGB/s) in both directions.
> > However, nfs appears to be terribly slow, especially for writing:
> >
> > root at crest:~ # dd if=/dev/zero of=/net/hellpool/Z bs=1024k count=1000
> > 1000+0 records in
> > 1000+0 records out
> > 1048576000 bytes transferred in 20.263190 secs (51747824 bytes/sec)
> >
> >
> > Reading appears to be faster, but still far away from full bandwidth:
> >
> > root at crest:~ # dd of=/dev/null if=/net/hellpool/Z bs=1024k
> > 1000+0 records in
> > 1000+0 records out
> > 1048576000 bytes transferred in 5.129869 secs (204406000 bytes/sec)
> >
> >
> > We have already tried to tune rsize/wsize parameters, but they appear to
> > have little (if any) impact on these results. Also, neither stripping down
> > rxsum, txsum, tso etc. from the interface nor increasing MTU to 9000 for
> > jumbo frames did improve anything.
> > It is quite embarrassing to achieve way less than 1GBE performance with
> > 10GBE equipment. Are there any hints what else might be causing this (and
> > how to fix it)?
> >
> >
> > cu
> >   Gerrit
> > _______________________________________________
> > freebsd-net at freebsd.org mailing list
> > http://lists.freebsd.org/mailman/listinfo/freebsd-net
> > To unsubscribe, send any mail to "freebsd-net-unsubscribe at freebsd.org"
> >
> _______________________________________________
> freebsd-net at freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-net
> To unsubscribe, send any mail to "freebsd-net-unsubscribe at freebsd.org"