Bad performance of 7.0 nfs client with Solaris nfs server

[Alfred Perlstein]

* Kris Kennaway <kris at FreeBSD.org> [080220 13:42] wrote:
> Chuck Swiger wrote:
> >On Feb 20, 2008, at 1:01 PM, Alfred Perlstein wrote:
> >>>Take a look at the level of packet fragmentation you are encountering;
> >>>yes, this is expected and things will work but there is extra latency
> >>>added when the IP stack has to reassemble packets before the data can
> >>>be delivered.  Try setting the NFS rsize/wsize to 1024 or perhaps 1400
> >>>and see whether that improves performance.
> >>>
> >>>Or, if your switch and NICs support it, see whether you can get Gb
> >>>Ethernet jumbo frames working so that you don't have to fragment for
> >>>2K or 4K data packets....
> >>
> >>TCP mounts do not have this problem.  You can safely use
> >>32k or higher sizes with TCP without fragmentation.
> >
> >Oh, sure.  But there is a bit more overhead with TCP transport than 
> >UDP-- for local (switched) networks, UDP generally seems to be a 
> >win...TCP seems to be a better choice over a VPN or some similar kind of 
> >WAN.
> 
> Actually this is no longer true.  At modern LAN speeds (e.g. gige) you 
> can transmit packets fast enough that two things happen:
> 
> 1) UDP socket buffer overruns are common, leading to packet loss.
> 
> 2) the 16-bit sequence numbers wrap *much* faster than the IP fragment 
> lifetime (30 seconds).
> 
> These combine to cause data corruption when fragmented packets are 
> dropped and then reassembled with a later transmission of the same 
> sequence number.
> 
> TCP mounts should be used whenever possible thesedays (I flipped the 
> default mode in 8.0 the other day).

Additionally with smaller read/write sizes you must generate more
RPCs, for instance using a tcp read size of 32k will allow the
client to generate a single NFS_READ request for that 32k, however
a UDP mount at 1.2k will need to generate appoximately 26 requests
and the server will have to service that many requests as well.



-- 
- Alfred Perlstein