cvs commit: src/sys/nfsclient nfs.h nfs_socket.c nfs_subs.c nfsmount.h

[Kris Kennaway]

Mohan Srinivasan wrote:
> mohans      2007-10-12 19:12:21 UTC
> 
>   FreeBSD src repository
> 
>   Modified files:
>     sys/nfsclient        nfs.h nfs_socket.c nfs_subs.c nfsmount.h 
>   Log:
>   NFS MP scaling changes.
>   - Eliminate the hideous nfs_sndlock that serialized NFS/TCP request senders
>     thru the sndlock.
>   - Institute a new nfs_connectlock that serializes NFS/TCP reconnects. Add
>     logic to wait for pending request senders to finish sending before
>     reconnecting. Dial down the sb_timeo for NFS/TCP sockets to 1 sec.
>   - Break out the nfs xid manipulation under a new nfs xid lock, rather than
>     over loading the nfs request lock for this purpose.
>   - Fix some of the locking in nfs_request.
>   Many thanks to Kris Kennaway for his help with this and for initiating the
>   MP scaling analysis and work. Kris also tested this patch thorougly.
>   Approved by: re@ (Ken Smith)

For the benefit of others: this change improved throughput by about 10% 
at high I/O loads with a dual core client, and by a factor of 10 on an 8 
core client (this was mostly the home-brew nfs_sndlock, which mohan 
correctly describes :-).

Mohan's previous commit that increases the nfs server socket buffer size 
is also very important for NFS performance.  Without it I was only 
getting 1-2MB/sec throughput over 10Gb ethernet with UDP mounts, because 
the minuscule 32kb socket buffer was constantly filling up and forcing 
retransmits.  With the new default of 256KB I still get full buffers 
with 10ge, so you may need to increase this further to eliminate this 
problem.  It might be OK for gige speeds, although I was still seeing 
some buffer full events, so maybe we should consider increasing the 
default sockbuf size to 512KB or so if this is widespread.

As a side comment there is a bug in either the nfs client or server that 
corrupts I/O when there is packet loss with UDP mounts (the default). 
TCP mounts handle this at the TCP layer.

Kris