A defensive NFS server (sbwait, flow control)

[Don Lewis]

On 12 Mar, Tim Borgeaud wrote:
> Hi again FreeBSD folks,
> 
> A short while ago I sent a couple of emails regarding the idea of 'fair
> share' NFS scheduling. Amongst others, Garrett Wollman replied, and also
> sent a related email "Implementing backpressure in the NFS server". The
> common theme: to provide a level of service in situations where requests
> from some clients might tie up a lot of resources.
> 
> There are various issues to consider. We might say that we're looking  at
> 'defensive' functionality, and we've made some experimental progress to:
> 
>   round robin requests from different users
> 
>   provide some limits (flow control) on the number of requests read
>   from a single transport
> 
> Testing has highlighted the issue that Garrett mentioned. A client can make
> a set of concurrent requests and then tie up nfsd threads as they attempt
> to send replies.
> 
> To be more specific, we seem to be hitting the situation in which an nfsd
> thread sits in the "sbwait" state, waiting for a reply to be drained (by a
> suspended client). Other threads subsequently pick up requests for the same
> transport and then queue up waiting for a lock on the same socket.
> 
> I'm not sure of the exact situation in which the sbwait arises. It's easily
> repeatable only where there are multiple concurrent requests from the same
> transport.
> 
> Our testing is fairly synthetic, but it looks like this is a situation that
> has been noticed before. Having a pool of spare threads doesn't seem like a
> very robust solution. In fact, if more than the server minimum get tied up,
> then, if load and threads fall, we end up with _all_ remaining threads
> blocked (no more nfs service).
> 
> How to address this particular issue is not obvious to me. I assume there
> are options including:
> 
>  - prevent thread blocking when sending replies
> 
>  - timeouts for sending replies (NFS or RPC level?)
> 
>  - serialize the sending of nfs/rpc replies to avoid multiple
>    nfsd threads waiting on the same transport.
> 
> Does anyone have any thoughts about this? Either this particular issue or
> more general direction for a 'defensive' server?

If a thread blocks when sending a reply, stop reading any new requests
from that transport.  That should limit the number of threads that get
tied up.