scp more perfectly fills the pipe than NFS/TCP

[Matthew Dillon]

    Play with the read-ahead mount options for NFS, but it might require
    more work with that kind of latency.  You need to be able to have
    a lot of RPC's in-flight to maintain the pipeline with higher latencies.
    At least 16 and possibly more.

    It might be easier to investigate why the latency is so high and fix
    that first.  10ms is way too high for a LAN.

    I remember there was some work in the FreeBSD tree to clean up the
    client-side NFS rpc mechanics but if they are still threaded (kernel
    thread or user thread, doesn't matter) with one synchronous RPC per
    thread then a large amount of read-ahead will cause the requests to be
    issued out of order over the wire (for both TCP and UDP NFS mounts),
    which really messes up the server-side heuristics.  Plus the
    client-side threads wind up competing with each other for the
    socket lock.  So there is a limit to how large a read-ahead you
    can specify and still get good results.

    If they are using a single kernel thread for socket reading and a
    single kernel thread for socket writing (i.e. a 100% async RPC model,
    which is what DFly uses), then you can boost the read-ahead to 50+.
    At that point the socket buffer becomes the limiting factor in the
    pipeline.

    Make sure the NFS mount is TCP (It defaults to TCP in FreeBSD 8+).  UDP
    mounts will not perform well with any read-ahead greater then 3 or 4
    RPCs because occassional seek latencies on the server will cause
    random UDP RPCs to timeout and retry, which completely destroys
    performance.  UDP mounts have no understanding of the RPC queue backlog
    on the server and treat each RPC independently for timeout/retry
    purposes.  So one minor stall can blow up every single pending RPC
    backed up behind the one that stalled.

						-Matt