Re: zfs/nfsd performance limiter

Adam Stylinski <kungfujesus06@gmail.com> wrote:
[stuff snipped]
>
> However, in general, RPC RTT will define how well NFS performs and not
> the I/O rate for a bulk file read/write.
Lets take this RPC RTT thing a step further...
- If I got the math right, at 40Gbps, 1Mbyte takes about 200usec on the wire.
Without readahead, the protocol looks like this:
Client                                     Server (time going down the screen)
        small Read request --->
        <-- 1Mbyte reply
        small Read request -->
        <-- 1Mbyte reply
The 1Mbyte replies take 200usec on the wire.

Then suppose your ping time is 400usec (I see about 350usec on my little lan).
- The wire is only transferring data about half of the time, because the small
  request message takes almost as long as the 1Mbyte reply.

As you can see, readahead (where multiple reads are done concurrently)
is critical for this case. I have no idea how Linux decides to do readahead.
(FreeBSD defaults to 1 readahead, with a mount option that can increase
 that.)

Now, net interfaces normally do interrupt  moderation. This is done to
avoid an interrupt storm during bulk data transfer. However, interrupt
moderation results in interrupt delay for handling the small Read request
message.
--> Interrupt moderation can increase RPC RTT. Turning it off, if possible,
      might help.

So, ping the server from the client to see what your RTT roughly is.
Also, you could look at some traffic in wireshark, to see what readahead
is happening and what the RPC RTT is.
(You can capture with "tcpdump", but wireshark knows how to decode
 NFS properly.)

As you can see, RPC traffic is very different from bulk data transfer.

rick

> Btw, writing is a very different story than reading, largely due to the need
> to commit data/metadata to stable storage while writing.
>
> I can't help w.r.t. ZFS nor high performance nets (my fastest is 1Gbps), rick
>
> >  You mention iperf. Please post the options you used when invoking iperf and it's output.
>
> Setting up the NFS client as a "server", since it seems that the
> terminology is a little bit flipped with iperf, here's the output:
>
> -----------------------------------------------------------
> Server listening on 5201 (test #1)
> -----------------------------------------------------------
> Accepted connection from 10.5.5.1, port 11534
> [  5] local 10.5.5.4 port 5201 connected to 10.5.5.1 port 43931
> [ ID] Interval           Transfer     Bitrate
> [  5]   0.00-1.00   sec  3.81 GBytes  32.7 Gbits/sec
> [  5]   1.00-2.00   sec  4.20 GBytes  36.1 Gbits/sec
> [  5]   2.00-3.00   sec  4.18 GBytes  35.9 Gbits/sec
> [  5]   3.00-4.00   sec  4.21 GBytes  36.1 Gbits/sec
> [  5]   4.00-5.00   sec  4.20 GBytes  36.1 Gbits/sec
> [  5]   5.00-6.00   sec  4.21 GBytes  36.2 Gbits/sec
> [  5]   6.00-7.00   sec  4.10 GBytes  35.2 Gbits/sec
> [  5]   7.00-8.00   sec  4.20 GBytes  36.1 Gbits/sec
> [  5]   8.00-9.00   sec  4.21 GBytes  36.1 Gbits/sec
> [  5]   9.00-10.00  sec  4.20 GBytes  36.1 Gbits/sec
> [  5]  10.00-10.00  sec  7.76 MBytes  35.3 Gbits/sec
> - - - - - - - - - - - - - - - - - - - - - - - - -
> [ ID] Interval           Transfer     Bitrate
> [  5]   0.00-10.00  sec  41.5 GBytes  35.7 Gbits/sec                  receiver
> -----------------------------------------------------------
> Server listening on 5201 (test #2)
> -----------------------------------------------------------
>
> On Sun, May 22, 2022 at 3:45 AM John <jwd@freebsd.org> wrote:
> >
> > ----- Adam Stylinski's Original Message -----
> > > Hello,
> > >
> > > I have two systems connected via ConnectX-3 mellanox cards in ethernet
> > > mode.  They have their MTU's maxed at 9000, their ring buffers maxed
> > > at 8192, and I can hit around 36 gbps with iperf.
> > >
> > > When using an NFS client (client = linux, server = freebsd), I see a
> > > maximum rate of around 20gbps.  The test file is fully in ARC.  The
> > > test is performed with an NFS mount nconnect=4 and an rsize/wsize of
> > > 1MB.
> > >
> > > Here's the flame graph of the kernel of the system in question, with
> > > idle stacks removed:
> > >
> > > https://gist.github.com/KungFuJesus/918c6dcf40ae07767d5382deafab3a52#file-nfs_fg-svg
> > >
> > > The longest functions seems like maybe it's the ERMS aware memcpy
> > > happening from the ARC?  Is there maybe a missing fast path that could
> > > take fewer copies into the socket buffer?
> >
> > Hi Adam -
> >
> >    Some items to look at and possibly include for more responses....
> >
> > - What is your server system? Make/model/ram/etc. What is your
> >   overall 'top' cpu utilization 'top -aH' ...
> >
> > - It looks like you're using a 40gb/s card. Posting the output of
> >   'ifconfig -vm' would provide additional information.
> >
> > - Are the interfaces running cleanly? 'netstat -i' is helpful.
> >
> > - Inspect 'netstat -s'. Duplicate pkts? Resends? Out-of-order?
> >
> > - Inspect 'netstat -m'. Denied? Delayed?
> >
> >
> > - You mention iperf. Please post the options you used when
> >   invoking iperf and it's output.
> >
> > - You appear to be looking for through-put vs low-latency. Have
> >   you looked at window-size vs the amount of memory allocated to the
> >   streams. These values vary based on the bit-rate of the connection.
> >   Tcp connections require outstanding un-ack'd data to be held.
> >   Effects values below.
> >
> >
> > - What are your values for:
> >
> > -- kern.ipc.maxsockbuf
> > -- net.inet.tcp.sendbuf_max
> > -- net.inet.tcp.recvbuf_max
> >
> > -- net.inet.tcp.sendspace
> > -- net.inet.tcp.recvspace
> >
> > -- net.inet.tcp.delayed_ack
> >
> > - What threads/irq are allocated to your NIC? 'vmstat -i'
> >
> > - Are the above threads floating or mapped? 'cpuset -g ...'
> >
> > - Determine best settings for LRO/TSO for your card.
> >
> > - Disable nfs tcp drc
> >
> > - What is your atime setting?
> >
> >
> >    If you really think you have a ZFS/Kernel issue, and you're
> > data fits in cache, dump ZFS, create a memory backed file system
> > and repeat your tests. This will purge a large portion of your
> > graph.  LRO/TSO changes may do so also.
> >
> >    You also state you are using a Linux client. Are you using
> > the MLX affinity scripts, buffer sizing suggestions, etc, etc.
> > Have you swapped the Linux system for a fbsd system?
> >
> >    And as a final note, I regularly use Chelsio T62100 cards
> > in dual home and/or LACP environments in Supermicro boxes with 100's
> > of nfs boot (Bhyve, QEMU, and physical system) clients per server
> > with no network starvation or cpu bottlenecks.  Clients boot, perform
> > their work, and then remotely request image rollback.
> >
> >
> >    Hopefully the above will help and provide pointers.
> >
> > Cheers
> >
>