Re: optimising nfs and nfsd

On Tue, Oct 31, 2023 at 10:18 AM void <void@f-m.fm> wrote:
>
> On Mon, Oct 30, 2023 at 05:31:50PM -0700, Rick Macklem wrote:
>
> >Well, here's a couple more things to look at:
> >- Number of nfsd threads. I prefer to set the min/max to the same
> >  value (which is what the "-n" option on nfsd does).  Then, after
> >  the server has been running for a while in production load, I do:
> >  # ps axHl | fgrep nfsd
> >  and I look to see how many of the threads have a TIME of
> >  0:00.00. (These are extra tthreads that are not needed.)
> >  If there is a moderate number of these, I consider it aok.
> >  If there are none of these, more could improve NFS performance.
> >  If there are lots of these, the number can be decreased, but they
> >  don't result in much overhead, so I err on the large # side.
> >  - If you have min set to less than max, the above trick doesn't
> >    work, but I'd say that if the command shows the max# of threads,
> >    it could be increased.
> >This number can be configured via options on the nfsd command line.
> >If you aren't running nfsd in a jail, you can also fiddle with them via
> >the sysctls:
> >vfs.nfsd.minthreads
> >vfs.nfsd.maxthreads
>
> root@storage:/root# ps axHl | ug -e server | ug -e "0:00.00" | ug -e rpcsvc | wc -l
>        26
>
> How many is a moderate number? Right now there's three clients connected
> using version 3, one of the clients is doing a lot of (small) i/o
26 that are doing nothing should be more than enough. (I probably would
consider 8 a moderate number.)

>
> root@storage:/root# ug nfs /etc/rc.conf*
> /etc/rc.conf
>      19: nfs_server_enable="YES"
>
> I've not specified -n anywhere, so all defaults, nothing in /etc/exports.
> All nfsd capability is via the sharenfs zfs property
I think the default is 8/core. Anyhow, you seem to have enough.

>
> >The caveat is that, if the NFS server is also doing other things,
> >increasing the number of nfsd threads can result in nfsd "hogging"
> >the system.
>
> NFSD is the machine's only role, so am concerned only with I guess
> high-availability/responsiveness, and tuning, if it needs it so it
> functions best with the hardware.
>
> The cpu on the nfs server is a Xeon E5-2407 @2.2GHz. HT is disabled,
> so 4 cpus available, and there's 64GB RAM. I can't do much about the CPU
> apart from enabling HT which might not be an issue as there's no
> routed ipv4. I can increase the RAM if required though.
>
> This top(1) output is fairly typical:
>
> last pid: 55448;  load averages:  0.04,  0.12,  0.14 up 12+10:20:03  14:55:45
> 54 processes:  1 running, 53 sleeping
> CPU:  0.1% user,  0.0% nice,  4.0% system,  0.1% interrupt, 95.8% idle
> Mem: 96K Active, 879M Inact, 182M Laundry, 59G Wired, 1965M Free
> ARC: 54G Total, 21G MFU, 31G MRU, 664M Header, 1362M Other
>       50G Compressed, 78G Uncompressed, 1.56:1 Ratio
>       Swap: 8192M Total, 8192M Free
>
>    PID USERNAME    THR PRI NICE   SIZE    RES STATE    C   TIME    WCPU COMMAND
>    82935 root         32 -60    0    12M  2276K rpcsvc   0 113:03   4.22% nfsd
>
> I could probably set vfs.zfs.arc.max. Right now it's at the default 0 = no maximum.
>
> >NFSv4 server hash table sizes:
> >Run "nfsstat -E -s" on the server after it has been up under production
> >load for a while.
>
> it seems v4 isn't running or able to run. The client gets an error
> if I try to specify vers=4 in the sharenfs property on the server:
You need:
nfsv4_server_enable="YES"
in /etc/rc.conf.

>
> zfs set sharenfs="vers=4 maproot=root -alldirs -network 192.168.1.0
> -mask 255.255.255.0" zroot/vms-volumes
>
> then "service mountd restart && service nfsd restart"
>
> then, on the client
> # mount /vms-volumes
> [tcp] 192.168.1.102:/zroot/vms-volumes: Permission denied
>
> removing the vers=4 then restart rpcbind & nfsd on the server, it works fine,
> but version 3.
You won't get better performance from NFSv4, so if NFSv3 is working for you,
I don't see a rason to change.

>
> >Look at the section near the end called "Server:".
> >The number under "Clients" should be roughly the number of client
> >systems that have NFSv4 mounts against the server.
>
> yep that shows 0 as expected as no v4. Here's the stats with just -s
>
> Server Info:
>        Getattr Setattr Lookup Readlink Read    Write    Create Remove
>        85998   942     176996 0        8661219 33713618 0      898
>        Rename  Link  Symlink  Mkdir    Rmdir Readdir    RdirPlus Access
>        9       0     0        15       43    62         55       23111
>        Mknod       Fsstat       Fsinfo     PathConf       Commit
>            0          191           39           42       111088
>
> Server Write
>       WriteOps     WriteRPC      Opsaved
>       33713618     33713618            0
> Server Cache
>         Inprog     Non-Idem       Misses
>              0            0     42775262
>
> thanks again for your help and advice, I'm taking notes :D
>
> The bit following
> >The two tunables:
> >vfs.nfsd.clienthashsize
> >vfs.nfsd.sessionhashsize
>
> I'll reply to in a followup email
None of these matter for NFSv3.

rick

>
> --
>