From nobody Tue Oct 31 20:34:46 2023 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4SKhhh0JFYz4yrWh for ; Tue, 31 Oct 2023 20:35:00 +0000 (UTC) (envelope-from rick.macklem@gmail.com) Received: from mail-pj1-x1034.google.com (mail-pj1-x1034.google.com [IPv6:2607:f8b0:4864:20::1034]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4SKhhg1ffKz4SmR for ; Tue, 31 Oct 2023 20:34:59 +0000 (UTC) (envelope-from rick.macklem@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20230601 header.b=EUbzjv49; spf=pass (mx1.freebsd.org: domain of rick.macklem@gmail.com designates 2607:f8b0:4864:20::1034 as permitted sender) smtp.mailfrom=rick.macklem@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-pj1-x1034.google.com with SMTP id 98e67ed59e1d1-2800f7c8125so156699a91.1 for ; Tue, 31 Oct 2023 13:34:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1698784497; x=1699389297; darn=freebsd.org; h=content-transfer-encoding:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=L13QB782fbIW86zXVH4oxIhAuGHcXG82TH+cU3i43Ec=; b=EUbzjv49JDkUSV/macCeuHgQkAieL1wNVf1RPbOR03MLrWwZccdl5lsqg8Qv50ZM5o 2ipD0gbPzUfgyKdpVkf23+HXyOlOmhAbPtIbV7+eDpMdccVyJPRtlNkVPoUeAW9IShPG CQVnQQnPdH9mfRNtW2TKyCQd2D0+Ra8B+xA7PdGxYtElkiiSfR2922X5JwmrGnVayxUg 50yB9bYbFh8aVXwd+unTU47ZeBZYhZuj+jGJmrjrfmdizgQOwd7To4EAqziRqreYFZQ8 ughssOBYPUBPiErIwR6wvwaWsU2B+yGewj/XMZoV0PsAvEj41dAa5WqZd7j6Hglt0N+n fWBQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1698784497; x=1699389297; h=content-transfer-encoding:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=L13QB782fbIW86zXVH4oxIhAuGHcXG82TH+cU3i43Ec=; b=ekOLnTpVQspd+B2l5raImi+dMxuGonlGx9w2tr5pT7CRXv9WWneQPMwquKMzhurAg+ svwuldo0oj6JX401PosNJ32Cou5PMQC57B/UH9Yee2nlBHpej+hryvLGWiY0URdTXz7k lp1kip98wsyxTMqdrEkC5DHas5Z9g4UN4kLDn/BvK3hDt5wdGKmUMAww6ooKfyat6TxD pjMt8jfAmoFdOlVhTwFL+CYT85rvwo2bmRIl1qdhg0jisH4BfMUQRkQ1Ma55wCFTKZnI 7WAyFqeMCtuNdAZ7LMIjsJzKL5IT8RFMWXymxAWx8d9CYwVv8tQ9/iWGXE3XKZqPwdMA eezg== X-Gm-Message-State: AOJu0YwO3C3hAGKu+PTf2/6w4Ca3z0MsW+5f0csg/kTxeqQbxmSksVZO IyBFMskHXAKS0mA2SAvBBmWTQef5Ad6/GEEerdkxlwqJcw== X-Google-Smtp-Source: AGHT+IFhjfL3NnLZuUaPLS8RehTriRSN78oKa/oruitXW1dbUhpvJbJcjJdZF77Z7W/e2TZpq9cGCs5+6CvE+tvRxdU= X-Received: by 2002:a17:90b:4d8e:b0:280:72b:397d with SMTP id oj14-20020a17090b4d8e00b00280072b397dmr850559pjb.20.1698784497601; Tue, 31 Oct 2023 13:34:57 -0700 (PDT) List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 References: <71fc15b3-521a-4701-9c53-f06570d4b97a@app.fastmail.com> In-Reply-To: From: Rick Macklem Date: Tue, 31 Oct 2023 13:34:46 -0700 Message-ID: Subject: Re: optimising nfs and nfsd To: freebsd-fs@freebsd.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Spamd-Result: default: False [-3.97 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-0.999]; NEURAL_HAM_SHORT(-0.99)[-0.990]; NEURAL_HAM_LONG(-0.98)[-0.981]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20230601]; MIME_GOOD(-0.10)[text/plain]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; FREEMAIL_FROM(0.00)[gmail.com]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; ARC_NA(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::1034:from]; DKIM_TRACE(0.00)[gmail.com:+]; MID_RHS_MATCH_FROMTLD(0.00)[]; TO_DN_NONE(0.00)[]; TAGGED_FROM(0.00)[]; RCVD_COUNT_ONE(0.00)[1]; RCVD_TLS_LAST(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; MIME_TRACE(0.00)[0:+]; MLMMJ_DEST(0.00)[freebsd-fs@freebsd.org] X-Rspamd-Queue-Id: 4SKhhg1ffKz4SmR X-Spamd-Bar: --- On Tue, Oct 31, 2023 at 10:18=E2=80=AFAM void wrote: > > On Mon, Oct 30, 2023 at 05:31:50PM -0700, Rick Macklem wrote: > > >Well, here's a couple more things to look at: > >- Number of nfsd threads. I prefer to set the min/max to the same > > value (which is what the "-n" option on nfsd does). Then, after > > the server has been running for a while in production load, I do: > > # ps axHl | fgrep nfsd > > and I look to see how many of the threads have a TIME of > > 0:00.00. (These are extra tthreads that are not needed.) > > If there is a moderate number of these, I consider it aok. > > If there are none of these, more could improve NFS performance. > > If there are lots of these, the number can be decreased, but they > > don't result in much overhead, so I err on the large # side. > > - If you have min set to less than max, the above trick doesn't > > work, but I'd say that if the command shows the max# of threads, > > it could be increased. > >This number can be configured via options on the nfsd command line. > >If you aren't running nfsd in a jail, you can also fiddle with them via > >the sysctls: > >vfs.nfsd.minthreads > >vfs.nfsd.maxthreads > > root@storage:/root# ps axHl | ug -e server | ug -e "0:00.00" | ug -e rpcs= vc | wc -l > 26 > > How many is a moderate number? Right now there's three clients connected > using version 3, one of the clients is doing a lot of (small) i/o 26 that are doing nothing should be more than enough. (I probably would consider 8 a moderate number.) > > root@storage:/root# ug nfs /etc/rc.conf* > /etc/rc.conf > 19: nfs_server_enable=3D"YES" > > I've not specified -n anywhere, so all defaults, nothing in /etc/exports. > All nfsd capability is via the sharenfs zfs property I think the default is 8/core. Anyhow, you seem to have enough. > > >The caveat is that, if the NFS server is also doing other things, > >increasing the number of nfsd threads can result in nfsd "hogging" > >the system. > > NFSD is the machine's only role, so am concerned only with I guess > high-availability/responsiveness, and tuning, if it needs it so it > functions best with the hardware. > > The cpu on the nfs server is a Xeon E5-2407 @2.2GHz. HT is disabled, > so 4 cpus available, and there's 64GB RAM. I can't do much about the CPU > apart from enabling HT which might not be an issue as there's no > routed ipv4. I can increase the RAM if required though. > > This top(1) output is fairly typical: > > last pid: 55448; load averages: 0.04, 0.12, 0.14 up 12+10:20:03 14:5= 5:45 > 54 processes: 1 running, 53 sleeping > CPU: 0.1% user, 0.0% nice, 4.0% system, 0.1% interrupt, 95.8% idle > Mem: 96K Active, 879M Inact, 182M Laundry, 59G Wired, 1965M Free > ARC: 54G Total, 21G MFU, 31G MRU, 664M Header, 1362M Other > 50G Compressed, 78G Uncompressed, 1.56:1 Ratio > Swap: 8192M Total, 8192M Free > > PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU C= OMMAND > 82935 root 32 -60 0 12M 2276K rpcsvc 0 113:03 4.22%= nfsd > > I could probably set vfs.zfs.arc.max. Right now it's at the default 0 =3D= no maximum. > > >NFSv4 server hash table sizes: > >Run "nfsstat -E -s" on the server after it has been up under production > >load for a while. > > it seems v4 isn't running or able to run. The client gets an error > if I try to specify vers=3D4 in the sharenfs property on the server: You need: nfsv4_server_enable=3D"YES" in /etc/rc.conf. > > zfs set sharenfs=3D"vers=3D4 maproot=3Droot -alldirs -network 192.168.1.0 > -mask 255.255.255.0" zroot/vms-volumes > > then "service mountd restart && service nfsd restart" > > then, on the client > # mount /vms-volumes > [tcp] 192.168.1.102:/zroot/vms-volumes: Permission denied > > removing the vers=3D4 then restart rpcbind & nfsd on the server, it works= fine, > but version 3. You won't get better performance from NFSv4, so if NFSv3 is working for you= , I don't see a rason to change. > > >Look at the section near the end called "Server:". > >The number under "Clients" should be roughly the number of client > >systems that have NFSv4 mounts against the server. > > yep that shows 0 as expected as no v4. Here's the stats with just -s > > Server Info: > Getattr Setattr Lookup Readlink Read Write Create Remove > 85998 942 176996 0 8661219 33713618 0 898 > Rename Link Symlink Mkdir Rmdir Readdir RdirPlus Access > 9 0 0 15 43 62 55 23111 > Mknod Fsstat Fsinfo PathConf Commit > 0 191 39 42 111088 > > Server Write > WriteOps WriteRPC Opsaved > 33713618 33713618 0 > Server Cache > Inprog Non-Idem Misses > 0 0 42775262 > > thanks again for your help and advice, I'm taking notes :D > > The bit following > >The two tunables: > >vfs.nfsd.clienthashsize > >vfs.nfsd.sessionhashsize > > I'll reply to in a followup email None of these matter for NFSv3. rick > > -- >