From nobody Tue Oct 31 17:18:19 2023 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4SKcKs6szjz4ydQQ for ; Tue, 31 Oct 2023 17:18:25 +0000 (UTC) (envelope-from void@f-m.fm) Received: from wout2-smtp.messagingengine.com (wout2-smtp.messagingengine.com [64.147.123.25]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4SKcKr1Fr1z4Cwg for ; Tue, 31 Oct 2023 17:18:24 +0000 (UTC) (envelope-from void@f-m.fm) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=f-m.fm header.s=fm3 header.b=XSvvxEkq; dkim=pass header.d=messagingengine.com header.s=fm3 header.b=PRAPmetA; spf=pass (mx1.freebsd.org: domain of void@f-m.fm designates 64.147.123.25 as permitted sender) smtp.mailfrom=void@f-m.fm; dmarc=pass (policy=none) header.from=f-m.fm Received: from compute3.internal (compute3.nyi.internal [10.202.2.43]) by mailout.west.internal (Postfix) with ESMTP id 9243B3200A93 for ; Tue, 31 Oct 2023 13:18:22 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute3.internal (MEProxy); Tue, 31 Oct 2023 13:18:22 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=f-m.fm; h=cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:sender :subject:subject:to:to; s=fm3; t=1698772702; x=1698859102; bh=y7 eqnJYtOsotvBMvVFcAmDRmn3aNkTJqoyjnMNVBy+4=; b=XSvvxEkqXGnCFOJhQa cZX+atcNDMHkBbAlpoYVp02TzjJzHvE8Jw/MyVi9WdtFLNL/YNRSvgZy4caNdOQU 6VBlghbe/1g2yJK3wYaJ+QI3vVnGnx5CuCfFAOOxHS5fvYlgel3HIIso542CQOp8 /muVLrWPGey1WWDYJ7StoeK/tOUhuBJkI5CUyWg5VJF2L+1H3Gh9QFOzScdxygcT DfZPlAnPbA79SAgij3Mfezz8PSUILstFrjEe82ohBo2kJqWNQPaOJkWZfDY5izJZ cd8o0K4eprAyQiYKeQX148KwOYSYKR0WcwPlf4gEWin97zGcpCbB8bz8Qgjdi7eB wzqg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:sender:subject :subject:to:to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender :x-sasl-enc; s=fm3; t=1698772702; x=1698859102; bh=y7eqnJYtOsotv BMvVFcAmDRmn3aNkTJqoyjnMNVBy+4=; b=PRAPmetA+8dEcd84GECh083iYo0iR 2+pI1wa0ikl/J5wgaLrCgyd+IHUxtgy3k4+Ojs9piAKFGfXKLHpmbyOo7V9F1vta fP9og/SNoLY3Mimngx2qkhL09VQjZyntN/l3i3ehCyuZXly3kY6MUqPxGkdR2/+6 quNahrnsU6NBN+ufXrhHZ3RxIzwAuUVTstQw5VIHXWrW+QQqcP6eeabmqVMRyATW CUV3ipcA7/VdRI09TDRQcmKFKf3ZzziG8VjghJFS6LvlZxWWxJvK8r8f6qfTTlkW PW1kmSPNghBk2cdRwctq1A1EM9k2Wje6rIFVySA2CrNWPMzqkj4oVOwDw== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvkedruddtvddgleejucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucenucfjughrpeffhffvuffkfhggtggujgesthdtre dttddtvdenucfhrhhomhepvhhoihguuceovhhoihgusehfqdhmrdhfmheqnecuggftrfgr thhtvghrnhepkeeluddvlefhieelfefggffhffektdehleelgfdugfdvgeekjeejuddthe ehgfeunecuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhep vhhoihgusehfqdhmrdhfmh X-ME-Proxy: Feedback-ID: i2541463c:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA for ; Tue, 31 Oct 2023 13:18:21 -0400 (EDT) Date: Tue, 31 Oct 2023 17:18:19 +0000 From: void To: freebsd-fs@freebsd.org Subject: Re: optimising nfs and nfsd Message-ID: Mail-Followup-To: freebsd-fs@freebsd.org References: <71fc15b3-521a-4701-9c53-f06570d4b97a@app.fastmail.com> List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii; format=flowed Content-Disposition: inline In-Reply-To: X-Spamd-Result: default: False [-4.80 / 15.00]; DWL_DNSWL_LOW(-1.00)[messagingengine.com:dkim]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-0.999]; NEURAL_HAM_SHORT(-1.00)[-0.999]; DMARC_POLICY_ALLOW(-0.50)[f-m.fm,none]; MID_RHS_NOT_FQDN(0.50)[]; RWL_MAILSPIKE_VERYGOOD(-0.20)[64.147.123.25:from]; R_DKIM_ALLOW(-0.20)[f-m.fm:s=fm3,messagingengine.com:s=fm3]; R_SPF_ALLOW(-0.20)[+ip4:64.147.123.25]; MIME_GOOD(-0.10)[text/plain]; RCVD_IN_DNSWL_LOW(-0.10)[64.147.123.25:from]; ASN(0.00)[asn:29838, ipnet:64.147.123.0/24, country:US]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; TO_DN_NONE(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; RCPT_COUNT_ONE(0.00)[1]; MLMMJ_DEST(0.00)[freebsd-fs@freebsd.org]; FREEMAIL_FROM(0.00)[f-m.fm]; FREEMAIL_ENVFROM(0.00)[f-m.fm]; TO_MATCH_ENVRCPT_ALL(0.00)[]; DKIM_TRACE(0.00)[f-m.fm:+,messagingengine.com:+]; RCVD_TLS_LAST(0.00)[]; FROM_HAS_DN(0.00)[]; MIME_TRACE(0.00)[0:+]; FROM_EQ_ENVFROM(0.00)[]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[] X-Rspamd-Queue-Id: 4SKcKr1Fr1z4Cwg X-Spamd-Bar: ---- On Mon, Oct 30, 2023 at 05:31:50PM -0700, Rick Macklem wrote: >Well, here's a couple more things to look at: >- Number of nfsd threads. I prefer to set the min/max to the same > value (which is what the "-n" option on nfsd does). Then, after > the server has been running for a while in production load, I do: > # ps axHl | fgrep nfsd > and I look to see how many of the threads have a TIME of > 0:00.00. (These are extra tthreads that are not needed.) > If there is a moderate number of these, I consider it aok. > If there are none of these, more could improve NFS performance. > If there are lots of these, the number can be decreased, but they > don't result in much overhead, so I err on the large # side. > - If you have min set to less than max, the above trick doesn't > work, but I'd say that if the command shows the max# of threads, > it could be increased. >This number can be configured via options on the nfsd command line. >If you aren't running nfsd in a jail, you can also fiddle with them via >the sysctls: >vfs.nfsd.minthreads >vfs.nfsd.maxthreads root@storage:/root# ps axHl | ug -e server | ug -e "0:00.00" | ug -e rpcsvc | wc -l 26 How many is a moderate number? Right now there's three clients connected using version 3, one of the clients is doing a lot of (small) i/o root@storage:/root# ug nfs /etc/rc.conf* /etc/rc.conf 19: nfs_server_enable="YES" I've not specified -n anywhere, so all defaults, nothing in /etc/exports. All nfsd capability is via the sharenfs zfs property >The caveat is that, if the NFS server is also doing other things, >increasing the number of nfsd threads can result in nfsd "hogging" >the system. NFSD is the machine's only role, so am concerned only with I guess high-availability/responsiveness, and tuning, if it needs it so it functions best with the hardware. The cpu on the nfs server is a Xeon E5-2407 @2.2GHz. HT is disabled, so 4 cpus available, and there's 64GB RAM. I can't do much about the CPU apart from enabling HT which might not be an issue as there's no routed ipv4. I can increase the RAM if required though. This top(1) output is fairly typical: last pid: 55448; load averages: 0.04, 0.12, 0.14 up 12+10:20:03 14:55:45 54 processes: 1 running, 53 sleeping CPU: 0.1% user, 0.0% nice, 4.0% system, 0.1% interrupt, 95.8% idle Mem: 96K Active, 879M Inact, 182M Laundry, 59G Wired, 1965M Free ARC: 54G Total, 21G MFU, 31G MRU, 664M Header, 1362M Other 50G Compressed, 78G Uncompressed, 1.56:1 Ratio Swap: 8192M Total, 8192M Free PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND 82935 root 32 -60 0 12M 2276K rpcsvc 0 113:03 4.22% nfsd I could probably set vfs.zfs.arc.max. Right now it's at the default 0 = no maximum. >NFSv4 server hash table sizes: >Run "nfsstat -E -s" on the server after it has been up under production >load for a while. it seems v4 isn't running or able to run. The client gets an error if I try to specify vers=4 in the sharenfs property on the server: zfs set sharenfs="vers=4 maproot=root -alldirs -network 192.168.1.0 -mask 255.255.255.0" zroot/vms-volumes then "service mountd restart && service nfsd restart" then, on the client # mount /vms-volumes [tcp] 192.168.1.102:/zroot/vms-volumes: Permission denied removing the vers=4 then restart rpcbind & nfsd on the server, it works fine, but version 3. >Look at the section near the end called "Server:". >The number under "Clients" should be roughly the number of client >systems that have NFSv4 mounts against the server. yep that shows 0 as expected as no v4. Here's the stats with just -s Server Info: Getattr Setattr Lookup Readlink Read Write Create Remove 85998 942 176996 0 8661219 33713618 0 898 Rename Link Symlink Mkdir Rmdir Readdir RdirPlus Access 9 0 0 15 43 62 55 23111 Mknod Fsstat Fsinfo PathConf Commit 0 191 39 42 111088 Server Write WriteOps WriteRPC Opsaved 33713618 33713618 0 Server Cache Inprog Non-Idem Misses 0 0 42775262 thanks again for your help and advice, I'm taking notes :D The bit following >The two tunables: >vfs.nfsd.clienthashsize >vfs.nfsd.sessionhashsize I'll reply to in a followup email --