From nobody Wed Jun 15 03:08:33 2022 X-Original-To: dev-commits-src-main@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 5F5F1858695; Wed, 15 Jun 2022 03:08:42 +0000 (UTC) (envelope-from cy.schubert@cschubert.com) Received: from omta002.cacentral1.a.cloudfilter.net (omta002.cacentral1.a.cloudfilter.net [3.97.99.33]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "Client", Issuer "CA" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4LN9HY4YqZz3rvD; Wed, 15 Jun 2022 03:08:41 +0000 (UTC) (envelope-from cy.schubert@cschubert.com) Received: from shw-obgw-4004a.ext.cloudfilter.net ([10.228.9.227]) by cmsmtp with ESMTP id 15F9orc9FWi4Q1JO3ozEFn; Wed, 15 Jun 2022 03:08:07 +0000 Received: from spqr.komquats.com ([70.66.148.124]) by cmsmtp with ESMTPA id 1JOTotCj2sF601JOUoSmP7; Wed, 15 Jun 2022 03:08:35 +0000 X-Authority-Analysis: v=2.4 cv=Z8n/oVdA c=1 sm=1 tr=0 ts=62a94d33 a=Cwc3rblV8FOMdVN/wOAqyQ==:117 a=Cwc3rblV8FOMdVN/wOAqyQ==:17 a=kj9zAlcOel0A:10 a=JPEYwPQDsx4A:10 a=XW9rUjSBAAAA:8 a=6I5d2MoRAAAA:8 a=pGLkceISAAAA:8 a=YxBL1-UpAAAA:8 a=EkcXrb_YAAAA:8 a=s1nZ-UPooWrn4tJnjxEA:9 a=CjuIK1q_8ugA:10 a=OkXUG-eLNbkA:10 a=s9rocZTD7jChZwxfYMT6:22 a=IjZwj45LgO3ly-622nXo:22 a=Ia-lj3WSrqcvXOmTRaiG:22 a=LK5xJRSDVpKd5WXXoEvA:22 Received: from slippy.cwsent.com (slippy [10.1.1.91]) by spqr.komquats.com (Postfix) with ESMTP id 83677121; Tue, 14 Jun 2022 20:08:33 -0700 (PDT) Received: by slippy.cwsent.com (Postfix, from userid 1000) id 79F9A9B; Tue, 14 Jun 2022 20:08:33 -0700 (PDT) X-Mailer: exmh version 2.9.0 11/07/2018 with nmh-1.7+dev Reply-to: Cy Schubert From: Cy Schubert X-os: FreeBSD X-Sender: cy@cwsent.com X-URL: http://www.cschubert.com/ To: Mateusz Guzik , Doug Ambrisko , Doug Ambrisko cc: src-committers@freebsd.org, dev-commits-src-all@freebsd.org, dev-commits-src-main@freebsd.org, Rick Macklem Subject: Re: git: 6468cd8e0ef9 - main - mount: add vnode usage per file system with mount -v In-reply-to: References: <202206131457.25DEvJDU044469@gitrepo.freebsd.org> Comments: In-reply-to Mateusz Guzik message dated "Mon, 13 Jun 2022 21:56:38 +0200." List-Id: Commit messages for the main branch of the src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-main List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-dev-commits-src-main@freebsd.org X-BeenThere: dev-commits-src-main@freebsd.org Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Date: Tue, 14 Jun 2022 20:08:33 -0700 Message-Id: <20220615030833.79F9A9B@slippy.cwsent.com> X-CMAE-Envelope: MS4xfNywmcCdPj8d02WSLGsLqUI3GvRVZ/sr5fa8JUu4H/3RjmyhohHrFmt9NkVE/2nKN6Oza9iu6/EKbMqt+TJayEytylWZv1i/RAdjP2Md0uP7SCAJ07Sy Bx6rpsgJaM7fP6468yN1vRv4WrBiXIF7EvL7tlT3pk7/pt5AXhfw5lcK2w0DTWnezUemU2bSHpnZ633aQJNgHJ+1wmabzssuVU239uxKog8VCdPf677oJMBJ 6FUyENAiI6iO6JcTIlkNKwFtqJe77hlc55Nt9dn3YwS2JDHPadurU+mXbo5jJRCvkmCukhvXWs+2qK8zPI02Nu1djehUqpbVS2k6w5D6/IdTMkYUNb8q+Fc8 fSyu8pWWiY9rkrn+yKNgQ36sCwWPirgBCRx1ZR9jZHJjOydo3/dmE3u6KMfeeOFJDgm8byAujkKm7EEZsbpXt3/u4WGo0g== X-Rspamd-Queue-Id: 4LN9HY4YqZz3rvD X-Spamd-Bar: ++ Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=none (mx1.freebsd.org: domain of cy.schubert@cschubert.com has no SPF policy when checking 3.97.99.33) smtp.mailfrom=cy.schubert@cschubert.com X-Spamd-Result: default: False [2.05 / 15.00]; HAS_REPLYTO(0.00)[Cy.Schubert@cschubert.com]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_DN_SOME(0.00)[]; MV_CASE(0.50)[]; RCVD_COUNT_THREE(0.00)[4]; NEURAL_HAM_SHORT(-0.31)[-0.309]; RCPT_COUNT_SEVEN(0.00)[7]; FREEMAIL_TO(0.00)[gmail.com,ambrisko.com,freebsd.org]; RECEIVED_SPAMHAUS_PBL(0.00)[70.66.148.124:received]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; ASN(0.00)[asn:16509, ipnet:3.96.0.0/15, country:US]; MIME_TRACE(0.00)[0:+]; RCVD_IN_DNSWL_LOW(-0.10)[3.97.99.33:from]; ARC_NA(0.00)[]; REPLYTO_EQ_FROM(0.00)[]; FROM_HAS_DN(0.00)[]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[cschubert.com: no valid DMARC record]; AUTH_NA(1.00)[]; NEURAL_SPAM_MEDIUM(0.92)[0.918]; TO_MATCH_ENVRCPT_SOME(0.00)[]; NEURAL_SPAM_LONG(0.14)[0.142]; R_SPF_NA(0.00)[no SPF record]; MLMMJ_DEST(0.00)[dev-commits-src-all,dev-commits-src-main]; RCVD_TLS_LAST(0.00)[] X-ThisMailContainsUnwantedMimeParts: N In message , Mateusz Guzik writes: > On 6/13/22, Doug Ambrisko wrote: > > On Mon, Jun 13, 2022 at 06:43:31PM +0200, Mateusz Guzik wrote: > > | On 6/13/22, Doug Ambrisko wrote: > > | > The branch main has been updated by ambrisko: > > | > > > | > URL: > > | > > > https://cgit.FreeBSD.org/src/commit/?id=6468cd8e0ef9d1d3331e9de26cd2be59bc7 > 78494 > > | > > > | > commit 6468cd8e0ef9d1d3331e9de26cd2be59bc778494 > > | > Author: Doug Ambrisko > > | > AuthorDate: 2022-06-13 14:56:38 +0000 > > | > Commit: Doug Ambrisko > > | > CommitDate: 2022-06-13 14:56:38 +0000 > > | > > > | > mount: add vnode usage per file system with mount -v > > | > > > | > This avoids the need to drop into the ddb to figure out vnode > > | > usage per file system. It helps to see if they are or are not > > | > being freed. Suggestion to report active vnode count was from > > | > kib@ > > | > > > | > Reviewed by: kib > > | > Differential Revision: https://reviews.freebsd.org/D35436 > > | > --- > > | > sbin/mount/mount.c | 7 +++++++ > > | > sys/kern/vfs_mount.c | 12 ++++++++++++ > > | > sys/sys/mount.h | 4 +++- > > | > 3 files changed, 22 insertions(+), 1 deletion(-) > > | > > > | > diff --git a/sbin/mount/mount.c b/sbin/mount/mount.c > > | > index 79d9d6cb0caf..bd3d0073c474 100644 > > | > --- a/sbin/mount/mount.c > > | > +++ b/sbin/mount/mount.c > > | > @@ -692,6 +692,13 @@ prmount(struct statfs *sfp) > > | > xo_emit("{D:, }{Lw:fsid}{:fsid}", fsidbuf); > > | > free(fsidbuf); > > | > } > > | > + if (sfp->f_nvnodelistsize != 0 || sfp->f_avnodecount != > 0) { > > | > + xo_open_container("vnodes"); > > | > + xo_emit("{D:, > > | > }{Lwc:vnodes}{Lw:count}{w:count/%ju}{Lw:active}{:active/%ju}", > > | > + (uintmax_t)sfp->f_nvnodelistsize, > > | > + (uintmax_t)sfp->f_avnodecount); > > | > + xo_close_container("vnodes"); > > | > + } > > | > } > > | > xo_emit("{D:)}\n"); > > | > } > > | > diff --git a/sys/kern/vfs_mount.c b/sys/kern/vfs_mount.c > > | > index 71a40fd97a9c..e3818b67e841 100644 > > | > --- a/sys/kern/vfs_mount.c > > | > +++ b/sys/kern/vfs_mount.c > > | > @@ -2610,6 +2610,8 @@ vfs_copyopt(struct vfsoptlist *opts, const char > > *name, > > | > void *dest, int len) > > | > int > > | > __vfs_statfs(struct mount *mp, struct statfs *sbp) > > | > { > > | > + struct vnode *vp; > > | > + uint32_t count; > > | > > > | > /* > > | > * Filesystems only fill in part of the structure for updates, > we > > | > @@ -2624,6 +2626,16 @@ __vfs_statfs(struct mount *mp, struct statfs > > *sbp) > > | > sbp->f_version = STATFS_VERSION; > > | > sbp->f_namemax = NAME_MAX; > > | > sbp->f_flags = mp->mnt_flag & MNT_VISFLAGMASK; > > | > + sbp->f_nvnodelistsize = mp->mnt_nvnodelistsize; > > | > + > > | > + count = 0; > > | > + MNT_ILOCK(mp); > > | > + TAILQ_FOREACH(vp, &mp->mnt_nvnodelist, v_nmntvnodes) { > > | > + if (vrefcnt(vp) > 0) /* racy but does not matter */ > > | > + count++; > > | > + } > > | > + MNT_IUNLOCK(mp); > > | > + sbp->f_avnodecount = count; > > | > > > | > > | libc uses statfs for dir walk (see gen/fts.c), most notably find > > | immediately runs into it. As such the linear scan by default is a > > | non-starter. > > | > > | I don't know if mount is the right place to dump this kind of info to > > | begin with, but even so, it should only happen with a dedicated flag. > > | > > | As statfs does not take any flags on its own, there is no way to > > | prevent it from doing the above walk. Perhaps a dedicated sysctl which > > | takes mount point id could do the walk instead, when asked. > > | > > | Short of making the walk optional I'm afraid this will have to be > > reverted. > > > > Just to be clear, this isn't breaking things but is not optimal for > > things that don't need this extra info. > > > > It's not "not optimal", it's a significant overhead which taxes > frequent users which don't benefit from it. Indeed this is not optimal. Since this revision NFSv4 performance has tanked. An installworld over NFSv4 which used to take approximately 15 minutes took all night, not even finishing by morning. The NFS server, a 4 core machine, had a load average of 18 with three NFS clients attempting installworld with nfsd using over 90% of the cycles on the machine. I was able to reproduce the problem by running a series of tar cf /dev/null /usr/obj in parallel, on a single NFSv4 client to essentially DoS the NFSv4 server. The workaround was to fall back to NFSv3, which was unaffected by this revision. I reached out to our resident NFS person (rmacklem@) who suggested reverting this revision, restoring the network to pre-regression state. > > For more data I plugged dtrace -n 'fbt::__vfs_statfs:entry { > @[execname] = count(); }' while package building, then i got tons of > hits: > [snip] > expr 13992 > install 14090 > dirname 14921 > mv 17404 > ghc-stage1 17577 > grep 18998 > xgcc 23832 > cpp 29282 > cc1 36961 > sh 70575 > rm 73904 > ld.lld 87784 > sed 88803 > c++ 98175 > cat 115811 > cc 449725 > [...] > -- > Mateusz Guzik > -- Cheers, Cy Schubert FreeBSD UNIX: Web: http://www.FreeBSD.org NTP: Web: https://nwtime.org e**(i*pi)+1=0