From nobody Wed Jun 15 15:07:52 2022 X-Original-To: dev-commits-src-main@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 31C7F8508B7; Wed, 15 Jun 2022 15:07:55 +0000 (UTC) (envelope-from ambrisko@ambrisko.com) Received: from mail2.ambrisko.com (mail2.ambrisko.com [70.91.206.91]) by mx1.freebsd.org (Postfix) with ESMTP id 4LNTFP743Jz4hjH; Wed, 15 Jun 2022 15:07:53 +0000 (UTC) (envelope-from ambrisko@ambrisko.com) IronPort-SDR: W8t5mAwJlJ8Bwecshrz8VW33AMdK6VpcklHZMrWP71uDUYR8PYazg1OxOS7l047yfAr7nY8/hh EOu3mYgH8hGFqomO4obTXLz7PjgIVOF1w= X-Ambrisko-Me: Yes IronPort-Data: A9a23:zytfpKAi773cqRVW/63jw5YqxClBgxIJ4kV8jS/XYbTApDMmgj0Pm mUWCGjUOKmOZTagKtB3OY+yp0wH7MLSy9RqHQtv/xmBbZ7qRekppDi9Bh2tV8+qw0GqoHtPt 63yUPGZRCwJZiS0SimFY+GJQUZUhfngqorUUIYoCwgpLeNWYH5JZSFLx4bVtrVVbe2RWGthj z9ST/r3YzdJ0xYsWo4dBjnqRBlH5JwesxtA1rAyiGwiUPYzShAo4JwjyaGZdxMUQ6FVGPK3X eDKyPew+2nD/gwuDZWulbOTnk8iG++Ie1HQ2jwPHfLKbhtq/kTe1o4yMfAGaFxUjB2Amtpry c5Ostq7TgJB0qjkxb9ECEYGe815FegckFPdGlCkq8fWxErDdnrw3/h0F2k4J4AV8P17R2ZU+ pQwLTICcwKChMq5xbu6Tq9ngcFLBMDuNpkeoXJn5T/DAPssWpyFRL/FjfdC2T47hcRDGN7DY MUEZD53bFLLanVnNloJTZk5meKpmmK6aCdT7VuRoYI5+WXexhd9lr/3P7L9dtGUAM5Sm0OZu nmD5H7yRxQTNdq3xyCI/3i3iqnIhyyTcKkDUoaz9uNgjUeP7mcIAx0LX1ahvb+yjUvWZj71A yT44QI1oLIs+VbxCND4VQe5u32D+BUbXrJt/yQBwFnl4sLpD8yxXwDolxZNN44rstEYXzsv2 gPblt/lH2Y24rSQQ2ic7bSThTq3MzIUNmwFIyQDSFJdsdXkpYgyiDPJT8piQPPt14ypQWmoz mDYtjU6ipUSkdUPi/ex82fYjm/+vZPOVAM0uFnaBzr38gNjaYe5TIW08lyHv+1YJYOUQwDZ7 ngJksST9s4UCpSJmHDfSekBBuvwtf+AOifdml1oN5Ak/S6s4H2kO4tX5WgmdktuN88Ffx7vY VPS6V4Jvc4PZCPyYPYuMYyrCskswazxLvjfV6jZPohUf5x8VA6b5yUyN0Se6H/gzRo3mqYlN JbFLcv1VSQGCb5qxSadTvsG1eN53Tg3wG7eSMypzxmj1rbCNneZRa1faQmPaPw09qWNpEPc9 txFNtCJzFNUV+inOnvb9osaLFYrK3knBMCr85UGKrbbegc2Sns8D/LxwK86f903lqtYoe7E4 3WhVxIK01H4n3DGdV2HZywxcr/pRpoj/3s3MTZ2ZQSz1mI9bJz1qq4afYE2ZrogsudkyKcsH fUCfsyBBNVJSyjGq2lFNMih9NQ6eUT5nx+KMgqkfCM7LsxpSAH+89P5ehfiqXsVBS2tuMpi+ 7Ct22s3m3bYq9iO2CoOVM+S8g== IronPort-HdrOrdr: A9a23:bbisyaxg4mndi2MmVhYpKrPwLL1zdoMgy1knxilNoNJuE/Bws/ re+8jztCWE7Ar5N0tNpTntAsa9qDbnhPhICOoqTNKftWvdyQiVxehZhOOIrgEIWReOlNK1vp 0BT0ERMrPN5WgRt6zH3DU= Received: from server2.ambrisko.com (HELO internal.ambrisko.com) ([192.168.1.2]) by ironport2.ambrisko.com with ESMTP; 15 Jun 2022 07:01:21 -0700 Received: from ambrisko.com (localhost [127.0.0.1]) by internal.ambrisko.com (8.17.1/8.17.1) with ESMTPS id 25FF7qDF066629 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO); Wed, 15 Jun 2022 08:07:52 -0700 (PDT) (envelope-from ambrisko@ambrisko.com) X-Authentication-Warning: internal.ambrisko.com: Host localhost [127.0.0.1] claimed to be ambrisko.com Received: (from ambrisko@localhost) by ambrisko.com (8.17.1/8.17.1/Submit) id 25FF7qDT066628; Wed, 15 Jun 2022 08:07:52 -0700 (PDT) (envelope-from ambrisko) Date: Wed, 15 Jun 2022 08:07:52 -0700 From: Doug Ambrisko To: Cy Schubert Cc: Mateusz Guzik , Doug Ambrisko , src-committers@freebsd.org, dev-commits-src-all@freebsd.org, dev-commits-src-main@freebsd.org, Rick Macklem Subject: Re: git: 6468cd8e0ef9 - main - mount: add vnode usage per file system with mount -v Message-ID: References: <202206131457.25DEvJDU044469@gitrepo.freebsd.org> <20220615030833.79F9A9B@slippy.cwsent.com> <20220615140514.77BEDD7@slippy.cwsent.com> List-Id: Commit messages for the main branch of the src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-main List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-dev-commits-src-main@freebsd.org X-BeenThere: dev-commits-src-main@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Rspamd-Queue-Id: 4LNTFP743Jz4hjH X-Spamd-Bar: - Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=none (mx1.freebsd.org: domain of ambrisko@ambrisko.com has no SPF policy when checking 70.91.206.91) smtp.mailfrom=ambrisko@ambrisko.com X-Spamd-Result: default: False [-1.36 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.71)[-0.710]; FREEFALL_USER(0.00)[ambrisko]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; HAS_XAW(0.00)[]; DMARC_NA(0.00)[ambrisko.com]; AUTH_NA(1.00)[]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; NEURAL_HAM_SHORT(-0.65)[-0.653]; RCPT_COUNT_SEVEN(0.00)[7]; MLMMJ_DEST(0.00)[dev-commits-src-all,dev-commits-src-main]; R_SPF_NA(0.00)[no SPF record]; RCVD_NO_TLS_LAST(0.10)[]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:7922, ipnet:70.88.0.0/14, country:US]; FREEMAIL_CC(0.00)[gmail.com,freebsd.org,uoguelph.ca]; MID_RHS_MATCH_FROM(0.00)[] X-ThisMailContainsUnwantedMimeParts: N On Wed, Jun 15, 2022 at 07:23:51AM -0700, Doug Ambrisko wrote: | On Wed, Jun 15, 2022 at 07:05:14AM -0700, Cy Schubert wrote: | | Can we revert this, please. It breaks NFSv4. | | It would be nice if you could try the proposed partial revert. | I'm planning to commit that shortly. It is in ce00b11940ab. Please let me know how that works. Thanks, Doug A. | | In message <20220615030833.79F9A9B@slippy.cwsent.com>, Cy Schubert writes: | | > In message om> | | > , Mateusz Guzik writes: | | > > On 6/13/22, Doug Ambrisko wrote: | | > > > On Mon, Jun 13, 2022 at 06:43:31PM +0200, Mateusz Guzik wrote: | | > > > | On 6/13/22, Doug Ambrisko wrote: | | > > > | > The branch main has been updated by ambrisko: | | > > > | > | | > > > | > URL: | | > > > | > | | > > > https://cgit.FreeBSD.org/src/commit/?id=6468cd8e0ef9d1d3331e9de26cd2be59b | | > c7 | | > > 78494 | | > > > | > | | > > > | > commit 6468cd8e0ef9d1d3331e9de26cd2be59bc778494 | | > > > | > Author: Doug Ambrisko | | > > > | > AuthorDate: 2022-06-13 14:56:38 +0000 | | > > > | > Commit: Doug Ambrisko | | > > > | > CommitDate: 2022-06-13 14:56:38 +0000 | | > > > | > | | > > > | > mount: add vnode usage per file system with mount -v | | > > > | > | | > > > | > This avoids the need to drop into the ddb to figure out vnode | | > > > | > usage per file system. It helps to see if they are or are not | | > > > | > being freed. Suggestion to report active vnode count was from | | > > > | > kib@ | | > > > | > | | > > > | > Reviewed by: kib | | > > > | > Differential Revision: https://reviews.freebsd.org/D35436 | | > > > | > --- | | > > > | > sbin/mount/mount.c | 7 +++++++ | | > > > | > sys/kern/vfs_mount.c | 12 ++++++++++++ | | > > > | > sys/sys/mount.h | 4 +++- | | > > > | > 3 files changed, 22 insertions(+), 1 deletion(-) | | > > > | > | | > > > | > diff --git a/sbin/mount/mount.c b/sbin/mount/mount.c | | > > > | > index 79d9d6cb0caf..bd3d0073c474 100644 | | > > > | > --- a/sbin/mount/mount.c | | > > > | > +++ b/sbin/mount/mount.c | | > > > | > @@ -692,6 +692,13 @@ prmount(struct statfs *sfp) | | > > > | > xo_emit("{D:, }{Lw:fsid}{:fsid}", fsidbuf); | | > > > | > free(fsidbuf); | | > > > | > } | | > > > | > + if (sfp->f_nvnodelistsize != 0 || sfp->f_avnodecount != | | > > 0) { | | > > > | > + xo_open_container("vnodes"); | | > > > | > + xo_emit("{D:, | | > > > | > }{Lwc:vnodes}{Lw:count}{w:count/%ju}{Lw:active}{:active/%ju}", | | > > > | > + (uintmax_t)sfp->f_nvnodelistsize, | | > > > | > + (uintmax_t)sfp->f_avnodecount); | | > > > | > + xo_close_container("vnodes"); | | > > > | > + } | | > > > | > } | | > > > | > xo_emit("{D:)}\n"); | | > > > | > } | | > > > | > diff --git a/sys/kern/vfs_mount.c b/sys/kern/vfs_mount.c | | > > > | > index 71a40fd97a9c..e3818b67e841 100644 | | > > > | > --- a/sys/kern/vfs_mount.c | | > > > | > +++ b/sys/kern/vfs_mount.c | | > > > | > @@ -2610,6 +2610,8 @@ vfs_copyopt(struct vfsoptlist *opts, const char | | > > > *name, | | > > > | > void *dest, int len) | | > > > | > int | | > > > | > __vfs_statfs(struct mount *mp, struct statfs *sbp) | | > > > | > { | | > > > | > + struct vnode *vp; | | > > > | > + uint32_t count; | | > > > | > | | > > > | > /* | | > > > | > * Filesystems only fill in part of the structure for updates, | | > > we | | > > > | > @@ -2624,6 +2626,16 @@ __vfs_statfs(struct mount *mp, struct statfs | | > > > *sbp) | | > > > | > sbp->f_version = STATFS_VERSION; | | > > > | > sbp->f_namemax = NAME_MAX; | | > > > | > sbp->f_flags = mp->mnt_flag & MNT_VISFLAGMASK; | | > > > | > + sbp->f_nvnodelistsize = mp->mnt_nvnodelistsize; | | > > > | > + | | > > > | > + count = 0; | | > > > | > + MNT_ILOCK(mp); | | > > > | > + TAILQ_FOREACH(vp, &mp->mnt_nvnodelist, v_nmntvnodes) { | | > > > | > + if (vrefcnt(vp) > 0) /* racy but does not matter */ | | > > > | > + count++; | | > > > | > + } | | > > > | > + MNT_IUNLOCK(mp); | | > > > | > + sbp->f_avnodecount = count; | | > > > | > | | > > > | | | > > > | libc uses statfs for dir walk (see gen/fts.c), most notably find | | > > > | immediately runs into it. As such the linear scan by default is a | | > > > | non-starter. | | > > > | | | > > > | I don't know if mount is the right place to dump this kind of info to | | > > > | begin with, but even so, it should only happen with a dedicated flag. | | > > > | | | > > > | As statfs does not take any flags on its own, there is no way to | | > > > | prevent it from doing the above walk. Perhaps a dedicated sysctl which | | > > > | takes mount point id could do the walk instead, when asked. | | > > > | | | > > > | Short of making the walk optional I'm afraid this will have to be | | > > > reverted. | | > > > | | > > > Just to be clear, this isn't breaking things but is not optimal for | | > > > things that don't need this extra info. | | > > > | | > > | | > > It's not "not optimal", it's a significant overhead which taxes | | > > frequent users which don't benefit from it. | | > | | > Indeed this is not optimal. Since this revision NFSv4 performance has | | > tanked. An installworld over NFSv4 which used to take approximately 15 | | > minutes took all night, not even finishing by morning. The NFS server, a 4 | | > core machine, had a load average of 18 with three NFS clients attempting | | > installworld with nfsd using over 90% of the cycles on the machine. I was | | > able to reproduce the problem by running a series of tar cf /dev/null | | > /usr/obj in parallel, on a single NFSv4 client to essentially DoS the NFSv4 | | > server. | | > | | > The workaround was to fall back to NFSv3, which was unaffected by this | | > revision. | | > | | > I reached out to our resident NFS person (rmacklem@) who suggested | | > reverting this revision, restoring the network to pre-regression state. | | > | | > > | | > > For more data I plugged dtrace -n 'fbt::__vfs_statfs:entry { | | > > @[execname] = count(); }' while package building, then i got tons of | | > > hits: | | > > [snip] | | > > expr 13992 | | > > install 14090 | | > > dirname 14921 | | > > mv 17404 | | > > ghc-stage1 17577 | | > > grep 18998 | | > > xgcc 23832 | | > > cpp 29282 | | > > cc1 36961 | | > > sh 70575 | | > > rm 73904 | | > > ld.lld 87784 | | > > sed 88803 | | > > c++ 98175 | | > > cat 115811 | | > > cc 449725 | | > > | | > [...] | | > > -- | | > > Mateusz Guzik | | > > | | > | | > | | > -- | | > Cheers, | | > Cy Schubert | | > FreeBSD UNIX: Web: http://www.FreeBSD.org | | > NTP: Web: https://nwtime.org | | > | | > e**(i*pi)+1=0 | | > | | > | |