From nobody Thu Apr 21 13:44:02 2022 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id ED24211CC9C4 for ; Thu, 21 Apr 2022 13:44:29 +0000 (UTC) (envelope-from Alexander@leidinger.net) Received: from mailgate.Leidinger.net (mailgate.leidinger.net [IPv6:2a00:1828:2000:313::1:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature ECDSA (P-256) client-digest SHA256) (Client CN "mailgate.leidinger.net", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Kkf0X5GL8z4c06 for ; Thu, 21 Apr 2022 13:44:28 +0000 (UTC) (envelope-from Alexander@leidinger.net) Received: from outgoing.leidinger.net (p5b165562.dip0.t-ipconnect.de [91.22.85.98]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-256) server-digest SHA256 client-signature ECDSA (P-256) client-digest SHA256) (Client CN "outgoing.leidinger.net", Issuer "R3" (verified OK)) by mailgate.Leidinger.net (Postfix) with ESMTPSA id 08FA12CA41; Thu, 21 Apr 2022 15:44:12 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=leidinger.net; s=outgoing-alex; t=1650548654; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=I8knDJe2WlQGuPb4/nU7gDzOYDEjFu89Api2hsWjvec=; b=frobDpTVjv+RUdzy1YC851cDfoinPQGpcy5pDHCGOSlj91SDiyB6UUSAIVo3MHOsxXqcpE qC10Eq6S/tS4zGwpylT+QESu+m87sXLgOrW/dVT0JEG5/+Ri+mu6iVOIAK4tfgJiztHC88 HLXxAaAcIJd+xQdWX4zXRhnPJuHSmyzoJaQaRdcVUPXKynLCpkPGfTOFv1cBsdVflQGu1z wEWDhuTo8UBZfoNys+Xt0mnKtCPs12E2by3VweGL9z1R8zKMpmnQRdvncwC9DvFDi9HJaS haLg4rP7LYkiGM1Of+mQfR4zyaOsavPe7hqAqCtZ6WVNVxyuT+392StReG56hQ== Received: from webmail.leidinger.net (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-256) server-digest SHA256) (Client did not present a certificate) by outgoing.leidinger.net (Postfix) with ESMTPS id 3364C3716; Thu, 21 Apr 2022 15:44:04 +0200 (CEST) Date: Thu, 21 Apr 2022 15:44:02 +0200 Message-ID: <20220421154402.Horde.I6m2Om_fxqMtDMUqpiZAxtP@webmail.leidinger.net> From: Alexander Leidinger To: Mateusz Guzik Cc: Doug Ambrisko , freebsd-current@freebsd.org Subject: Re: nullfs and ZFS issues References: <20220420113944.Horde.5qBL80-ikDLIWDIFVJ4VgzX@webmail.leidinger.net> <20220421083310.Horde.r7YT8777_AvGU_6GO1cC90G@webmail.leidinger.net> In-Reply-To: Accept-Language: de,en Content-Type: multipart/signed; boundary="=_mCoQQKdVyX4TDmpIgE_NwAt"; protocol="application/pgp-signature"; micalg=pgp-sha256 List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 X-Rspamd-Queue-Id: 4Kkf0X5GL8z4c06 X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=leidinger.net header.s=outgoing-alex header.b=frobDpTV; dmarc=pass (policy=quarantine) header.from=leidinger.net; spf=pass (mx1.freebsd.org: domain of Alexander@leidinger.net designates 2a00:1828:2000:313::1:5 as permitted sender) smtp.mailfrom=Alexander@leidinger.net X-Spamd-Result: default: False [-4.15 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+mx]; RCVD_COUNT_THREE(0.00)[3]; MID_RHS_MATCH_FROMTLD(0.00)[]; DKIM_TRACE(0.00)[leidinger.net:+]; DMARC_POLICY_ALLOW(-0.50)[leidinger.net,quarantine]; NEURAL_HAM_SHORT(-0.99)[-0.994]; SIGNED_PGP(-2.00)[]; FREEMAIL_TO(0.00)[gmail.com]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; ASN(0.00)[asn:34240, ipnet:2a00:1828::/32, country:DE]; RECEIVED_SPAMHAUS_PBL(0.00)[91.22.85.98:received]; ARC_NA(0.00)[]; R_DKIM_ALLOW(-0.20)[leidinger.net:s=outgoing-alex]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; NEURAL_HAM_LONG(-1.00)[-0.998]; MIME_GOOD(-0.20)[multipart/signed,text/plain]; NEURAL_SPAM_MEDIUM(0.94)[0.940]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MLMMJ_DEST(0.00)[freebsd-current]; RCVD_TLS_ALL(0.00)[] X-ThisMailContainsUnwantedMimeParts: N This message is in MIME format and has been PGP signed. --=_mCoQQKdVyX4TDmpIgE_NwAt Content-Type: text/plain; charset=utf-8; format=flowed; DelSp=Yes Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Quoting Mateusz Guzik (from Thu, 21 Apr 2022=20=20 14:50:42=20+0200): > On 4/21/22, Alexander Leidinger wrote: >> I tried nocache on a system with a lot of jails which use nullfs, >> which showed very slow behavior in the daily periodic runs (12h runs >> in the night after boot, 24h or more in subsequent nights). Now the >> first nightly run after boot was finished after 4h. >> >> What is the benefit of not disabling the cache in nullfs? I would >> expect zfs (or ufs) to cache the (meta)data anyway. >> > > does the poor performance show up with > https://people.freebsd.org/~mjg/vnlru_free_pick.diff ? I would like to have all the 22 jails run the periodic scripts a=20=20 second=20night in a row before trying this. > if the long runs are still there, can you get some profiling from it? > sysctl -a before and after would be a start. > > My guess is that you are the vnode limit and bumping into the 1 second sl= eep. That would explain the behavior I see since I added the last jail=20=20 which=20seems to have crossed a threshold which triggers the slow=20=20 behavior. Current=20status (with the 112 nullfs mounts with nocache): kern.maxvnodes: 10485760 kern.numvnodes: 3791064 kern.freevnodes: 3613694 kern.cache.stats.heldvnodes: 151707 kern.vnodes_created: 260288639 The maxvnodes value is already increased by 10 times compared to the=20=20 default=20value on this system. Bye, Alexander. --=20 http://www.Leidinger.net=20Alexander@Leidinger.net: PGP 0x8F31830F9F2772BF http://www.FreeBSD.org netchild@FreeBSD.org : PGP 0x8F31830F9F2772BF --=_mCoQQKdVyX4TDmpIgE_NwAt Content-Type: application/pgp-signature Content-Description: Digitale PGP-Signatur Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQIzBAABCAAdFiEER9UlYXp1PSd08nWXEg2wmwP42IYFAmJhX6IACgkQEg2wmwP4 2IZTeA//cJhz26bByMcB3GGJiE1hbe/vkBg5GmXXecD5mD6BSDbBoMM3lA5IW/v5 a9JDus3iY+oeAhyJO5ig5znRwVkfpOhHbO9dFqIdp6mwIN1k2+91AMr6SMTEHhnu cvlLVUnlhZeSLzW4ty79iKsLYXyheC06MMrHVawrJs/i22XOS4+zPyT33N/WNMLs UGNqWFuyYw5+PQCGcX46NSbEZYiDLN9AMM9lXwY2RnY8ziX5JnW6lEU88OrW7O/v F5Y2uDj1UPsmkPwsaVtblDpzpcI/bEoR2/YECSjX07kVJ/r5cLI9FMbAxBn4Nbyb mKpYYYX+ORzmS9uKML/pZJUsATCM552AALU8LLtbUyZ2V09uOo3nsICdDbB8CAvi A/E9acuGqQFbHE/iXQrU+U01GmRrb9A+OG7oYMHmMqQccmi5hDYWzXpGzrU/8jGU J2q/32chB2czyYg9QT0U4KVKuxE2Gpn2Vxl60A1EmrLTtnbsbMugJIQOr2fpekYo 1BoFdqyUpfL5VP+GuPiTzCPikLWEAFk4v1wVpdd6vd0MMXcghyTZ9hXoWTAYW80V xnSpT92sI/VoMyjcTjPAwevvdfbSKtSPOg2WKa3/XTJUKCn0m+x2LQwTydjO+olp NP63zlpAwuIQzg46pNdX3PYCy+418Dwb0vh9JkoHKgPxfWsxe/0= =OibP -----END PGP SIGNATURE----- --=_mCoQQKdVyX4TDmpIgE_NwAt--