From nobody Sat Mar 02 06:51:22 2024 X-Original-To: stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Tmwc50Tzdz5C9HF for ; Sat, 2 Mar 2024 06:51:49 +0000 (UTC) (envelope-from kib@freebsd.org) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4Tmwc41K0pz42cW; Sat, 2 Mar 2024 06:51:48 +0000 (UTC) (envelope-from kib@freebsd.org) Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=fail reason="No valid SPF, No valid DKIM" header.from=freebsd.org (policy=none); spf=softfail (mx1.freebsd.org: 2001:470:d5e7:1::1 is neither permitted nor denied by domain of kib@freebsd.org) smtp.mailfrom=kib@freebsd.org Received: from tom.home (kib@localhost [127.0.0.1] (may be forged)) by kib.kiev.ua (8.18.1/8.18.1) with ESMTP id 4226pM3r013465; Sat, 2 Mar 2024 08:51:25 +0200 (EET) (envelope-from kib@freebsd.org) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua 4226pM3r013465 Received: (from kostik@localhost) by tom.home (8.18.1/8.18.1/Submit) id 4226pMh7013464; Sat, 2 Mar 2024 08:51:22 +0200 (EET) (envelope-from kib@freebsd.org) X-Authentication-Warning: tom.home: kostik set sender to kib@freebsd.org using -f Date: Sat, 2 Mar 2024 08:51:22 +0200 From: Konstantin Belousov To: Rick Macklem Cc: Ronald Klop , Garrett Wollman , stable@freebsd.org, rmacklem@freebsd.org Subject: Re: 13-stable NFS server hang Message-ID: References: <1020651467.1592.1709280020993@localhost> List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: X-Spam-Status: No, score=-2.9 required=5.0 tests=ALL_TRUSTED,BAYES_00 autolearn=ham autolearn_force=no version=4.0.0 X-Spam-Checker-Version: SpamAssassin 4.0.0 (2022-12-14) on tom.home X-Spamd-Bar: - X-Spamd-Result: default: False [-1.50 / 15.00]; SUSPICIOUS_RECIPS(1.50)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-0.998]; MIME_GOOD(-0.10)[text/plain]; DMARC_POLICY_SOFTFAIL(0.10)[freebsd.org : No valid SPF, No valid DKIM,none]; TAGGED_RCPT(0.00)[]; ARC_NA(0.00)[]; HAS_XAW(0.00)[]; FREEFALL_USER(0.00)[kib]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:6939, ipnet:2001:470::/32, country:US]; TO_DN_SOME(0.00)[]; FREEMAIL_TO(0.00)[gmail.com]; MLMMJ_DEST(0.00)[stable@freebsd.org]; MISSING_XM_UA(0.00)[]; R_SPF_SOFTFAIL(0.00)[~all:c]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_TLS_LAST(0.00)[]; FROM_HAS_DN(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; RCPT_COUNT_FIVE(0.00)[5] X-Rspamd-Queue-Id: 4Tmwc41K0pz42cW On Fri, Mar 01, 2024 at 06:23:56AM -0800, Rick Macklem wrote: > On Fri, Mar 1, 2024 at 12:00 AM Ronald Klop wrote: > > > > Interesting read. > > > > Would it be possible to separate locking for admin actions like a client mounting an fs from traffic flowing for file operations? > Well, the NFS server does not really have any concept of a mount. > What I am referring to is the ClientID maintained for NFSv4 mounts, > which all the open/lock/session/layout state hangs off of. > > For most cases, this state information can safely be accessed/modified > via a mutex, but there are three exceptions: > - creating a new ClientID (which is done by the ExchangeID operation) > and typically happens when a NFS client does a mount. > - delegation Recall (which only happens when delegations are enabled) > One of the reasons delegations are not enabled by default on the > FreeBSD server. > - the DestroyClientID which is typically done by a NFS client during dismount. > For these cases, it is just too difficult to do them without sleeping. > As such, there is a sleep lock which the nfsd threads normally acquire shared > when doing NFSv4 operations, but for the above cases the lock is aquired > exclusive. > - I had to give the exclusive lock priority over shared lock > acquisition (it is a > custom locking mechanism with assorted weirdnesses) because without > that someone reported that new mounts took up to 1/2hr to occur. > (The exclusive locker waited for 30min before all the other nfsd threads > were not busy.) > Because of this priority, once a nfsd thread requests the exclusive lock, > all other nfsd threads executing NFSv4 RPCs block after releasing their > shared lock, until the exclusive locker releases the exclusive lock. Normal lockmgr locks + TDP_DEADLKTREAT private thread flag provide the property of pref. exclusive waiters in presence of the shared waiters. I think this is what you described above.