From nobody Sat Mar 02 14:12:40 2024 X-Original-To: stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Tn6P345cvz5CJsf for ; Sat, 2 Mar 2024 14:12:55 +0000 (UTC) (envelope-from kib@freebsd.org) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4Tn6P24N0dz4pvY; Sat, 2 Mar 2024 14:12:54 +0000 (UTC) (envelope-from kib@freebsd.org) Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=fail reason="No valid SPF, No valid DKIM" header.from=freebsd.org (policy=none); spf=softfail (mx1.freebsd.org: 2001:470:d5e7:1::1 is neither permitted nor denied by domain of kib@freebsd.org) smtp.mailfrom=kib@freebsd.org Received: from tom.home (kib@localhost [127.0.0.1] (may be forged)) by kib.kiev.ua (8.18.1/8.18.1) with ESMTP id 422ECew7020064; Sat, 2 Mar 2024 16:12:43 +0200 (EET) (envelope-from kib@freebsd.org) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua 422ECew7020064 Received: (from kostik@localhost) by tom.home (8.18.1/8.18.1/Submit) id 422ECeNt020063; Sat, 2 Mar 2024 16:12:40 +0200 (EET) (envelope-from kib@freebsd.org) X-Authentication-Warning: tom.home: kostik set sender to kib@freebsd.org using -f Date: Sat, 2 Mar 2024 16:12:40 +0200 From: Konstantin Belousov To: Rick Macklem Cc: Ronald Klop , Garrett Wollman , stable@freebsd.org, rmacklem@freebsd.org Subject: Re: 13-stable NFS server hang Message-ID: References: <1020651467.1592.1709280020993@localhost> List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: X-Spam-Status: No, score=-2.9 required=5.0 tests=ALL_TRUSTED,BAYES_00 autolearn=ham autolearn_force=no version=4.0.0 X-Spam-Checker-Version: SpamAssassin 4.0.0 (2022-12-14) on tom.home X-Spamd-Bar: - X-Spamd-Result: default: False [-1.50 / 15.00]; SUSPICIOUS_RECIPS(1.50)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-0.997]; MIME_GOOD(-0.10)[text/plain]; DMARC_POLICY_SOFTFAIL(0.10)[freebsd.org : No valid SPF, No valid DKIM,none]; TAGGED_RCPT(0.00)[]; ARC_NA(0.00)[]; HAS_XAW(0.00)[]; FREEFALL_USER(0.00)[kib]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:6939, ipnet:2001:470::/32, country:US]; TO_DN_SOME(0.00)[]; FREEMAIL_TO(0.00)[gmail.com]; MLMMJ_DEST(0.00)[stable@freebsd.org]; MISSING_XM_UA(0.00)[]; R_SPF_SOFTFAIL(0.00)[~all:c]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_TLS_LAST(0.00)[]; FROM_HAS_DN(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; RCPT_COUNT_FIVE(0.00)[5] X-Rspamd-Queue-Id: 4Tn6P24N0dz4pvY On Sat, Mar 02, 2024 at 05:40:08AM -0800, Rick Macklem wrote: > On Fri, Mar 1, 2024 at 10:51 PM Konstantin Belousov wrote: > > > > CAUTION: This email originated from outside of the University of Guelph. Do not click links or open attachments unless you recognize the sender and know the content is safe. If in doubt, forward suspicious emails to IThelp@uoguelph.ca. > > > > > > On Fri, Mar 01, 2024 at 06:23:56AM -0800, Rick Macklem wrote: > > > On Fri, Mar 1, 2024 at 12:00 AM Ronald Klop wrote: > > > > > > > > Interesting read. > > > > > > > > Would it be possible to separate locking for admin actions like a client mounting an fs from traffic flowing for file operations? > > > Well, the NFS server does not really have any concept of a mount. > > > What I am referring to is the ClientID maintained for NFSv4 mounts, > > > which all the open/lock/session/layout state hangs off of. > > > > > > For most cases, this state information can safely be accessed/modified > > > via a mutex, but there are three exceptions: > > > - creating a new ClientID (which is done by the ExchangeID operation) > > > and typically happens when a NFS client does a mount. > > > - delegation Recall (which only happens when delegations are enabled) > > > One of the reasons delegations are not enabled by default on the > > > FreeBSD server. > > > - the DestroyClientID which is typically done by a NFS client during dismount. > > > For these cases, it is just too difficult to do them without sleeping. > > > As such, there is a sleep lock which the nfsd threads normally acquire shared > > > when doing NFSv4 operations, but for the above cases the lock is aquired > > > exclusive. > > > - I had to give the exclusive lock priority over shared lock > > > acquisition (it is a > > > custom locking mechanism with assorted weirdnesses) because without > > > that someone reported that new mounts took up to 1/2hr to occur. > > > (The exclusive locker waited for 30min before all the other nfsd threads > > > were not busy.) > > > Because of this priority, once a nfsd thread requests the exclusive lock, > > > all other nfsd threads executing NFSv4 RPCs block after releasing their > > > shared lock, until the exclusive locker releases the exclusive lock. > > Normal lockmgr locks + TDP_DEADLKTREAT private thread flag provide the > > property of pref. exclusive waiters in presence of the shared waiters. > > I think this is what you described above. > It also has some weird properties, like if there are multiple requestors > for the exclusive lock, once one thread gets it (the threads are nfsd worker > threads and indistinct), the others that requested an exclusive thread are > unblocked without the lock being issued to them. This sounds to me as LK_SLEEPFAIL feature of lockmgr. Do not underestimate the amount of weird features in it. > They then check if the exclusive lock is still needed (usually not, since > the other thread has dealt with the case where it was needed) and > then they can acquire a shared lock. > Without this, there were cases where several threads would acquire > the exclusive lock and then discover that the lock was not needed and > just release it again. > > It also uses an assortment of weird flags/call args. > > rick >