From nobody Sat Mar 02 13:40:08 2024 X-Original-To: stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Tn5gc540Wz5ClPy for ; Sat, 2 Mar 2024 13:40:28 +0000 (UTC) (envelope-from rick.macklem@gmail.com) Received: from mail-pl1-x635.google.com (mail-pl1-x635.google.com [IPv6:2607:f8b0:4864:20::635]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Tn5gc3NDjz4jyZ; Sat, 2 Mar 2024 13:40:28 +0000 (UTC) (envelope-from rick.macklem@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-pl1-x635.google.com with SMTP id d9443c01a7336-1dc09556599so28515295ad.1; Sat, 02 Mar 2024 05:40:28 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1709386826; x=1709991626; darn=freebsd.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=1slmapmEdIkS16Pqi+j8NKrv8fgkpwTTKhqXYr7ItA8=; b=m4FA+7kmPARQ7mrLbmeDyEt4f3q+rQBc33/dh8E2r9Ciz1LfH89e5+H0f2JqQuz5iC BZY0y0JD2sJHzu/NRCvfU26pWcWQKKi5zaq0qtAJur+vPApa+fL5OstaheVlK9TNEk5w 3TWdrWKn5iEg7dASVIPoZvjME6fRdTuG6lYvh1ykTSzdx5ohP94PBNQbQD/0c2jKTDfk klnKQBUfEiEqZr8f0GGX3IenEyNn84AdTelAWqdaZFWUXZ2g7rv6bhBQE4iOnswXyQH5 c1nClfaRE73ljDGbauJXZ9AYF35jTZi9whcnby/D0syEkpLLfS/ugrIHfuqOd5Cx+ZNi pNlw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709386826; x=1709991626; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=1slmapmEdIkS16Pqi+j8NKrv8fgkpwTTKhqXYr7ItA8=; b=hDJptOfZOJtRjkEwtRhVUIQQBvQdf4oysZg0FQdTdsh9UXixx3i7sxi9zFFeMuyKN1 0g4cBi4pB9sCXhkrlbTFzXOU9XJHgrGgadRcZuAFulshmX7UQHetTur3ADVGFEXo8H6r yA75QKh+hEdCnOXillg3fQzKW4xpfuY6szaqVbBahv9WsE+MUyg6Y6isnpu+Nbx9fnwe kC8kvUOm4niN/b8QUaoVUnfiUh9JH5Zs8VSlv3Trfw3/JILMFZYKmgI3C6GTinyrZEhr EgrxCfvk5lrrCWWTND+yNBlkQrOWJCIr59/h4sstb2jiDWqriDcBxoe5U+G3I8bJ6mOc fOfA== X-Forwarded-Encrypted: i=1; AJvYcCWWYs2+IoMAyV+4MiUg7ro6UBpB/0qCpXf5LclZ4DCaEqKDiMap0eEfRoBO/0vz9CbNCVapd3EhDqQcQlkC3tKZRNTAIV9sWSJoml4FQ6eE9vy2ndhisZiY X-Gm-Message-State: AOJu0Yw0suNYKAWdrXpgtWkJaFlFdqZDRdJCnzxrtRWmv5qFeMD7cewx 2E9HoppjpVaeXfoh6vQSJ/eHYOgmwIcSBOoiVaJ/jxqkFiwbYhQrvhFa7dX8TdaYFZCs/cDLy4Y rBOwkd2x7XqdsDuYloV1JNPJ2TupAU+4= X-Google-Smtp-Source: AGHT+IFoB+jF2wwvckyMraZ9NPPw+nMbi3VhLgZTi6VCI2bbbThyUIkBybKrR9OQnvA1OMikLdYfwu+0x7soU1G+1O4= X-Received: by 2002:a17:902:82c7:b0:1dc:affb:1f50 with SMTP id u7-20020a17090282c700b001dcaffb1f50mr4399281plz.47.1709386826006; Sat, 02 Mar 2024 05:40:26 -0800 (PST) List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org MIME-Version: 1.0 References: <1020651467.1592.1709280020993@localhost> In-Reply-To: From: Rick Macklem Date: Sat, 2 Mar 2024 05:40:08 -0800 Message-ID: Subject: Re: 13-stable NFS server hang To: Konstantin Belousov Cc: Ronald Klop , Garrett Wollman , stable@freebsd.org, rmacklem@freebsd.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; TAGGED_FROM(0.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US] X-Rspamd-Queue-Id: 4Tn5gc3NDjz4jyZ On Fri, Mar 1, 2024 at 10:51=E2=80=AFPM Konstantin Belousov wrote: > > CAUTION: This email originated from outside of the University of Guelph. = Do not click links or open attachments unless you recognize the sender and = know the content is safe. If in doubt, forward suspicious emails to IThelp@= uoguelph.ca. > > > On Fri, Mar 01, 2024 at 06:23:56AM -0800, Rick Macklem wrote: > > On Fri, Mar 1, 2024 at 12:00=E2=80=AFAM Ronald Klop wrote: > > > > > > Interesting read. > > > > > > Would it be possible to separate locking for admin actions like a cl= ient mounting an fs from traffic flowing for file operations? > > Well, the NFS server does not really have any concept of a mount. > > What I am referring to is the ClientID maintained for NFSv4 mounts, > > which all the open/lock/session/layout state hangs off of. > > > > For most cases, this state information can safely be accessed/modified > > via a mutex, but there are three exceptions: > > - creating a new ClientID (which is done by the ExchangeID operation) > > and typically happens when a NFS client does a mount. > > - delegation Recall (which only happens when delegations are enabled) > > One of the reasons delegations are not enabled by default on the > > FreeBSD server. > > - the DestroyClientID which is typically done by a NFS client during di= smount. > > For these cases, it is just too difficult to do them without sleeping. > > As such, there is a sleep lock which the nfsd threads normally acquire = shared > > when doing NFSv4 operations, but for the above cases the lock is aquire= d > > exclusive. > > - I had to give the exclusive lock priority over shared lock > > acquisition (it is a > > custom locking mechanism with assorted weirdnesses) because without > > that someone reported that new mounts took up to 1/2hr to occur. > > (The exclusive locker waited for 30min before all the other nfsd thre= ads > > were not busy.) > > Because of this priority, once a nfsd thread requests the exclusive l= ock, > > all other nfsd threads executing NFSv4 RPCs block after releasing the= ir > > shared lock, until the exclusive locker releases the exclusive lock. > Normal lockmgr locks + TDP_DEADLKTREAT private thread flag provide the > property of pref. exclusive waiters in presence of the shared waiters. > I think this is what you described above. It also has some weird properties, like if there are multiple requestors for the exclusive lock, once one thread gets it (the threads are nfsd worke= r threads and indistinct), the others that requested an exclusive thread are unblocked without the lock being issued to them. They then check if the exclusive lock is still needed (usually not, since the other thread has dealt with the case where it was needed) and then they can acquire a shared lock. Without this, there were cases where several threads would acquire the exclusive lock and then discover that the lock was not needed and just release it again. It also uses an assortment of weird flags/call args. rick