From nobody Mon Sep 23 01:54:43 2024 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4XBmK06fmHz5XF26 for ; Mon, 23 Sep 2024 01:55:00 +0000 (UTC) (envelope-from jdavidlists@gmail.com) Received: from mail-pf1-x436.google.com (mail-pf1-x436.google.com [IPv6:2607:f8b0:4864:20::436]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "WR4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4XBmJz4Wjgz45qM for ; Mon, 23 Sep 2024 01:54:59 +0000 (UTC) (envelope-from jdavidlists@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20230601 header.b=I0FB29A9; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of jdavidlists@gmail.com designates 2607:f8b0:4864:20::436 as permitted sender) smtp.mailfrom=jdavidlists@gmail.com Received: by mail-pf1-x436.google.com with SMTP id d2e1a72fcca58-71971d20a95so2555403b3a.3 for ; Sun, 22 Sep 2024 18:54:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1727056497; x=1727661297; darn=freebsd.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=KSZ+ya1FURC+FEa9Dx0IQEy/svHA3e6zv6svltt9wrI=; b=I0FB29A9wxXYniHak0ecBg+FhVQ7+ZrYO8eh8Fy6ixLiIwadVKqNcf1LayPYx7klig VGAPi3qc4/Yfp3M8J5k7n23kdiEcWK+cHeYjqarRAlirRMEf05YMAocpKf50u9M/kV8x kfPyeRLryVBYlSGpSV0GSSLFSWBiunfsBa8BqDp0VjImfLCptZi9Xk7DnEC9W6vmnt+a DoAtU+3B5kJADh0LhCEtHFrPmpTdxnUdWpCa+JXtLmQobGl0oO3jaOR3luvaJeDfZXX9 WZ64wkEU62zGxt0h+Nn2mOvitB0CFe3ZgHK3X3Tml5VfMQy4d12ia3yYl7JJ3EFTGky7 362g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1727056497; x=1727661297; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=KSZ+ya1FURC+FEa9Dx0IQEy/svHA3e6zv6svltt9wrI=; b=j7IdocAgFQMgZSabSIayCe0emTrVWZ1ze4ch6nX2faqGgZpWX76cRgzv/qIfGyZ00w PApLDuYSGyMUXuQ0ku/CwZyhCiToPiAFb5EhNq0f3Rxsn7AUiRd51d3FRtFWkwutWj/p bUc6bZOYw+7YzTXFZ0xZ08FhxVD7/7nBLFl4Dul54PAH0c+Ot68EUw4ZepB2YBecYvlS L6o2P5zzmqnQEp04bjWCPPzW1OwU9AhCD3Rp1BGFGTOaJhNDcwVAA54wtMjPGmik1RMj inH/OmLOQ1yiRM1grefSS5tWhgUofqWsOymdGfMJBy5n3xGrix12HEYbL9KEr4BbUhxs M3Og== X-Gm-Message-State: AOJu0YyHObn+o/53baAkol3GtPNc/YUhP7dK5SKKOSGQLPfMna04hNZo LbuDH9CDf1lQtU5fHcy9+WLONXQpXKreg0MmUQghkxHgl831ugjrkuTy5XZhyU4S7fYUPXC4dwl rcAM6BITQqsqQ2myHVG3RlyYXM3kF2PjBWtA= X-Google-Smtp-Source: AGHT+IERWf1Fid2h1V1nbYyuDRSb5YIVkm2a2FvcfLN1P/6wK1BQquXtlI4sX1vKPB25fzsq/UYmnYD5djqgVMutTDY= X-Received: by 2002:a05:6a00:124f:b0:717:869c:2c60 with SMTP id d2e1a72fcca58-7199ca3445dmr14560413b3a.26.1727056497170; Sun, 22 Sep 2024 18:54:57 -0700 (PDT) List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@FreeBSD.org MIME-Version: 1.0 References: In-Reply-To: From: J David Date: Sun, 22 Sep 2024 21:54:43 -0400 Message-ID: Subject: Re: panic: nfsv4root ref cnt cpuid = 1 To: Rick Macklem Cc: FreeBSD FS Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Spamd-Result: default: False [-4.00 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-0.997]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20230601]; MIME_GOOD(-0.10)[text/plain]; RCVD_TLS_LAST(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; TO_DN_ALL(0.00)[]; FREEMAIL_FROM(0.00)[gmail.com]; MIME_TRACE(0.00)[0:+]; TAGGED_FROM(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; ARC_NA(0.00)[]; FREEMAIL_TO(0.00)[gmail.com]; FROM_HAS_DN(0.00)[]; MISSING_XM_UA(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; TO_MATCH_ENVRCPT_SOME(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; MID_RHS_MATCH_FROMTLD(0.00)[]; TAGGED_RCPT(0.00)[]; MLMMJ_DEST(0.00)[freebsd-fs@freebsd.org]; RCVD_COUNT_ONE(0.00)[1]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::436:from] X-Rspamd-Queue-Id: 4XBmJz4Wjgz45qM X-Spamd-Bar: --- On Sun, Sep 22, 2024 at 7:22=E2=80=AFPM Rick Macklem wrote: > I think I know what causes the crashes. The attached trivial patch should > work around them, but if you cannot apply a source kernel patch, the > only workaround would be to get rid of "oneopenown". Unfortunately, FreeBSD NFS is unusable for us without oneopenown due to the issues with CPU load once enough Opens pile up. (I think our issues may be at least part of the reason you added the line about shared libraries to the man page.) > If you cannot get rid of the "oneopenown" or apply the kernel source patc= h, > getting rid of the nullfs mount or enabling delegations might also work a= round > this. We're not using delegations because of, "This can only be enabled when the file systems being exported to NFSv4 clients are not being accessed locally on the server." The filesystem is exported read only; updates are infrequent but must be atomic, so they are performed via ZFS snapshot receive on the server. There are two reasons we have been using nullfs. Both are related to our use case: large number of jobs (jails) where this filesystem must appear. The first and simpler issue is reserved port exhaustion. However, the filesystem is read-only and we've made a change on our end to make it the only filesystem exported from that server. So it should be perfectly safe to mount noresvport; that is a very solvable problem with a few changes to our code. The second issue is filesystem caching. With one NFS mount the client machine can (theoretically) keep one copy of the most frequently hit content from this in cache for everybody. If that's the behavior with nullfs but not with separate mounts, switching could lead to an increase in RAM usage or network traffic enough to adversely affect performance. What do you think? If an NFS filesystem needs to be mounted in 300-500 different places, is it better to NFS mount it once and nullfs it 500 times or better to NFS mount it 500 times? You certainly know vastly more about the likely implications than I do. I will find a way to try the patch. Thanks!