From nobody Fri May 31 19:39:01 2024 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4VrYN60nv2z5LFcX for ; Fri, 31 May 2024 19:39:18 +0000 (UTC) (envelope-from jdavidlists@gmail.com) Received: from mail-pg1-x52e.google.com (mail-pg1-x52e.google.com [IPv6:2607:f8b0:4864:20::52e]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "WR4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4VrYN46XF5z4Mq0 for ; Fri, 31 May 2024 19:39:16 +0000 (UTC) (envelope-from jdavidlists@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20230601 header.b=W3dBpj8v; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of jdavidlists@gmail.com designates 2607:f8b0:4864:20::52e as permitted sender) smtp.mailfrom=jdavidlists@gmail.com Received: by mail-pg1-x52e.google.com with SMTP id 41be03b00d2f7-6bfd4b88608so1858972a12.1 for ; Fri, 31 May 2024 12:39:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1717184355; x=1717789155; darn=freebsd.org; h=content-transfer-encoding:to:subject:message-id:date:from :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=W4QHMAJ1ILfade4FUBkFt7iM1hDVYntxcHUrWnj+2m4=; b=W3dBpj8viOVe22J0qLweQZQ3hxQm8HyMZBkljIzRS4F77BK5PSPZK1hyRsHrtFfPpW /9ynJrmx7HzndjraS+zqBGE1ttIniuxoMP+1P8zdMPm2Olj+fg/YNsTuHwmBgkVYbV3B frtHP++qpd7mQYodUT3O1HxKBStJaIctYjp8dtYJteS74SymyCQcqyYB2c8NffMnqYYC Yg/geze1yCSL4OxJWAJII/dCAU67WIeNmRNIlVpLRJhefYh0H1noQNYjZgyBkqmkXpFY fArCLXzxB7mb5djlaluqYCxut2O3hCgVaxIQ2eKcrgfPVfbIdtBOQFGbigaUNqZgtW7G 1qGQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1717184355; x=1717789155; h=content-transfer-encoding:to:subject:message-id:date:from :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=W4QHMAJ1ILfade4FUBkFt7iM1hDVYntxcHUrWnj+2m4=; b=F9R+vdGqraAJWmcb000yReFo/EvhRvcy3dGkJ/+VdQYcd9qTLF5ANVQQ8mREtmeN44 EhWewG+4NKAW5R75F8wGPaqWUX0MvZ73eOUm1RLvxydRT0fmE5MT0FmCKcVkVJJENbj2 xYpvcoS4JjahXsUj1/qgPFczS2jV/VDMaZSdztSYNlck+ZJTkC2RFa7kmPiN6QRCIayt Oix2X1T6HhXjYWGLB2PiZjpfhXjkTlKxsBEoKVDERHWKHU8BSFCgX8MRrnaUp8i3MmgJ RUWn26QAM5WJW+1fiOkXqvLBVz5XmZd2Ya55MpQfzcVYuRJe/rGPg4s3205T5a8Cemp+ 30Wg== X-Gm-Message-State: AOJu0YxoDELxYveCTJsb9d2AAEYZ1Ec4lWZe5GpRWYWk0CdRobEwsk6q VpG821JmKu2e9PI9zYpZI+HDng68q9H2Bq+jn3slK4mSvQdh+0O65rNi1qfXCtQjBFY9eoP2Ny/ lcvseHGGeZ1Knm5udDUTFVi3rGkA+Q2AE X-Google-Smtp-Source: AGHT+IEN8dQPEvFWFaj5qSKewIU+4IJvbdRxFKX/rN14n2VEgmHyHi6w5hj8iGumxIYvJbFtP4eXKMNSKiC4IMZy1Kc= X-Received: by 2002:a17:90a:ac03:b0:2b9:e3:3561 with SMTP id 98e67ed59e1d1-2c1dc5d8b76mr2555527a91.43.1717184354591; Fri, 31 May 2024 12:39:14 -0700 (PDT) List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@FreeBSD.org MIME-Version: 1.0 From: J David Date: Fri, 31 May 2024 15:39:01 -0400 Message-ID: Subject: NFSv4 hangs on 13.3 To: FreeBSD FS Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Spamd-Bar: --- X-Spamd-Result: default: False [-3.99 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-0.999]; NEURAL_HAM_SHORT(-1.00)[-0.995]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20230601]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; MIME_GOOD(-0.10)[text/plain]; RCVD_TLS_LAST(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; ARC_NA(0.00)[]; FROM_HAS_DN(0.00)[]; MIME_TRACE(0.00)[0:+]; RCPT_COUNT_ONE(0.00)[1]; FREEMAIL_ENVFROM(0.00)[gmail.com]; FREEMAIL_FROM(0.00)[gmail.com]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; MID_RHS_MATCH_FROMTLD(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; FROM_EQ_ENVFROM(0.00)[]; TAGGED_FROM(0.00)[]; MLMMJ_DEST(0.00)[freebsd-fs@freebsd.org]; RCVD_COUNT_ONE(0.00)[1]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; MISSING_XM_UA(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::52e:from] X-Rspamd-Queue-Id: 4VrYN46XF5z4Mq0 In an attempt to narrow down the problems we have with NFS v4.2, we've set up a FreeBSD 13.3 server that mirrors some read-only data from our Linux NFS servers. We are still observing occasional hangs on the FreeBSD 13.3 client machines= . When these hangs occur, anything that touches the mountpoint (including "umount -N") hangs indefinitely. There is nothing in dmesg. The only even slightly informative data I could gather is an "ls /mount/point" kstack: $ sudo procstat -kk 94676 PID TID COMM TDNAME KSTACK 94676 976876 ls - mi_switch+0xbf sleeplk+0xea lockmgr_slock_hard+0x3a5 nfs_lock+0x29 vop_sigdefer+0x2a _vn_lock+0x47 vfs_cache_root+0x9d vfs_root_sigdefer+0x35 lookup+0x88a namei+0x24a kern_statat+0xf8 sys_fstatat+0x27 amd64_syscall+0x110 fast_syscall_common+0xf8 During this time, the NFS server continues serving other clients, and the affected client can access other NFS servers without hanging. The "nfsstat -m" for this mountpoint looks like this: nfsv4,minorversion=3D2,oneopenown,tcp,resvport,nconnect=3D1,hard,cto,nolock= d,sec=3Dsys,acdirmin=3D3,acdirmax=3D60,acregmin=3D5,acregmax=3D60,nametimeo= =3D60,negnametimeo=3D60,rsize=3D65536,wsize=3D65536,readdirsize=3D65536,rea= dahead=3D1,wcommitsize=3D16777216,timeout=3D120,retrans=3D2147483647 with these mount flags in /etc/fstab: ro,nfsv4,minorversion=3D2,tcp,nosuid,noatime,nolockd,oneopenown Is there anything more we can do to help identify this issue? Thanks for any advice!