From nobody Fri Sep 29 22:55:00 2023 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Ry5KH5Drnz4tnj6 for ; Fri, 29 Sep 2023 22:55:15 +0000 (UTC) (envelope-from jdavidlists@gmail.com) Received: from mail-pj1-x102c.google.com (mail-pj1-x102c.google.com [IPv6:2607:f8b0:4864:20::102c]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Ry5KF4szbz3gW9 for ; Fri, 29 Sep 2023 22:55:13 +0000 (UTC) (envelope-from jdavidlists@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20230601 header.b=W9y4ifSd; spf=pass (mx1.freebsd.org: domain of jdavidlists@gmail.com designates 2607:f8b0:4864:20::102c as permitted sender) smtp.mailfrom=jdavidlists@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-pj1-x102c.google.com with SMTP id 98e67ed59e1d1-2776ca9adb7so6981371a91.1 for ; Fri, 29 Sep 2023 15:55:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1696028112; x=1696632912; darn=freebsd.org; h=to:subject:message-id:date:from:mime-version:from:to:cc:subject :date:message-id:reply-to; bh=E8ke00uF3hQ8//rJNdzGH7d5/cJScLpTiHxaW7H1qZ8=; b=W9y4ifSdiE6XBQuuMf1+N2uUrhzVqjh8eFslGtC26Uw1iz4FPN1sRIeMAM/yyO9/eV 6yF7slN/HtiTrlCskGB6LYxKkCFbXYQISHuVBp2jeY9IKIhMo/HEtj3vphXw9gGW616/ neR4plTdSfGremIvD3zqMmpWH3NbPTVbnaFu9bmiTA7XClnX60MlWcHSWZUDDCtb9fcE IixuNbRHdPTcEMFDQJoRcyAdj9u0EA905K3bgLZOGH9K9InQc1EW67AKSkAwjgoA9DlM h+7WNvkfWnn3+I1cW+MowCN9Ye6bWviPSYtJiDtC5d1ffJhrqZFWWb3d2pmKCuULvZRU 7mCA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1696028112; x=1696632912; h=to:subject:message-id:date:from:mime-version:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=E8ke00uF3hQ8//rJNdzGH7d5/cJScLpTiHxaW7H1qZ8=; b=KKuNtkl2mqLI4kSGTWdfnexLgVgV9gvkP6zKMDBsVHy3VU82QRAFuUiW22FHY/nci2 39SwSGzJ0/84t6Mie13FLuyZamuNZ167/YhN38Lpj11gBmlqsLaVFcoI5e4u/UkWtktV WYHEjGauRYA4zv0lVwzBEzX3tssxpfWChWjOYOLTMwqrbDT844CtkyQ20ZeK7HvpQPBC IVWZBrRm605s8/j3XYzg4kdqfvFtEFjrWV9NWDuXHoQmIG1JLcPQALizCm/3Fky9Tda5 43UC1cAEKCxMH5TlPa2abhvcACcxhmYZk3Mc4dP0/YqijIu8ZUWbQ9YH2uFr+jrgOKrK QBXg== X-Gm-Message-State: AOJu0YxATGQZJx1K7kS7YNRyZcSfKUd15ItnCdjagOA/r57msgGL6xDS zlPohzpVfD3N5z34OCNhpZt6qaD3qR1ZnDBTdKkwWju3gBk= X-Google-Smtp-Source: AGHT+IEB8k7RH5b+bWqsHLyOnB5rl4KKezcXTkpuKaDhieV0BU2EYQfCXpCT6PUPXJSO4kTQwzXNRGY30ffA5ULprdg= X-Received: by 2002:a17:90a:7345:b0:271:8195:8 with SMTP id j5-20020a17090a734500b0027181950008mr4992708pjs.36.1696028111631; Fri, 29 Sep 2023 15:55:11 -0700 (PDT) List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 From: J David Date: Fri, 29 Sep 2023 18:55:00 -0400 Message-ID: Subject: FreeBSD 13.2 NFS client mount hangs To: FreeBSD FS Content-Type: text/plain; charset="UTF-8" X-Spamd-Bar: --- X-Spamd-Result: default: False [-3.83 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-0.99)[-0.990]; NEURAL_HAM_SHORT(-0.84)[-0.838]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36:c]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20230601]; MIME_GOOD(-0.10)[text/plain]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; FREEMAIL_FROM(0.00)[gmail.com]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; ARC_NA(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::102c:from]; DKIM_TRACE(0.00)[gmail.com:+]; TO_DN_ALL(0.00)[]; MID_RHS_MATCH_FROMTLD(0.00)[]; TAGGED_FROM(0.00)[]; RCVD_COUNT_ONE(0.00)[1]; RCVD_TLS_LAST(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; MIME_TRACE(0.00)[0:+]; MLMMJ_DEST(0.00)[freebsd-fs@freebsd.org] X-Rspamd-Queue-Id: 4Ry5KF4szbz3gW9 I have noticed a new (to me) hang on FreeBSD NFS client machines running 13.2-RELEASE-p2. It's happened twice this week to Apache processes. It's the root EUID process and it appears to happen while the process is starting up or reconfiguring. I.e., while it's reading the configs. The configs are not on NFS storage. But the vhost document roots are. The process ps looks like this: 0 19557 19548 3 25 5 25248 12036 nfstry DN - 0:12.85 /usr/local/apache/2.4/bin/httpd -D FOREGROUND -f /usr/local/apache/2.4/conf/httpd.conf The procstat -kk looks like: PID TID COMM TDNAME KSTACK 19557 100341 httpd - mi_switch+0xc2 sleepq_timedwait+0x2f _sleep+0x1ce clnt_vc_call+0x866 clnt_reconnect_call+0x626 newnfs_request+0xc36 nfscl_request+0x5a nfsrpc_getattr+0xbb nfs_close+0x489 vop_sigdefer+0x2b VOP_CLOSE_APV+0x1c vn_close1+0x16a vn_closefile+0x3d _fdrop+0x11 closef+0x24b closefp_impl+0x69 amd64_syscall+0x10c fast_syscall_common+0xf8 The process slowly gains CPU time (a few hundredths per minute) but is immune to kill -9 so it doesn't seem to be coming out of the kernel at any point. I tried running procstat -kk every few seconds to see if I would get anything different to show what it's doing. Most are the same as above, but I also got this: 19557 100341 httpd - mi_switch+0xc2 sleepq_timedwait+0x2f _sleep+0x1ce nfs_catnap+0x47 newnfs_request+0x14b3 nfscl_request+0x5a nfsrpc_getattr+0xbb nfs_close+0x489 vop_sigdefer+0x2b VOP_CLOSE_APV+0x1c vn_close1+0x16a vn_closefile+0x3d _fdrop+0x11 closef+0x24b closefp_impl+0x69 amd64_syscall+0x10c fast_syscall_common+0xf8 (This differs starting at the newnfs_request after nfscl_request+0x5a.) I started unmounting NFS filesystems until I hit one where umount hung. An ls on that filesystem also hung. However, an ls of that filesystem from another client machine worked fine, so it does appear to be a client-side issue rather than a server problem. umount -f also hung. umount -N did unmount it very quickly and that caused all the hanging umounts and the httpd process to exit immediately. I didn't find anything good in the syslog or dmesg. The only thing related to nfs are a handful of "nfsv4 err=10068" that look like they were way back near when the system booted (about 5 days ago). The mount flags are: nfsv4,minorversion=2,oneopenown,tcp,resvport,nconnect=1,hard,cto,sec=sys,acdirmin=3,acdirmax=60,acregmin=5,acregmax=60,nametimeo=60,negnametimeo=60,rsize=65536,wsize=65536,readdirsize=65536,readahead=1,wcommitsize=16777216,timeout=120,retrans=2147483647 Is there any other information I could provide or try to catch next time that would help debug this? Thanks!