From nobody Mon May 30 03:07:59 2022 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 3141B1B5C220 for ; Mon, 30 May 2022 03:08:09 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from CAN01-QB1-obe.outbound.protection.outlook.com (mail-qb1can01on2066.outbound.protection.outlook.com [40.107.66.66]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "DigiCert Cloud Services CA-1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4LBL2H46R5z4vjq; Mon, 30 May 2022 03:08:07 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=lhysXhYT3LuHpFf3z25Zw5ZSjWvxk4AZ734GgdXgHSW5F/JmrFcmNeoB4aWpunxm+ZxIPH6mbjZxP86n/4oK/yCCJSwxNBCuC+YgtEtur8RYtDqIFtP5RHEsphrnI2p+/WjdzKNc2/xTphVVbFXoCN8wAACFD71FcZzZZz2PyuguOEGa7PneEb8bXGVvL7KSahJhOL3bAOVvQzszgY8yHo1kNeL6KyYuBd7hZs+p2/SEWfTfAtzGeCJnetB97UHI3xymn7Ffuyr2CU9JXUTmptQNVs9Ih+mYiz6HiVtKqquWseDqUnvss6ui9jdRhrA/e/SYWQ7WeMrgx5bi+whlVQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=LpEkLBAfMDTxR4FKGXnG4/pt6nXPCa+ogSPDpW4XFVc=; b=csZ8x2qwDjvpWbpJeqRwlI/PHiQjwhcWgIUf3wnEAPJRJnYAMlqJAQTTKNoXvWIKQOlu0DRzsvXdSIvLZ+zwdjWA8Jgn1HottfgFOnobS3w4gBTIuIVno7N1HIpt2SvIip2tnjNexbabY+xx8ICQ+V9iVh5EFSd8ZAL1alAh98PmdCvaq0sUSl1Ox5gCPpKxwbzfZ5G8hj0yONJLJUEVRIeISORvAzMq230bh+pqff7JgfMJQc/Q1lZdhg5f4AEN0A6PmmCYEkQJQMME/lBBES7XnVZ1OB9sPQAsf0cNx8H9643DVohHMBduSIw5OR9565UksapahqLHEdBphYbHPw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=uoguelph.ca; dmarc=pass action=none header.from=uoguelph.ca; dkim=pass header.d=uoguelph.ca; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=uoguelph.ca; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=LpEkLBAfMDTxR4FKGXnG4/pt6nXPCa+ogSPDpW4XFVc=; b=a4vXmmV6qMAliEk0/EIM8l60Ytx0Rg5cP6FxwxJOW//vAFDutPSnrprx9c/0BR10AwRzTj3DUjdubvZ/EHq9pRArDJzLpXxdOsuHTN0wctA2sdocEcvPjCZliS8Zmr6S0Dtnqs/jjy5hKT/HORIoJgEcoxLlcxbr9NPB3DUVw5UkjmGpJ96YlIe+KYKuNZgExnl7h/gecHTsLEKNH/clBijXkGd5Ux0j7x1iR4LVq3hsY8gScXl4Y9XsRnH1fyEK7nYENXoTaRPonf2nNRDmNFEg1YoiBiFIyOTla52+7OOPRmVVo4ky/mXxly949hRdq9GNRE2GlwgzOxOuKJLbDw== Received: from YQBPR0101MB9742.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:c01:81::14) by YTBPR01MB3039.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:b01:22::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5293.13; Mon, 30 May 2022 03:08:00 +0000 Received: from YQBPR0101MB9742.CANPRD01.PROD.OUTLOOK.COM ([fe80::b921:251e:4a0b:54fc]) by YQBPR0101MB9742.CANPRD01.PROD.OUTLOOK.COM ([fe80::b921:251e:4a0b:54fc%6]) with mapi id 15.20.5293.019; Mon, 30 May 2022 03:07:59 +0000 From: Rick Macklem To: Kurt Jaeger CC: "freebsd-fs@freebsd.org" Subject: Re: FreeBSD 12.3/13.1 NFS client hang Thread-Topic: FreeBSD 12.3/13.1 NFS client hang Thread-Index: AQHYcgY+3LPS/CtmUk+ZVTnbRrY6qK0zLx2fgADCMYCAAHUPJ4AAZZKmgAF0ZoCAAH2M+g== Date: Mon, 30 May 2022 03:07:59 +0000 Message-ID: References: In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: suggested_attachment_session_id: 1f16b545-96f3-33d4-8dd5-769cc4e28e2b x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 66469f0f-bbfa-40e3-1a19-08da41e99a55 x-ms-traffictypediagnostic: YTBPR01MB3039:EE_ x-microsoft-antispam-prvs: x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: NwM8F6K50YwLtKEoAIciPIx/oj/FGMiks08nBptWH0LADs3J/c3V/ha8TRcXiqwswzku3UILkSxmg1Jmd2hOQK7VqdW4vGtOeVKCLSe6c+CZf1FH5jZVwN62YOViu78mkSEbFOqzEMcwCIxluWlo0WcmqrL6OWw/mYDBgz6nGAQMsEIPNTosdwvbejPvLOaduSl9HAjkj2SnK+epOg0cIdmr0Jw6SOcf3rGTXUTzU6bfmc5hCJWUgSiI2pB/zpMkQrDJnUPYGhV0QY8hxNTmABEiY5PVC/KDCuo+GufogUWfz77vUTlXNFP5mV75ZPn03gTqqgzcfvAScSKXPPmj6JVxpj6xFCxoPbvIpxoiYkZBt1OVxpBjhyEGK2BwyeyXB0R8kibyLXntdFyZRNxsRImgUM1HTjpKv8RcGXmWYq9aB3DPv2+5y2nqisikwGiQOQwvxgnbvfusga1klH25rZ+uyHnaiTaGdH3I5SbOvjvGM0xqDUHZmKVRkmdV3mGlAkZZkVn8IfQ4iZHXuI4DlcZAJFxcANeaIxQs4wFR98e/A69RUSyEh+A6H5MUYyKcM1dD2s8g9W1jQCHZsB2ADAX9b2pSyuOOnKl8YIyqOMq1OmJZT1vcvT8lSIF8o4v9oL302ALiJchTa29YcTZRsw72t7Spj7KQ2jLjP81Osiw6lbz1EiFvbpxHc596OjivWnPpCKHJHviMKGa27aTqhQ== x-forefront-antispam-report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:YQBPR0101MB9742.CANPRD01.PROD.OUTLOOK.COM;PTR:;CAT:NONE;SFS:(13230001)(4636009)(366004)(4326008)(8676002)(91956017)(64756008)(66446008)(66556008)(76116006)(66476007)(450100002)(66946007)(8936002)(52536014)(38100700002)(5660300002)(86362001)(186003)(508600001)(33656002)(71200400001)(9686003)(6916009)(786003)(6506007)(7696005)(316002)(2906002)(55016003)(83380400001)(122000001)(38070700005);DIR:OUT;SFP:1101; x-ms-exchange-antispam-messagedata-chunkcount: 2 x-ms-exchange-antispam-messagedata-0: =?iso-8859-1?Q?bcdnkW6BshbmwHMvRIFvt41lzs9XPscZy2CxFDs2QlOoZpXHYFgHb8qcOI?= =?iso-8859-1?Q?1wjWWcrAdLLYIzPRI1GTk2A0KddAjUVfflDnzG7m4VAmKXnNnChPZM809j?= =?iso-8859-1?Q?KdvivrpaZUnT9inip84p6mtlasfLhfTCTBSHzMviTqWuhuvMYhAkFgR0D1?= =?iso-8859-1?Q?Sw/8hhGKrU6XqFAZY6WYkdvxOFtiQ6zDQHWff3pbK5cH/Q1LqyD2w3bSyS?= =?iso-8859-1?Q?nQjOgwnzvIkHtbZ+tqbeZ5prQh8cvuQSzReT3UOvnniQ2B9CasAJ+9+NwX?= =?iso-8859-1?Q?mESv2EVzP6UHgeM6pQQR895bPBnLRLmtTZR5v5Sm7AXF7YV54vvEjdG6tI?= =?iso-8859-1?Q?wKj65SGpQyn41EnxhkcvUAiZRy/uot/9o+2I5z40zDodAO7ofnpdJ8G3Jf?= =?iso-8859-1?Q?8NU89LkTiaKbs1jlY4be+8kmjUcguwATQ/KGZ8CNsFeVKOdKgFHq7YAts7?= =?iso-8859-1?Q?ER1SW4+eQZHEN5SbFMVOcRluRYjFSA7lZAaovx22gTozLH5ESD4KW45XF4?= =?iso-8859-1?Q?wO2pxiC2PLnujPX7UoU3uQm2dFvc4LgZb7VE+qC1yP/JZXMjIGePndW9OW?= =?iso-8859-1?Q?zdUwkEMcjAM+ZCkZwWIBfsli3g4sUN+ItSYu9Rti7aTShUvoFfeGoW/cTv?= =?iso-8859-1?Q?DQZg3LxGJeAT7gfH8JSLquqcIIhpCX+dnyiHzVJ+pwGUHt2cFJWtdLxoL9?= =?iso-8859-1?Q?WM9nLyLz59gc71Xma1TT80IEjgJUelU6iy+TElU52/xJn0bym7FWVYuKkq?= =?iso-8859-1?Q?9e/p3sU9uOxUocuZuslGNxWOgC7vmsY/ui94ZuYQFSi357GxAl+ErJJPqh?= =?iso-8859-1?Q?Aybp4M9ZYZV9c07ILbSREfjg20Xyaso0NoQzfg+HOAT1SxaHttJA/su9Jm?= =?iso-8859-1?Q?zB/Mq/HbBye1KLV/dCK6NUzwL5Hh1gs7YOcqZt6rL5m5PY36+7/9ArfpuE?= =?iso-8859-1?Q?/e1jt8B7EvoN34UGkz1SWdCHlBpxb12Xlwznh3f1WG310cxTqicW8Ahqyo?= =?iso-8859-1?Q?zWEjyodXYCupGI0t0H/F8O/607kFklniWtmBMmpWBNZx6wjxI5v0dtUi/j?= =?iso-8859-1?Q?KmoSQxB1Nbmj9yK/VvykcbHYtv9TwCXIEJvm2o6IQBO7K3QvSsL16EUIMq?= =?iso-8859-1?Q?O4Y6uLhv/HdpWyYczJt6VDB0fdDek96R/trgpwo5fudFSFR+x8Y1InhNyS?= =?iso-8859-1?Q?vHfLUaM+2yfzLq15FM4Fd2ayD4VQOnY0pjKqxzqxfilGkAyjkc4gxGgXxK?= =?iso-8859-1?Q?2Hg6/XHLMg0oSzkMeO2XXOzI4HgjpzKuHC5TFfbYJPXLpkTnyoK9Y/0NFE?= =?iso-8859-1?Q?Tck+3TRCBtxntJ48yWAP7+wId+HviimbGIN0CaU5vJEeXIrdWXXfy+/z9c?= =?iso-8859-1?Q?fCRPPLfyuCY0EWYhYh24FZCM12RxLrUNzCm0HqulYsQvyuS2WnzY2gombf?= =?iso-8859-1?Q?9XP6VyxnYauyMWytBzs2khZzXGPNnC2GySpVPbKlGvO80FTZhK9q7ZLfVW?= =?iso-8859-1?Q?nDyYZOOX8X/mBgJjwo8imVeOsZi5yV4wgTDnIDv6Q4YgJVVWvzV5LnTgLw?= =?iso-8859-1?Q?8qqsaoOKDpLiqqGakAgkevqmZ4tTooSql9mzJ2uoyMqm0j9rVJeBkCIKsf?= =?iso-8859-1?Q?bC9zMjvlOsEhlfgGnmTpCrErJqtLIrIdj8zcRufGA/VGZV0Q4PJIOWKFgP?= =?iso-8859-1?Q?88RgczSPXIhXds4/bylV9/Zt7D8NCGzBFCnPvVavBV2EDb52EGZv2yDAeN?= =?iso-8859-1?Q?xuN7Nox6+gs2VW+8h598kVs4wftRSGLCwHlH4BmLQHWIzgsIbsjL8jAnZz?= =?iso-8859-1?Q?2c8rRqeeWS5iqd727K7RZPuj+9hFiVaQpOH1y/OXEl7XL4fp7OkAnq5YXj?= =?iso-8859-1?Q?ii?= x-ms-exchange-antispam-messagedata-1: WeFgzrRP/9GVlw== Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 X-OriginatorOrg: uoguelph.ca X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: YQBPR0101MB9742.CANPRD01.PROD.OUTLOOK.COM X-MS-Exchange-CrossTenant-Network-Message-Id: 66469f0f-bbfa-40e3-1a19-08da41e99a55 X-MS-Exchange-CrossTenant-originalarrivaltime: 30 May 2022 03:07:59.8488 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: be62a12b-2cad-49a1-a5fa-85f4f3156a7d X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: teKWuH6hWXy1kLnaOQKCzN8LCZ8DVOAfft8nuRFiZw6b0ZinPUSCFia1T0eTTTl++iUknIvMStNG/E8vWA8bpw== X-MS-Exchange-Transport-CrossTenantHeadersStamped: YTBPR01MB3039 X-Rspamd-Queue-Id: 4LBL2H46R5z4vjq X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=uoguelph.ca header.s=selector2 header.b=a4vXmmV6; arc=pass ("microsoft.com:s=arcselector9901:i=1"); dmarc=pass (policy=none) header.from=uoguelph.ca; spf=pass (mx1.freebsd.org: domain of rmacklem@uoguelph.ca designates 40.107.66.66 as permitted sender) smtp.mailfrom=rmacklem@uoguelph.ca X-Spamd-Result: default: False [-4.62 / 15.00]; TO_DN_EQ_ADDR_SOME(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; R_DKIM_ALLOW(-0.20)[uoguelph.ca:s=selector2]; FREEFALL_USER(0.00)[rmacklem]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:40.107.0.0/16]; MIME_GOOD(-0.10)[text/plain]; NEURAL_SPAM_SHORT(0.38)[0.378]; NEURAL_HAM_LONG(-1.00)[-1.000]; DWL_DNSWL_LOW(-1.00)[uoguelph.ca:dkim]; RCVD_COUNT_THREE(0.00)[3]; DKIM_TRACE(0.00)[uoguelph.ca:+]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[40.107.66.66:from]; DMARC_POLICY_ALLOW(-0.50)[uoguelph.ca,none]; MLMMJ_DEST(0.00)[freebsd-fs]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:8075, ipnet:40.104.0.0/14, country:US]; ARC_ALLOW(-1.00)[microsoft.com:s=arcselector9901:i=1]; RWL_MAILSPIKE_POSSIBLE(0.00)[40.107.66.66:from] X-ThisMailContainsUnwantedMimeParts: N Kurt Jaeger wrote:=0A= >=0A= > Hi!=0A= >=0A= > > > Kurt Jaeger wrote:=0A= > > > > > > I'm having issues with the NFS clients on FreeBSD 12.3 and 13.1= =0A= =0A= > > > > I have it with an 13.0p7 client against an 13.1 server with=0A= > > > > a hanging soft-mount (I tried unmount to change it to a hard mount)= .=0A= [...]=0A= > > I take this back. I just did a fairly trivial test of this and it worke= d.=0A= >=0A= > > Looking at the "ps" output, I don't think your case is a "NFS protocol = hang".=0A= >=0A= > Today I upgraded the client to 13.1, and changed the nfs mount=0A= > from soft to hard,intr.=0A= >=0A= > nfsstat -M now show this:=0A= >=0A= > # nfsstat -m=0A= > :/serv on /office/serv=0A= >nfsv3,tcp,resvport,nconnect=3D1,hard,intr,cto,lockd,sec=3Dsys,acdirmin=3D3= ,acdirmax=3D60,acregmin=3D5,acregmax=3D60,nametimeo=3D60,negnametimeo=3D60,= rsize=3D65536,wsize=3D65536,readdirsize=3D65536,readahead=3D1,wcommitsize= =3D16777216,timeout=3D120,retrans=3D2=0A= >=0A= Since you are using NFSv3 mounts, the comments w.r.t. hard vs soft are=0A= not relevant.=0A= =0A= > > We use RCS for single files, quite a lot. Sometimes the co -l or the=0A= > > ci -u would not return and hang as well.=0A= >=0A= > We still have rpc.lockd and rpc.statd running on the server,=0A= > are they still used or not ? What happens.=0A= Since you are using NFSv3, if you want the locks to be visible across=0A= multiple clients then, yes, you need to run these.=0A= Alternately, you can use the "nolockd" mount option, which avoids=0A= using rpc.lockd and creates file locks visible within the client only.=0A= (NFSv4 does not use rpc.lockd, rpc.statd.)=0A= =0A= > We also use vi to edit files, and in the past they reliably showed=0A= > us when someone on nfsclient A had a file in vi and someone on=0A= > nfsclient B tried to edit the same file.=0A= > =0A= > Let's see if the problems re-occurs with hard mounts.=0A= It probably will, since you are using NFSv3 and, as I said, the hang you=0A= posted "ps" output for was not even an NFS problem.=0A= --> It appears to be a VM problem.=0A= =0A= > I'll investigate how to move to nfsv4 with a test box as well, if=0A= > this helps ?=0A= Again, I do not think this is an NFS problem, so switching to NFSv4=0A= will not resolve this.=0A= =0A= > > I suspect that some process/thread is hung for something non-NFS=0A= > > while holding a lock on a NFS vnode. "umount -N" won't know how=0A= > > to unhang this process/thread.=0A= > =0A= > > Just a hunch, but I'd suspect one of the threads sleeping on=0A= > > "vmopar", although I'm not a vm guy.=0A= > > What I don't know how to do is figure out what thread(s) are=0A= > > holding vnode locks?=0A= >=0A= > Given that simple NFS mounts and editing with vi on nfs mounts should=0A= > be very basic use-cases, it's time to find ways to debug this 8-(=0A= Look for a VM problem...=0A= =0A= > This also implies that switching from soft->hard won't fix the problem.= =0A= >=0A= > Thanks for the encouragement 8-}=0A= >=0A= > > In summary, I don't think your hang is anything like Andreas's, rick=0A= >=0A= > I'll report as soon as I have a another hang.=0A= I suggest you report this to a mailing list where the VM guys hang=0A= out, with a subject line like "processes hung sleeping on vmopar".=0A= Don't even mention NFS in the subject line. Reporting it here with=0A= NFS in the subject line won't get it seem by the VM people.=0A= (I could be wrong w.r.t. it being a VM problem, but I am pretty=0A= sure it is not an NFS problem. It just happens to result in NFS=0A= vnode being locked, which results in the hangs you observe.)=0A= =0A= rick=0A= =0A= --=0A= pi@FreeBSD.org +49 171 3101372 Now what ?=0A= =0A=