From nobody Mon May 30 04:31:10 2022 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 3FCE11B69718 for ; Mon, 30 May 2022 04:31:19 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from CAN01-YT3-obe.outbound.protection.outlook.com (mail-yt3can01on2086.outbound.protection.outlook.com [40.107.115.86]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "DigiCert Cloud Services CA-1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4LBMtG1k26z3Km9; Mon, 30 May 2022 04:31:18 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=kCsP8ky8UfCtfFJ6rE8wInZb7/RA0eOJiacKc4dMWhojJ0Mq1+SAp8etkcnB/2fjjfD+5mf6tjrz/18pmU2H9bg9xZtBROFa5KNkzmqsed2Lcd3VIx9tZNb4iD4Ny6Np4JdpYoC574jD+aTWk4s3LG0cxw1TeDeqzrmTXxfz6yOMwGM0yO/gFEOzCXTlMH5lAF/kSZhu2cfmiaVW4lJCfgVe1D7jO/mEH1Tx7Gh67V7QeqmCH8YiT7ejyFZxugJDhr0pbfkuCUOiDVYe4uwX0lTGAjarJa1jbC9rotC9dAwJYwrV9z5Et9ziPJByMVJTe75QrNBO3+s4TCgvLf2y7Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=MfEEWOeqKfzfil8X+5whvJHd+ZMRmw5/vGTL7IUr+ME=; b=KIxjWf5RiBGP/+6YkgKLjIZFiHBs7xs3kCFc8qyZo+v+GyrCNx+hS5ad4wZBiqOfDfxUOFNgyb4GmDm3jWoliJdq3aOmqMxL9wyVGo6sTlQ73z2MfBvpoVSp3qAvNxnRhIaRoTX0D8zuXyzgazqDjJaOBwA4ykS7PIaCLa3bs4d4RI26x/3jTNTRV/oGarjzME/paxPb8ZBPPjeDGU7Me58WtDxHafwLVjMlNJ0Rkbwww+yfDX6hFAC+ITvmPOsSzQTA4mcFEYpXqSeXT18ewUkzIenIsSMgCEhgdpv/9N5oiLctnIlLfT1FAMhx5SrdmGfmczxxJopNxQWpIbE4uw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=uoguelph.ca; dmarc=pass action=none header.from=uoguelph.ca; dkim=pass header.d=uoguelph.ca; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=uoguelph.ca; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=MfEEWOeqKfzfil8X+5whvJHd+ZMRmw5/vGTL7IUr+ME=; b=Vt7jyf+c6viOaA/EvoWwKwRBHYletp0GJTCGhldYV2je+AqqtdWYkr90+kOWM1k5KBolOejoIEaAE67GcUR+4VqVuFHmjTkcimEbyPBK63PpJ6w5zQ3WkgOP9RIYcUz3nTYRs+Yw7tGzEdqE7ilrEBAiI8EEJT7DBcyCk8CcpnddMdGbnMSPHchHGR10zGH4pgh9uNY9WqKdR562PkI7VkVj6YzuzSWLHa9yq3lLzFI3+QYRXebfnYmkf5ENFACwBjfqVcTlPqDphcYWgH/p89IlkDLKFhbc9dVDF9DQwMCe8XTbW8JTA2ZkBTHmgBkYu82ORLtHZs0NMuG+DjzEeA== Received: from YQBPR0101MB9742.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:c01:81::14) by YTBPR01MB2543.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:b01:21::13) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5293.13; Mon, 30 May 2022 04:31:10 +0000 Received: from YQBPR0101MB9742.CANPRD01.PROD.OUTLOOK.COM ([fe80::b921:251e:4a0b:54fc]) by YQBPR0101MB9742.CANPRD01.PROD.OUTLOOK.COM ([fe80::b921:251e:4a0b:54fc%6]) with mapi id 15.20.5293.019; Mon, 30 May 2022 04:31:10 +0000 From: Rick Macklem To: Kurt Jaeger CC: "freebsd-fs@freebsd.org" Subject: Re: FreeBSD 12.3/13.1 NFS client hang Thread-Topic: FreeBSD 12.3/13.1 NFS client hang Thread-Index: AQHYcgY+3LPS/CtmUk+ZVTnbRrY6qK0zLx2fgADCMYCAAHUPJ4AAZZKmgAF0ZoCAAJTuzg== Date: Mon, 30 May 2022 04:31:10 +0000 Message-ID: References: In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: suggested_attachment_session_id: f0611781-1866-a83d-c66b-91a66c0b5f95 x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 26043dcc-5334-45af-ef28-08da41f53907 x-ms-traffictypediagnostic: YTBPR01MB2543:EE_ x-microsoft-antispam-prvs: x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: TIvkIYocSJ8sVFi+63sNbJQJtiNIVeH8EB+9BuPCuIzOOoeOu73fOBq6FnI8UNzRTNuCNBkoUuhL6hSrpCq2OP8zaMuQ6+9V952YlbHurqE8cZi2NXQdhxnI94BfolznNf0iRb+cAFVCbSTpz6fTSApt3fT14HqH3E7m3WNiQRFQTLAs87OFDfwWMYp4yJ4edy2zB8nAmBlzDekbdS9hiK3pbYwTqG2hNRSLwJayenth3y+nPwlrffgXx7NdK/Mbqj33+3KiOfYuBK/YhWLneKJ0DTe4XKEiDQzLGZ0f/TK43L91w4VSTy7y/kTuuuXGadhaK/2YaCr5mIfDVW6YGUjv8nbWmlxA6nzSNlH9hnEjIChLiGVttY2cGYkrhauHcqdWIh5GCJB9M125vu7qFuhLNwa0JHuSPQDT3X8A1mT8a0wUnepC9gPUKocgB1nvkFmXi4fDPEPQnnHzYeiLnAkUQJx1DebyJPJjQTc8jcuvoNM0s0f5WiNuXCy7i0e62L1B9r19SwclndHqL7Lmfh15AGYz/KCCLuVPiwl3CduzzHf5XnK9YqHoE2DCk/VkkQotmBo9aoN+q5YWuCqVvYNaAoh+N78Yrx/nd/IveINLXs19ykIzuvRjhUjBE9OuMePW2RnCoQjGYYrlWRhhhPVRoFwla4ZGPjC8o/NB8B2s4pcDIdb4axWpwGBQtzSI6Y6TI8VCfqgY90i/XL0J8g== x-forefront-antispam-report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:YQBPR0101MB9742.CANPRD01.PROD.OUTLOOK.COM;PTR:;CAT:NONE;SFS:(13230001)(4636009)(366004)(186003)(5660300002)(508600001)(55016003)(52536014)(86362001)(83380400001)(66446008)(76116006)(4326008)(66946007)(38100700002)(8676002)(6506007)(9686003)(91956017)(786003)(6916009)(7696005)(66476007)(64756008)(316002)(8936002)(33656002)(2906002)(450100002)(71200400001)(122000001)(38070700005)(66556008);DIR:OUT;SFP:1101; x-ms-exchange-antispam-messagedata-chunkcount: 2 x-ms-exchange-antispam-messagedata-0: =?iso-8859-1?Q?LyWxxd0Q1XoWlIna9QZijWuprDIvFfwkNQBow7h4M4RUc4CiBj9j25XyGy?= =?iso-8859-1?Q?32iyszOiIS2LSv/sCwt3dkst7CuMXlrKsvL63QEPk/+4BHgkjhebkYjfYN?= =?iso-8859-1?Q?BLx3Nt0KSalF7XCIzgVllKTSWI4YDyrT2F7k/v7xz0ou6Mf4xEydkRfe7P?= =?iso-8859-1?Q?CbNkVVOVWJ4TS0VZvXjFwD4pGE2egl3lxg7pqXjwwtOkOD4yj+n3WnU0Rp?= =?iso-8859-1?Q?chir++AWgD3E8Qt85KjloPjwKN/ybYrwXkRHEbl63Ze09LBRXXgz85OKpf?= =?iso-8859-1?Q?nS87ZkPF8mhth/n5w0FVMR/39R/QjdEEE5m+R0XYm9uEpYSWZLnauCKAhn?= =?iso-8859-1?Q?ckaAqB8bVWN2hA8uuZlIoelHe9QbUBBMCt8R+lSeIbNR2W+JbfSSSTQLuc?= =?iso-8859-1?Q?haSL7oznYoJ5kQLCdGUvuEEVpsmko8gKsmP9OaihOu574l5pxr4C9Q9ze5?= =?iso-8859-1?Q?p5x+oP4pe+PXRSYdBCv5ydDdnwzP6XKANyA/tQiR/oL7JWcKqyMd31g/oN?= =?iso-8859-1?Q?NdGaGqa2+etSIFKn7k1jtgnwUxBsUFq6JQQjtIKk8/O+baHG0zZGMQcRyZ?= =?iso-8859-1?Q?F/1+Ml795wRq4lmStCXeTfaa3U03b94w9kB66gw4V2OS3nHEhaXE5scMZw?= =?iso-8859-1?Q?gGWs+kzZ0J4BCR8SKSDaNA2S2LNe4JfoBpeK+uB1UI+SVWmsUgT+Efib5+?= =?iso-8859-1?Q?IRpjvuwnVF8R6dsLDP2EYQfExQu2ThWt6njupAN3OLXpw3Vf1kQAQC8rb+?= =?iso-8859-1?Q?jz+IxlxeJE6EWUQtxiut05vbgcSihjbmJt0XBBN+r40NuVvhbWDs8vSQws?= =?iso-8859-1?Q?79W7R1AdL0xIyz/S1tvHoKBcJBTinxA/E1/wCozG4FD3Gu5TgYgZINLfh7?= =?iso-8859-1?Q?CeT5/ANlDHazz4SM6SRQ1Ee8D8TjXRq18hGUySrzSbSOklIXw78dmnIxCp?= =?iso-8859-1?Q?RxQwtnEYVmd0XuWXXLXhr826svPKaNp6LHYahW8jOvK3adZQp7OBzfbJop?= =?iso-8859-1?Q?kgqO0LNVvxIpxljHgZQ1rYn3kHIwfojtvgS/OlWISTvj1fJa8wd69cD3m1?= =?iso-8859-1?Q?JNf1mgOtSqz3sUqb9Ryg7RzS2WCA6uDuG4Kuizj2EvWjDU/ZRL0CiUuDaT?= =?iso-8859-1?Q?GQybKNrsAhxwLziMgbpuLKNjNhwhIRt4zLvjPdiDRcexp2v2OvECcSrUqP?= =?iso-8859-1?Q?kaNg9S/jGKK5qyOwOE8llFa3+GtgUFp2w68afMmOnZfzTJmmhu3p1dyQtE?= =?iso-8859-1?Q?6SP/1Cs8cEZUEJRppYfwJXMCzZZIXf5MW4BGc5QcOSsKBcHNBCqxWsiEZP?= =?iso-8859-1?Q?3kLL2vPfIyQPuKVG+iIN8+u0E4DbQKz5Zgq4LT/+KFNUDUbKDyppz1LZri?= =?iso-8859-1?Q?VkQVXxlHUkkp51EPzG06hJv+OzYtop9c/2E+MEHIKNVG3dasyI2qwHzROl?= =?iso-8859-1?Q?XrmtIXSISGwGhUnf9t0BymIlMuf5gCeTZSqNfIet7l9FGwqkfAGtyQ2fOy?= =?iso-8859-1?Q?wCjOpHTFxhYgjQdMQ9whHR9mcTezGJls8FXklNndSB20XicJNDOzffNiQM?= =?iso-8859-1?Q?RCZt4uMzDdqOhl4azv31bidxzl68FxbuuqeRTPMZ+NAhA5PfTbVdjeZMx+?= =?iso-8859-1?Q?HqjJX02VMHHbOC7jIzzyn2NgJ9mGgE/+42VFlbUeCtaeojcC0kQ/aCUAag?= =?iso-8859-1?Q?WOeenR5h3w4MnxfAnrc69bJAhYF4gGvcAn1tcR+91HYXmEIDl/uZgyHTGq?= =?iso-8859-1?Q?6T7RwuMYc7HXhJP+MK+4jPyf3Dgjvh6pvTfQtBepr8A/60e2HnOG0NzNih?= =?iso-8859-1?Q?q2TZ9zRcQ9aEiG8JXAtHk650FyfCh6jNdr7XaPXvu8HWGF1zMtP6fPiAQ2?= =?iso-8859-1?Q?vl?= x-ms-exchange-antispam-messagedata-1: vK7B+OeUBUqupg== Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 X-OriginatorOrg: uoguelph.ca X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: YQBPR0101MB9742.CANPRD01.PROD.OUTLOOK.COM X-MS-Exchange-CrossTenant-Network-Message-Id: 26043dcc-5334-45af-ef28-08da41f53907 X-MS-Exchange-CrossTenant-originalarrivaltime: 30 May 2022 04:31:10.5492 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: be62a12b-2cad-49a1-a5fa-85f4f3156a7d X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: SQl0CH3hLrmy/lDQjzduSJGDW3m59JqXIjRjaWgXLLJMKZ/8VCIo/xZ6VRPCqdRJDFfIU++TowuvhGFU6rq2sA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: YTBPR01MB2543 X-Rspamd-Queue-Id: 4LBMtG1k26z3Km9 X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=uoguelph.ca header.s=selector2 header.b=Vt7jyf+c; arc=pass ("microsoft.com:s=arcselector9901:i=1"); dmarc=pass (policy=none) header.from=uoguelph.ca; spf=pass (mx1.freebsd.org: domain of rmacklem@uoguelph.ca designates 40.107.115.86 as permitted sender) smtp.mailfrom=rmacklem@uoguelph.ca X-Spamd-Result: default: False [-6.00 / 15.00]; TO_DN_EQ_ADDR_SOME(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; R_DKIM_ALLOW(-0.20)[uoguelph.ca:s=selector2]; FREEFALL_USER(0.00)[rmacklem]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:40.107.0.0/16]; MIME_GOOD(-0.10)[text/plain]; NEURAL_HAM_LONG(-1.00)[-1.000]; DWL_DNSWL_LOW(-1.00)[uoguelph.ca:dkim]; RCVD_COUNT_THREE(0.00)[3]; DKIM_TRACE(0.00)[uoguelph.ca:+]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[40.107.115.86:from]; NEURAL_HAM_SHORT(-1.00)[-1.000]; DMARC_POLICY_ALLOW(-0.50)[uoguelph.ca,none]; MLMMJ_DEST(0.00)[freebsd-fs]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:8075, ipnet:40.104.0.0/14, country:US]; ARC_ALLOW(-1.00)[microsoft.com:s=arcselector9901:i=1]; RWL_MAILSPIKE_POSSIBLE(0.00)[40.107.115.86:from] X-ThisMailContainsUnwantedMimeParts: N Kurt Jaeger wrote:=0A= [stuff snipped]=0A= >=0A= > Given that simple NFS mounts and editing with vi on nfs mounts should=0A= > be very basic use-cases, it's time to find ways to debug this 8-(=0A= To find out what processes are holding locked NFS vnode...=0A= Run a kernel with the debugger built into it.=0A= options DDB=0A= options KDB=0A= =0A= # sysctl debug.kdb.alt_break_to_debugger=3D1=0A= =0A= Then, when the hang has occurred, go to the console.=0A= Type =0A= You should go into the debugger:=0A= db>=0A= Then type "show alllocks".=0A= You are looking for the pids of the processes that hold NFS vnode locks.=0A= db> continue=0A= - Allows the system to run again.=0A= =0A= Then look through the output of "ps axHl" for the processes found above=0A= and see what they are sleeping on.=0A= (My hunch is that at least one of them is sleeping on "vmopar", but it=0A= is just a hunch.)=0A= =0A= These processes are your problem. Figure out how to stop them from=0A= sleeping on something else while holding an NFS vnode lock and you've=0A= fixed your problem. (Do not assume it is an NFS problem. I'm pretty sure=0A= it is not caused by NFS.)=0A= =0A= rick=0A= =0A= > This also implies that switching from soft->hard won't fix the problem.= =0A= =0A= Thanks for the encouragement 8-}=0A= =0A= > In summary, I don't think your hang is anything like Andreas's, rick=0A= =0A= I'll report as soon as I have a another hang.=0A= =0A= --=0A= pi@FreeBSD.org +49 171 3101372 Now what ?=0A= =0A=