From nobody Fri May 27 20:59:57 2022 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id DEC421B388A1 for ; Fri, 27 May 2022 21:00:05 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from CAN01-YT3-obe.outbound.protection.outlook.com (mail-yt3can01on2042.outbound.protection.outlook.com [40.107.115.42]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "DigiCert Cloud Services CA-1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4L8xyX4895z3LVc for ; Fri, 27 May 2022 21:00:04 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=JRJuTqMtqxoF/3SITTzfXxMAXuePvtj3UfivMARy5fsUboSe4dmDBfFmcHsctzSgufJ3zTLsucrswb3EPJ5CBKQWPxOWv5qn+BkE46c7tbUUoLOiTIeH/A9byA7xYliJivMx57VrgZ0BUBZakVcnkFKvILYMM1TAjfw+7MgGEFpAxiGmyz9vkDltDTDQHjviUEPpDqVc0w8tlgWZpXe+F4/5wM+pEZOvL5kBnq1X7KyD66J4Lr0cmPEa/w4OTbdPuSTv3+pTiwaxgfMlByeWRAfrFWUFFtgV84iS6TxPrSuXh+hgXpZROoiMdWsNSaRWsAqGbE7KMStXIqDXP9tBMw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=Z0rFnLWASoyZ0L8wL5HORsiWmTDCWZNNoMIJySvys/4=; b=RAEJxEGcPuPnSIWscZI8aLNNtk7MHzfDdK75fg7NjTbrEAl6TSOL9G1pycdVNbqVUQow1C7rBrLkwPjUHCyVKK2HU78N6cd/KZTBC7rtBO1JAmf9B5SRlXvpCmsGbqZZCipSuPWBzkgn8ZlNEYaj4HLC2kXyH1TEC7NJNeJZy8w7Bt7cVSG76ZFIcfSA9uWp8ID/dhdfAeRxhsHFdqQE34ppCM4OV+2rxKujuAHTzlMOYJutAfX+zu2rOqgxP0tyna1TFtr3VboHUjXf/boc2k96tkYX8+KrgBQDF9/AO8YHFmx6o2v0NtMyUm2ypaghnVQu6B5nYXnoDAMzp8ALZA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=uoguelph.ca; dmarc=pass action=none header.from=uoguelph.ca; dkim=pass header.d=uoguelph.ca; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=uoguelph.ca; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Z0rFnLWASoyZ0L8wL5HORsiWmTDCWZNNoMIJySvys/4=; b=J2K2wdOEOMzmHrbuJX0p6cFUk9z8J1qmes1QUwLG6kqu272MTgKKCI2401ISz3mbzAT8ovZy2VnY9XFzIoZE4Q7tskIZxAMJP887mUwwFfdYsE1olurXLyAx+G3ZbbR4fXWirkE0uDPgMTG8iPuDVs00bIrFvukYCWmhwuGCpu7HkRD989HN4WAcggBgFiwY7nMYfo9EwBJWofm8mc2qMTttc8g1rcd62eY1yR7jGMDQRZH0U/PykeZFx/HeoGKjbHF6ncZHrlOM6CrplMOkdzoi3ITWmvm6V1YOdmfE8i6M7RnZG0Bt3uxwHjwdPV+1ZApN4EnxQe5dt512OdrvnA== Received: from YQBPR0101MB9742.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:c01:81::14) by YT1PR01MB4741.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:b01:43::22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5293.13; Fri, 27 May 2022 20:59:57 +0000 Received: from YQBPR0101MB9742.CANPRD01.PROD.OUTLOOK.COM ([fe80::b921:251e:4a0b:54fc]) by YQBPR0101MB9742.CANPRD01.PROD.OUTLOOK.COM ([fe80::b921:251e:4a0b:54fc%6]) with mapi id 15.20.5293.016; Fri, 27 May 2022 20:59:57 +0000 From: Rick Macklem To: Andreas Kempe , "freebsd-fs@freebsd.org" Subject: Re: FreeBSD 12.3/13.1 NFS client hang Thread-Topic: FreeBSD 12.3/13.1 NFS client hang Thread-Index: AQHYcgY+3LPS/CtmUk+ZVTnbRrY6qK0zLx2f Date: Fri, 27 May 2022 20:59:57 +0000 Message-ID: References: In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: suggested_attachment_session_id: e57124a1-e28f-b903-5d1d-c5bc967679e9 x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: a5942a5f-9f8d-4e09-c287-08da4023db9c x-ms-traffictypediagnostic: YT1PR01MB4741:EE_ x-microsoft-antispam-prvs: x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: mNESCPvikzNuzpGgn0OqXQ2mMRH0V2UQO9w8LoFBZm/clJtBwnrLA1uD9i5l/ZP9BI+V5nZW83ErHs7T/qn0idFCYqlceTIm/jc90z5kVGr9cqSDU1OGH+JVQvw5OTP1s14nNUDTLg4yrDlHOquH9pF7T1jK2Mzq0Ag91AvzRXAbW7qgf8goGxZ8ljunH48VF1DGQsjetxV43l7jAR2ZwQNa7rD4tmHPDZBGx3LM1zVIdiF7zOV9D1LZxTYcHJUDizlTOT8hcm3yv0jfdArJ0Eu51/WnF2pNopQuCUqkQSmW0hYojuqd2LaDAczwXD7DqVt67/3Bynb0MEPeVDo5B2+1bqqFdL5r00VeMV5t8w3BMEA3qxO8H1jWC9Df8byWKqtV0fspka/ubtNCwsk8l5vDEB68TQZk27te8jwQcLJB3WtCssghyvt7nLhDu033tzfOg6eKXJne7fcBlNUmFW/PbeWjLcwTXcox0eUZsxRsydHmm8g8C0Jsv69ZIIvAPWZVKlxSI5bEFmmHNhZtjVrw03oZb7tdKSFwEAN+yN18CEJv+loggaEp7HUXYv+E4rtik68oSFFdL7Y0BwxUEc3E7bRwXshZoSyPMHVSlMdRmLSEWYvWBlg1ls0JbEM8REi7SWQbsRsLBdYiG5uWahpPuz5j268446KILlKybP2dEzG/rbjhwhu7lNN88pavBIsBvU/OCOD0YvjyjXGW3Q== x-forefront-antispam-report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:YQBPR0101MB9742.CANPRD01.PROD.OUTLOOK.COM;PTR:;CAT:NONE;SFS:(13230001)(4636009)(366004)(52536014)(2906002)(38100700002)(8936002)(33656002)(9686003)(91956017)(66946007)(76116006)(66446008)(64756008)(8676002)(5660300002)(66476007)(55016003)(66556008)(296002)(316002)(86362001)(83380400001)(110136005)(71200400001)(508600001)(186003)(38070700005)(6506007)(7696005)(122000001)(786003);DIR:OUT;SFP:1101; x-ms-exchange-antispam-messagedata-chunkcount: 2 x-ms-exchange-antispam-messagedata-0: =?iso-8859-1?Q?knO0Kb1cZPSZ+SLE3lLSAD2MYEtLzPggcI/PN0xGL+wZlQ1DYzwPYn1rE9?= =?iso-8859-1?Q?fFmuW8pWhVhyxYgbWWUKHnZ+LQCfCtord45WH8yazaFGPsKAY2c/QYSs5k?= =?iso-8859-1?Q?XtEmg57RrKA0BotQ9w3H4pmrIExdzxh4RmhQbzKWZsnCIgMxHqMCoG9Or2?= =?iso-8859-1?Q?/fpFo28wViIwoB+fqfGawH6wAHjcJO5aaR6RyXM5KR6qSMWa3hJEmt6SK8?= =?iso-8859-1?Q?ifA/54A+o8awanTMPc1v8B8cYwY8+VVrKgYZTlwCUSEzHXP4f+aYDcBn27?= =?iso-8859-1?Q?f0jJOYRLBAD1QhF/HF/KArVOelaHe6KU6eTKUhKTAgo/bdlOgFg3cVTFvj?= =?iso-8859-1?Q?BpPdMOw/d0lMwq1+awoUkjm/rqPltV+gri5gqtgHfGdVebNR5WhqYaHz7U?= =?iso-8859-1?Q?307xCHX1AfXj2584QXEkmtTFQXKVbz3C2RR0Q+m2X0odAyWbr9UBbmsb+i?= =?iso-8859-1?Q?9qBD7tXn+GbNxWItxhAoZ5HfyHWzzxqEEdKe42BKMmsvvLyR84o7UtIy8o?= =?iso-8859-1?Q?8foxOdJ0aSdOB1Egd2bXa5VW12iW0WtF230lmsahUlIES71lp1JKx69Ouo?= =?iso-8859-1?Q?nkfkN88a9JlJAC2cXJtQ8jjgeKd433tm3oLi0O7iH1uUaId+MCISJdVtjA?= =?iso-8859-1?Q?jLuliSJMKe1RA24NZQWNMnIFck34ZvnZ9+TNkc4NmIjj9HkbuM8qMBNuUI?= =?iso-8859-1?Q?3VD05fNmrkv6VT55B9nFpR5UymsCqX3nu3Twnb/0psqe9+uM+cLWkpe2Hv?= =?iso-8859-1?Q?bDAcZmHQpYJcq51AIZhz+23hwhax4JCtgId9epjwb+GZDXy+pW9ZWAP6Ak?= =?iso-8859-1?Q?esqwilACFp4hslDFjcwSypHGG+Op+msGy+trZXxLOeuZnD+IawoZ1u+enI?= =?iso-8859-1?Q?R3BLv76S7hVqunyFkWq8br/Bfeu/RGbFIDcZalYLN1wmoFeDjjK1GCxcV0?= =?iso-8859-1?Q?XcOPzA7mCmszF4djNrb6TXP7yg/BwnbHBMJ0NLS02DXHkJ/+po1h7Tg1Et?= =?iso-8859-1?Q?Qyx4WD/OdeXEkmVwucGluP9mXGffuQHRitM6THikOPq67nEnOD61UGU0z6?= =?iso-8859-1?Q?d+DS72TtFakn2hjeGhCr+VvRIQTBKaFzd7m7JK+GDDrX0TNHpVptqh+ZZ+?= =?iso-8859-1?Q?oZ05r6hKWp8WgqC9OOWZYtTSAX6ZTd4QGF6OTqJjTTWxTkXvGULLzbbFi9?= =?iso-8859-1?Q?QZw+nkeu0M31PUKgv89PcxvzZkrRs0RLbAI6lp4LO3EydQBN+e8Fix05fo?= =?iso-8859-1?Q?ZFfn5k/iWqOpVzzAeWPnGSBFClpymOLnK3AG5OBxRWYU25iYQrMxyzr4Qs?= =?iso-8859-1?Q?hi1BgsKxXREgTT62pwElKM/yFrGCRCQUOMNgF4NvInusyVbsXYQy0DNkjK?= =?iso-8859-1?Q?vUtF4kURLXaG7hY7WjWRJCqmpxiMVIFtlODYUUHQjsw3exZ8pc4fV4biQT?= =?iso-8859-1?Q?e8NTZKUNL+GO0tun549vLP7p567xn3P88u+eo3cewNT2Ac3RBs7KzGr58j?= =?iso-8859-1?Q?4lZI9xxK0k88fsvN3I1Ab6wfo+wLCZutZEG1PIn+acGr4VW27VyVnGburr?= =?iso-8859-1?Q?L+S1tynD1m6+gTCMh0+I6q1CPm29M3E48aTz68b5+iksi1UHI5hWlR+Kcl?= =?iso-8859-1?Q?+vzTgebir7IoVSrqVtCdfSN3ZbDU/Q1I1KQdQ4Yb9yOm/0J/9qj4+BWTcg?= =?iso-8859-1?Q?BifAOY8p221+TmmWxSsDWnsXesnZa5aw8o1CXPuF0ZarZThM3k8mZNouyR?= =?iso-8859-1?Q?97n+53U9Aiw5jBm2n4kkRbyXxRGeumOVY0bMFpE/Tr8ZHfBu7CTNKlOtks?= =?iso-8859-1?Q?xEtKJWP/dyZD54pOdw+lkBOGiJmsu62fn/rMQIFqwrY0KjoeHOBtHNfWeA?= =?iso-8859-1?Q?Do?= x-ms-exchange-antispam-messagedata-1: uj8U22P/Udy/CA== Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 X-OriginatorOrg: uoguelph.ca X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: YQBPR0101MB9742.CANPRD01.PROD.OUTLOOK.COM X-MS-Exchange-CrossTenant-Network-Message-Id: a5942a5f-9f8d-4e09-c287-08da4023db9c X-MS-Exchange-CrossTenant-originalarrivaltime: 27 May 2022 20:59:57.8707 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: be62a12b-2cad-49a1-a5fa-85f4f3156a7d X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: wPbi+dFMZouoFDfd2VLUj6BfI9zzMIIr1OrThNSs7DFwCyHOXt0AjOn1rhca9CymzI4Ym+qg+SD1A9UvF6nYUA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: YT1PR01MB4741 X-Rspamd-Queue-Id: 4L8xyX4895z3LVc X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=uoguelph.ca header.s=selector2 header.b=J2K2wdOE; arc=pass ("microsoft.com:s=arcselector9901:i=1"); dmarc=pass (policy=none) header.from=uoguelph.ca; spf=pass (mx1.freebsd.org: domain of rmacklem@uoguelph.ca designates 40.107.115.42 as permitted sender) smtp.mailfrom=rmacklem@uoguelph.ca X-Spamd-Result: default: False [-5.03 / 15.00]; TO_DN_EQ_ADDR_SOME(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; R_DKIM_ALLOW(-0.20)[uoguelph.ca:s=selector2]; FREEFALL_USER(0.00)[rmacklem]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:40.107.0.0/16]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; DWL_DNSWL_LOW(-1.00)[uoguelph.ca:dkim]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[uoguelph.ca:+]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[40.107.115.42:from]; NEURAL_HAM_SHORT(-0.03)[-0.026]; DMARC_POLICY_ALLOW(-0.50)[uoguelph.ca,none]; MLMMJ_DEST(0.00)[freebsd-fs]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:8075, ipnet:40.104.0.0/14, country:US]; ARC_ALLOW(-1.00)[microsoft.com:s=arcselector9901:i=1]; RWL_MAILSPIKE_POSSIBLE(0.00)[40.107.115.42:from] X-ThisMailContainsUnwantedMimeParts: N Andreas Kempe wrote:=0A= > Hello everyone!=0A= >=0A= > I'm having issues with the NFS clients on FreeBSD 12.3 and 13.1=0A= > systems hanging when using a CentOS 7 server.=0A= First, make sure you are using hard mounts. "soft" or "intr" mounts won't= =0A= work and will mess up the session sooner or later. (A messed up session cou= ld=0A= result in no free slots on the session and that will wedge threads in=0A= nfsv4_sequencelookup() as you describe.=0A= (This is briefly described in the BUGS section of "man mount_nfs".)=0A= =0A= Do a:=0A= # nfsstat -m=0A= on the clients and look for "hard".=0A= =0A= Next, is there anything logged on the console for the 13.1 client(s)?=0A= (13.1 has some diagnostics for things like a server replying with the=0A= wrong session slot#.)=0A= =0A= Also, maybe I'm old fashioned, but I find "ps axHl" useful, since it shows= =0A= where all the processes are sleeping.=0A= And "procstat -kk" covers all of the locks.=0A= =0A= > Below are procstat kstack $PID invocations showing where the processes=0A= > have hung. In the nfsv4_sequencelookup it seems hung waiting for=0A= > nfsess_slots to have an available slot. In the second nfs_lock case,=0A= > it seems the processes are stuck waiting on vnode locks.=0A= > =0A= > These issues seem to appear seemingly at random, but also if=0A= > operations that open a lot of files or create a lot of file locks are=0A= > used. An example that can often provoke a hang is performing a=0A= > recursive grep through a large file hierarchy like the FreeBSD=0A= > codebase.=0A= >=0A= > The NFS code is large and complicated so any advice is appriciated!=0A= Yea. I'm the author and I don't know exactly what it all does;-)\=0A= =0A= > Cordially,=0A= > Andreas Kempe=0A= >=0A= > Hang provoked when calling "grep -R SOME_STRING ." in the FreeBSD code ba= se.=0A= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=0A= > =0A= > PID TID COMM TDNAME KSTACK=0A= > 35585 101045 python3.8 - mi_switch sleepq_tim= edwait _sleep nfsv4_sequencelookup nfsv4_setsequence nfscl_reqstart nfsrpc_= lock nfsrpc_advlock nfs_advlock VOP_ADVLOCK_APV kern_fcntl kern_fcntl_freeb= sd amd64_syscall fast_syscall_common=0A= As you noted, this is waiting for a session slot to become available. Norma= l, so long as other RPCs=0A= are "in progress" that will release slots when replies are received. (If yo= ur mount was not hard, sooner or=0A= later, the client will give up waiting for the reply and the session slot w= ill not be released. Once all slots=0A= are "not released" you are hung.=0A= =0A= > 35585 101045 python3.8 - mi_switch sleepq_tim= edwait _sleep nfsv4_sequencelookup nfsv4_setsequence nfscl_reqstart nfsrpc_= lock nfsrpc_advlock nfs_advlock VOP_ADVLOCK_APV kern_fcntl kern_fcntl_freeb= sd amd64_syscall fast_syscall_common=0A= Same as above.=0A= =0A= > 44046 101189 vim - mi_switch sleepq_tim= edwait _sleep nfsv4_sequencelookup nfsv4_setsequence nfscl_reqstart nfsrpc_= accessrpc nfs34_access_otw nfs_access VOP_ACCESS_APV vn_dir_check_exec nfs_= lookup VOP_LOOKUP_APV lookup namei kern_statat sys_fstatat amd64_syscall=0A= Again.=0A= > 44046 101764 vim - mi_switch sleepq_cat= ch_signals sleepq_timedwait_sig _cv_timedwait_sig_sbt seltdwait kern_select= sys_select amd64_syscall fast_syscall_common=0A= > 44046 101853 vim - mi_switch sleepq_cat= ch_signals sleepq_wait_sig _sleep umtxq_sleep __umtx_op_sem2_wait sys__umtx= _op amd64_syscall fast_syscall_common=0A= > 44046 102164 vim - mi_switch sleepq_cat= ch_signals sleepq_wait_sig _sleep umtxq_sleep __umtx_op_sem2_wait sys__umtx= _op amd64_syscall fast_syscall_common=0A= > 44046 102165 vim - mi_switch sleepq_cat= ch_signals sleepq_wait_sig _sleep umtxq_sleep __umtx_op_sem2_wait sys__umtx= _op amd64_syscall fast_syscall_common=0A= > 44046 102457 vim - mi_switch sleepq_cat= ch_signals sleepq_wait_sig _sleep umtxq_sleep __umtx_op_sem2_wait sys__umtx= _op amd64_syscall fast_syscall_common=0A= > 44046 102472 vim - mi_switch sleepq_cat= ch_signals sleepq_wait_sig _sleep umtxq_sleep __umtx_op_sem2_wait sys__umtx= _op amd64_syscall fast_syscall_common=0A= I know nothing about umtx, so can't help here.=0A= =0A= > 44172 101824 tmux - mi_switch sleepq_tim= edwait _sleep nfsv4_sequencelookup nfsv4_setsequence nfscl_reqstart nfsrpc_= accessrpc nfs34_access_otw nfs_access VOP_ACCESS_APV vn_dir_check_exec nfs_= lookup VOP_LOOKUP_APV lookup namei kern_chdir amd64_syscall fast_syscall_co= mmon=0A= Another one waiting for a session slot.=0A= =0A= > Hang provoked randomly when trying to save an image in kolourpaint.=0A= > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=0A= >=0A= > PID TID COMM TDNAME KSTACK=0A= > 58062 159450 kolourpaint - mi_switch sleeplk lo= ckmgr_slock_hard nfs_lock vop_sigdefer _vn_lock vfs_cache_root vfs_root_sig= defer lookup namei kern_statat sys_fstatat amd64_syscall fast_syscall_commo= n=0A= Yep, waiting for a vnode lock, I think.=0A= =0A= > 58062 176390 kolourpaint - mi_switch sleepq_cat= ch_signals sleepq_wait_sig _cv_wait_sig seltdwait kern_poll sys_poll amd64_= syscall fast_syscall_common=0A= > 58062 176678 kolourpaint - mi_switch sleepq_cat= ch_signals sleepq_wait_sig _sleep kqueue_kevent kern_kevent_fp kern_kevent_= generic sys_kevent amd64_syscall fast_syscall_common=0A= >=0A= > PID TID COMM TDNAME KSTACK=0A= > 34291 101005 fish - mi_switch sleepq_wai= t sleeplk lockmgr_slock_hard VOP_LOCK1_APV nfs_lock VOP_LOCK1_APV _vn_lock = vget vfs_hash_get ncl_nget nfs_root lookup namei vn_open_cred kern_openat a= md64_syscall fast_syscall_common=0A= Also waiting for a vnode.=0A= =0A= > 34291 102492 fish - mi_switch sleepq_wai= t sleeplk lockmgr_slock_hard VOP_LOCK1_APV nfs_lock VOP_LOCK1_APV _vn_lock = vget vfs_hash_get ncl_nget nfs_root lookup namei kern_accessat amd64_syscal= l fast_syscall_common=0A= Again.=0A= > 34291 102493 fish - mi_switch sleepq_wai= t sleeplk lockmgr_slock_hard VOP_LOCK1_APV nfs_lock VOP_LOCK1_APV _vn_lock = vget vfs_hash_get ncl_nget nfs_root lookup namei kern_accessat amd64_syscal= l fast_syscall_common=0A= And again.=0A= =0A= > PID TID COMM TDNAME KSTACK=0A= > 204 100923 autounmountd - mi_switch sleepq_wait= sleeplk lockmgr_xlock_hard VOP_LOCK1_APV nfs_lock VOP_LOCK1_APV _vn_lock v= get vfs_hash_get ncl_nget nfs_statfs __vfs_statfs kern_getfsstat sys_getfss= tat amd64_syscall fast_syscall_common=0A= And again.=0A= =0A= Not very useful unless you have all the processes and their locks to try an= d figure out what is holding=0A= the vnode locks.=0A= =0A= rick=0A= =0A= =0A=