From nobody Sun Aug 21 22:19:56 2022 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4M9qg36jzBz4Ywp9 for ; Sun, 21 Aug 2022 22:19:59 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from CAN01-YQB-obe.outbound.protection.outlook.com (mail-yqbcan01on2085.outbound.protection.outlook.com [40.107.116.85]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "DigiCert Cloud Services CA-1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4M9qg22vjkz3p31 for ; Sun, 21 Aug 2022 22:19:58 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=E2pyPqE+x6V+cyVy3uwUTizOvWuClwM3aXVxu1h9P4ZfH8utbbrQGoiQ6dF75z8lFmoBx4j9FSO2mVl8b0SYHF/AUKdGl2J1w8nSVrGUiGMnW163eabiYp0+NCmCRaLPVoFXjUvnsBhjs1I4fdZf3JlWdykvxO06Y0wIS8QDE5hpnB/Q/ojfrQjCpo4i7huc+18ghB5CEOxNEHPDC79oTT2Opl56/xsDpTvbEGNl3Sx6v0z7HnnppcoN89cRcvcagGUuNV9yqeYu3pmGYuzUtI3sDpZX2baq6d6paCE0MzQ931UDD128UtqDbLIQNnifxpnJaAVoYyzs+e8A5vIYiQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=xPGvHvSuoeGmG2VuHTk8g4Cv38memPSyw3GscAnrk/Q=; b=fVds1Af/e+kk7p1TbPeHA2TN6a0Uld7bjUCONAFNuuhWd0GgNeezjftzJ1IF5s68B58Kb6DhU3GwCDYwWb8/RmVSHN5l7fq7dplRC4CUhzpMQVSFVh8G4RtkDNL3P98lQgg0w2xRN4xjtXKyh5nkCUMjkpDb8j6Z3jxvA7YO0HWUTTWq+qQL2f9i19h6GeAEY2QRAHZztSevkNTHmkTVTgkd2onSknL5NWmnTL68czwsj2BmgLd9iabXvF/WOm8MwknsnqqlVfCmrmi9JSDSTV29VcxIxqqE+4AVQbSGunP5HLRsLe2nKNt3VkC3pCV3kGNPfKqjOAw1O1TOfF2CyQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=uoguelph.ca; dmarc=pass action=none header.from=uoguelph.ca; dkim=pass header.d=uoguelph.ca; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=uoguelph.ca; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=xPGvHvSuoeGmG2VuHTk8g4Cv38memPSyw3GscAnrk/Q=; b=mtSDOZzTSp9IW6GG06ZObdZfQW9UcUZ/k6lN1oTjFh3AKjb6Ov+PDhHH7ib2QesX7dLSik3uFML9Crpny9uopGZW5gs9o6XRAmlkAIB2pyQHK/Md/v7/8AlNxVPag/oE3zmm5NZT3wDV9dHI3v683DUBZfuq9v3IvRYk9PpVW0k30jrybm1CQOuC7fujZ22nXtsG1cuSSUsH+DKIFLQJJuUw2OuYbwPUs1wb7bDrYi1AePqqBNQspwiNWUtepFahj+JlfRmwiowUe5iu6hKsQ7v1nZeQ5wPgzf9CnNH6HT1qduwPaTZzYJwItw1wZpWxJiit3iIy6D7iCBLl5ob4Jg== Received: from YT4PR01MB9736.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:b01:e6::17) by YT3PR01MB5041.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:b01:48::23) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5546.16; Sun, 21 Aug 2022 22:19:56 +0000 Received: from YT4PR01MB9736.CANPRD01.PROD.OUTLOOK.COM ([fe80::4c04:eddf:6168:b3ec]) by YT4PR01MB9736.CANPRD01.PROD.OUTLOOK.COM ([fe80::4c04:eddf:6168:b3ec%3]) with mapi id 15.20.5546.022; Sun, 21 Aug 2022 22:19:56 +0000 From: Rick Macklem To: Konstantin Belousov CC: FreeBSD Filesystems Subject: Re: SEEK_DATA/SEEK_HOLE with vnode locked Thread-Topic: SEEK_DATA/SEEK_HOLE with vnode locked Thread-Index: AQHYrQY7fWQ5YS0dF02y4OgRBDgrJa2owxGAgA/DcfWAAQ8LgIAAYqWL Date: Sun, 21 Aug 2022 22:19:56 +0000 Message-ID: References: In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: msip_labels: x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 63dc8940-3c54-4c94-88bb-08da83c34732 x-ms-traffictypediagnostic: YT3PR01MB5041:EE_ x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: QAuVSUfNYiUkGcb4hM+sFVToUXcf/6v9uYtum/XvDXY2r0CKoV8vvdMxQMcmDg1VYTm94vRMxTF0zLcBl9dCBjrQ1fxxmyzBtAnqbVUD69TPRTpMb1CiW1RP+jysKInosQVESBH8SB4p+LPoiR/l+fIgveDooRpui2/qLIa7tlGPVDxIsTI5oosv8oRy1HndUr0GbYNpp+gG/PtYn+PkMcclTp75F9RV4X+oLuZgtL3QtPkgnTXDgDtbDzvsQJEFF3/bsLuMc+Xj8MBk6pgbcrlf6qRXOf8o8kyf8IW6DsziKyNlYLjY3AFrIdSyJNjOjuLyPJx+jgq5iraWxA+WPSXCrZrGyFsDDFWvAnFmacEmtiLO8QaWELcfpEsGGaWvjuoeDkJjbpIiJVhtLsGzhYOELAIhuib8ywFe1uCGOtWGPGDlZESr90Yo3a+0UbOVpw+tS7uR0DBOHXmVH8mvseUYJMxI9iQSqfbDsLNDH/0U2Bb3amm/Mhgl3OGnBL3QrYiF1cZwYLzOpv5HUu60PFa5cudiYhXk8cBXeMfNJhsKL3ColaP2oN3iujm8wKwFrNJe7NJBLxdhTidW2khYIqRErw8dGgU5Ww9Ah5H45syOLSr7eAySbfdpeov5VCf7ELKwHHC2e1S84dH5LNC82ZhQ2C7ODZxfDOgbtlO8bc6SWy4NIAZkjFjXovEVp5s/QsT04CoQKA8rXtI6J+y9izmwdkTf2rx7qx3VnHp0ac3GgLAjzd+zkx5gGZqyW9G5gumN6jbdcwNFGuWTQeCvQg== x-forefront-antispam-report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:YT4PR01MB9736.CANPRD01.PROD.OUTLOOK.COM;PTR:;CAT:NONE;SFS:(13230016)(4636009)(366004)(396003)(346002)(39860400002)(136003)(376002)(66946007)(76116006)(33656002)(122000001)(66476007)(66446008)(64756008)(8676002)(4326008)(66556008)(86362001)(38070700005)(38100700002)(6916009)(91956017)(66574015)(9686003)(6506007)(41300700001)(478600001)(71200400001)(7696005)(186003)(41320700001)(786003)(316002)(55016003)(8936002)(83380400001)(2906002)(52536014)(5660300002);DIR:OUT;SFP:1101; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?iso-8859-1?Q?JM/s5bT9bLpFmmmER+J0/inibFWR7WdUS3uyjBE9WObqPaOPLOQEHLDw4H?= =?iso-8859-1?Q?St7hRd+/MVrbzobYWccuW9oeZBB8KhcxE1mRs0V+YVB4eSm7Oegc16WOEh?= =?iso-8859-1?Q?LXpb9NcYAhQifOseR3lPX6Q3snVkT5/lbhdxbRnnM6ijgmrzRHY7Izy+tM?= =?iso-8859-1?Q?0x6bMgU+WmO4K2vq5r7ldUjEVhCQbNuDoWgtXREtBOWAcAfveW6QLgUL4S?= =?iso-8859-1?Q?lJXGq+nl7wmHAP6qdDkDqgm9BWOr9AE6qvXlzw4deUPppcy0lEYIuteigj?= =?iso-8859-1?Q?y69Bmd305LG7UWf8aLPl7O9YMgAW+UCO1hwuH7W5OVADd5aI/Smpp37elg?= =?iso-8859-1?Q?aZifnL+6B2M6+UMpTIpiUg59jXMNdMg6JbufOCpPj6v4Zk4LRQMKFrOe4b?= =?iso-8859-1?Q?iDReLcC+N3ViWOwoHJIXnn551TR8atVFApjKkBvc4S4gn4JVUHY5H7cdaY?= =?iso-8859-1?Q?M2m9X67rjXcfFZWGxvjETj13aRy/V6GCJZNdewdcSb8+o4/v4CWna6efw5?= =?iso-8859-1?Q?FQHJwnmzo/HTKYM9jwof9bm2oJd+TMO5+8HUs7dHDVhsf+FSXhubzyhMxa?= =?iso-8859-1?Q?h7LUzrSInj4tnhRSg9T3lqr40leQmyCf1oXkbr34pnseGKPCIaRl0m6b7u?= =?iso-8859-1?Q?Mn/mRiM3DTedaK7lg4d85EiSa00B14aIFCCSaocPW0tky8m4QZp6eb+lB8?= =?iso-8859-1?Q?i0u8bM1PNP4oNOl3Oxepr74K/eQHd0ypLJCVKTjpE0sgUGEVrr9vPaHOUw?= =?iso-8859-1?Q?Pe4LmYFDsfNc++4Seg+J4PxZS6nFG8jJ3pKEgbo4oJIBB66jyiufUEuA1i?= =?iso-8859-1?Q?jkU5mHIpZHBE82F9Sh9yMQHk2Qt7Uw+Dceev+1eXyjg+UpnnOBUsRH7b04?= =?iso-8859-1?Q?DapfIhQAnGEe38IauKo0pUlSgp5tokxqRaS0ZBlNnLgoVddKCcT6/gVFuG?= =?iso-8859-1?Q?QiTCZFJiBqdZSe00IQBZmUstStYutihPy+0taPil37eB1sxt4Z7NASa9Mo?= =?iso-8859-1?Q?78dEqXfCsQcidz/AthObVM/B71eyjwhWfUDFrU2mlp4jRfkD8+1rAh5Xmx?= =?iso-8859-1?Q?c1mSQaBSCchKW93ZaD6Vx4e85sWqKz05AUj9BoPwN90OvQYgN7DeR44xKL?= =?iso-8859-1?Q?BwhKCltWavUeXNMglsK3N7BqJRqVj0OmK+LVTUyum8GMTjO/wfpK/lkcNU?= =?iso-8859-1?Q?SLB8Iz+dzr+XchaWRsjbgWxtXHDu01ra8QmESENY6bvc2RxzNAgnzpARwu?= =?iso-8859-1?Q?qV1AvnG6rNKuGIiQu44pAluGEnPZRdxv8SapT5bvv7T1wQmwaREcKsqDGj?= =?iso-8859-1?Q?rWEc6q/aBm95wpA7gZon5zNMBhRxhu1O6aJ4tHIjm0qduAJkXF1wlZ7VGN?= =?iso-8859-1?Q?5He1QN5I2olOevX9ex7F+2LLwqolmlLOiYLOcYGYtXp30W8LHoCbR8fktT?= =?iso-8859-1?Q?56c7o7AkDr1q+7Qr/4skTKkcQoA5OHVABpkA1XFMG7slbnnl4LIeXl7/Kc?= =?iso-8859-1?Q?PMk2ApYkufycGPa0cCB2jscRKg32GBd0pMigGlXLEaHKKj3gGLKRe7vQAn?= =?iso-8859-1?Q?+Cpsxh/rZVD4INs334e0hPq8CNwRJjcWA6IZyd4tZ+GQS1g4vdNR3qqIuD?= =?iso-8859-1?Q?E99sQZTz0Q7vHxXOtzZzovyyeZgHJLIiM1ylrMJTCiDjNVv3uI2u9bJsrc?= =?iso-8859-1?Q?91kUS2fAJXjmSciap0QlGTCd7MSQsb4eXBx4nZ/B?= Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 X-OriginatorOrg: uoguelph.ca X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: YT4PR01MB9736.CANPRD01.PROD.OUTLOOK.COM X-MS-Exchange-CrossTenant-Network-Message-Id: 63dc8940-3c54-4c94-88bb-08da83c34732 X-MS-Exchange-CrossTenant-originalarrivaltime: 21 Aug 2022 22:19:56.2077 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: be62a12b-2cad-49a1-a5fa-85f4f3156a7d X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: fXbfoitVOEnV7IsX/ZdFAg9tL8VdPN/+cngVJ9UDx2TOkltjlciyP0fe8NszJYG5AKVuyK42BTMCcpGZ7N0FEw== X-MS-Exchange-Transport-CrossTenantHeadersStamped: YT3PR01MB5041 X-Rspamd-Queue-Id: 4M9qg22vjkz3p31 X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=uoguelph.ca header.s=selector2 header.b=mtSDOZzT; arc=pass ("microsoft.com:s=arcselector9901:i=1"); dmarc=pass (policy=none) header.from=uoguelph.ca; spf=pass (mx1.freebsd.org: domain of rmacklem@uoguelph.ca designates 40.107.116.85 as permitted sender) smtp.mailfrom=rmacklem@uoguelph.ca X-Spamd-Result: default: False [-6.00 / 15.00]; DWL_DNSWL_LOW(-1.00)[uoguelph.ca:dkim]; ARC_ALLOW(-1.00)[microsoft.com:s=arcselector9901:i=1]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; DMARC_POLICY_ALLOW(-0.50)[uoguelph.ca,none]; R_SPF_ALLOW(-0.20)[+ip4:40.107.0.0/16]; R_DKIM_ALLOW(-0.20)[uoguelph.ca:s=selector2]; MIME_GOOD(-0.10)[text/plain]; MLMMJ_DEST(0.00)[freebsd-fs@freebsd.org]; FREEMAIL_TO(0.00)[gmail.com]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:8075, ipnet:40.104.0.0/14, country:US]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; DKIM_TRACE(0.00)[uoguelph.ca:+]; TO_DN_ALL(0.00)[]; FREEFALL_USER(0.00)[rmacklem]; RCPT_COUNT_TWO(0.00)[2]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; RWL_MAILSPIKE_POSSIBLE(0.00)[40.107.116.85:from] X-ThisMailContainsUnwantedMimeParts: N Konstantin Belousov wrote:=0A= > On Sun, Aug 21, 2022 at 12:02:48AM +0000, Rick Macklem wrote:=0A= > > Just to summarize this...=0A= > > I was able to do a VOP_SEEK() which would be called with a=0A= > > LK_SHARED locked vnode and it seemed to work fine.=0A= > >=0A= > > However, ReadPlus (which is like Read, but allows for=0A= > > holes to be represented as in the reply=0A= > > instead of a stream of 0 bytes) seems to be a performance=0A= > > dud.=0A= > >=0A= > > I was surprised how poorly it performed compares to ordinary=0A= > > Read. Typically it would take 60% longer to read a file. I tried=0A= > > sparse and non-sparse files of various sizes and they always=0A= > > took longer. (If I disabled SEEK_DATA/SEEK_HOLE in the server=0A= > > code, so it never actually did holes, it worked comparably to=0A= > > regular Read, so somehow the overhead of doing SEEK_DATA/SEEK_HOLE=0A= > > was a big performance hit. It was using LK_SHARED locks, so=0A= > > it wasn't serializing the reads, but I don't really know why it=0A= > > performed so poorly?)=0A= > What filesystem did you used on server?=0A= The 60% slower was for tests like this with UFS:=0A= - I created a file with a 1Gbyte hole, followed by 1Gbyte of data.=0A= - Then I read the file with "time dd if=3D of=3D/dev/null bs=3D10M"= =0A= after remounting over NFS (to avoid NFS client caching).=0A= Here's the elapsed time for 4 runs for a UFS exported fs:=0A= Read ReadPlus=0A= 20.4, 4.3, 4.6, 4.3 18.7, 7.6, 7.7, 7.3=0A= (The first run was right after booting, so there was nothing=0A= cached within UFS.)=0A= --> So, as you can see, it took about 60% longer via ReadPlus.=0A= =0A= Now, what about the same test on an exported ZFS fs:=0A= Read ReadPlus=0A= 6.4, 5.7, 5.6, 5.4 110.8, 113.3, 110.7, 110.9=0A= --> Yep, only about 20 times (or 2000% longer).=0A= =0A= For a kernel build over NFS, it took about 70% longer=0A= when on a ZFS exported fs (I can't remember the UFS=0A= number, but it was significantly longer.)=0A= =0A= So, yes, ZFS is a lot worse, but UFS is bad enough that=0A= I can't imagine anyone using ReadPlus instead of ordinary=0A= Read?=0A= =0A= LANs have gobs of bandwidth these days. WANs might=0A= benefit from the lack of long streams of 0 bytes, but some=0A= (like my little DSL modem for my internet connection) will=0A= compress them out anyhow, I think?=0A= =0A= > >=0A= > > Anyhow, unless the performance issue gets resolved, there is=0A= > > no reason to commit the code to FreeBSD's main.=0A= > > (NFSv4.2 operations, like ReadPlus, are all optional and are not=0A= > > required for an RFC conformant implementation.)=0A= > =0A= > Why not commit? It might make sense to add it, but guard under some=0A= > knob.=0A= Commit it with a "never use this, performance is terrible" doesn't=0A= make a lot of sense to me, unless the ZFS performance issue=0A= were somehow resolved.=0A= =0A= I am now actually concerned about copy_file_range(2), which uses=0A= SEEK_HOLE/SEEK_DATA. There is a patch under review that at least=0A= increases the blocksize for ZFS, but the effect of disabling the use of=0A= SEEK_HOLE/SEEK_DATA in copy_file_range(2) also needs to be=0A= explored.=0A= --> Retaining holes as unallocated regions is nice, but at the very=0A= least, it could compare va_size with va_bytes to decide if there=0A= are holes worth looking for.=0A= =0A= rick=0A= =0A=