From nobody Thu Dec 16 14:58:23 2021 X-Original-To: dev-commits-src-all@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 84F8518FEBC1; Thu, 16 Dec 2021 14:58:32 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from CAN01-TO1-obe.outbound.protection.outlook.com (mail-eopbgr670050.outbound.protection.outlook.com [40.107.67.50]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "DigiCert Cloud Services CA-1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4JFFc82VnTz3D3K; Thu, 16 Dec 2021 14:58:32 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=flfZMi18X2UDwKzMpKiFhchWTIn0HIigLbP19CwNw5ZvLi1GNvpMxzqy3ZFZyndPWdIqX7cV8SgG9HnNUK9WHTq3pgYz5BBaJB7tZ0JJ0SitPUuJM0nPizrlZtrwgNBYDVqE6PxTk0HrW460J6xIZCi50yF7XqGSiIjGR8359yd4jyOkX41U7+B4FeUHOF6BVHerCv8BtIxiFwBPtrf8M+qO/SKhvjYBOqCONcMlTPX1hZWVO59lH3EK55yTOi5pO5UblJwZuqsH7e3wrvXfJju4tf5UuB5V+gUuyW73ED2SnkSe52Pa29pEha814wNp9xAnY+1kY87BNYJsEnVP8w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=QSK1qPacar54zgKh26LVWa3kRElDfIBNknsP2dsDgG0=; b=X6n0OVAVuqxreukuyHZgx3HbrqgjVXg2NKwNo9eAUqq0lqDvWAaVud0TFPdIlzhIrXN5a3WGAT/7jwofVs65F4Zi1zy4tn9AepSiGNLP9arDtiZczdwJYO/Z8odzPvKgXQOoRG+GFvizV2hW7wZ/BEoLfcGZ/CX2BhUcCCUeEbKtvLJ/cESSYtxPJNoHvOAkgO/Qryt3lp2/W33O+FPLEL4DxDu/c8oyC5m8BjJbusCV5jEj+Zb2MQqJ0fNTOox2v/5Dyj56lcRwg/onTFrxuV18oaR5TU5tVPkGXgE5QeTEZbJ/bI/DRPXJHK1I3V+H2O+mA+PIncExw5MHMN2qDw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=uoguelph.ca; dmarc=pass action=none header.from=uoguelph.ca; dkim=pass header.d=uoguelph.ca; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=uoguelph.ca; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=QSK1qPacar54zgKh26LVWa3kRElDfIBNknsP2dsDgG0=; b=DUrW+Zf92bH5drxjvb1pREcMMtHeeDZZ2ivCg/o+d6/fVrJjGapySjZsW09okiJfHzf6/YYFWRHGm5BFey7Tvc12errYYPD4z9Og8RhE448jOE4kdszhlem3yrZQYzzDFyPlb5ShGV3rsN0UKF3Deb1mr1eUTr/i1GvqvoAD6ZFF8l9RSX8trLN1vgNfmYG6MHq4G5ke6pTw5pDODr+PO11G2ScUX6sdz9OcYtYCSuFpHSDyIRb+UK+MeNLFif4qMcYDHNi9yFWvNdqZ7CmVvl9d32WJSrTYTKa4UEBoCdGFZl2BC/22BWL3AiqJmuIMxDMCTWxIrHGfcLiuOAB2HA== Received: from YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:c00:19::29) by YQXPR0101MB1624.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:c00:1d::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4801.14; Thu, 16 Dec 2021 14:58:24 +0000 Received: from YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM ([fe80::e56f:b7a2:3830:5706]) by YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM ([fe80::e56f:b7a2:3830:5706%3]) with mapi id 15.20.4778.018; Thu, 16 Dec 2021 14:58:24 +0000 From: Rick Macklem To: Konstantin Belousov , Rick Macklem CC: "src-committers@freebsd.org" , "dev-commits-src-all@freebsd.org" , "dev-commits-src-main@freebsd.org" Subject: Re: git: 867c27c23a5c - main - nfscl: Change IO_APPEND writes to direct I/O Thread-Topic: git: 867c27c23a5c - main - nfscl: Change IO_APPEND writes to direct I/O Thread-Index: AQHX8jUPlyWpJN4Oqka3Z2/0kbadeKw1MXAS Date: Thu, 16 Dec 2021 14:58:23 +0000 Message-ID: References: <202112151639.1BFGdS2v011996@gitrepo.freebsd.org> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: suggested_attachment_session_id: 365d6b0b-cc53-be03-72ff-b26efe1ede0f x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: fca862d8-d994-443e-1002-08d9c0a48225 x-ms-traffictypediagnostic: YQXPR0101MB1624:EE_ x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:5797; x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: yqAiuubDaBFDZ5fnrtJKdesAfaOuUyYQfRRBVv1dtOa2AGI5NZv6rGT0PkPAcDyW0aoIryJyQ+bcQRBiISY/CO+OB40tho/fPMhZoTshJoITNrFIhJ+TuFxBRZmbzvQuzBINjmS8vgW378aYUA4unREOymsUxVq2X2BTLLqOavrWjP24dXIAzYZV2zojfmmH5kTkAjfLDViD19Xc1DGHWC07v7r2DU0f7xK2fTtDkL3sIkllt3bU5Ah2Jb9lYgRCar417mS00TSd/MEYelrBpvXOPlO4bPtn25qqFZwvM2XvS0z2byhzFJ91rhNLTg7zO11sg5mroOlIU8cDUXeWzPPpQArtgPh4Jb+V9K9Uxh69ApNvLQtKKWtn5bj91tJkqj34gSySHgFVTqPXbJOGoFWUa/NNVQN3djDSLX/y2FUK78/Zm1GdHET9tM3uqGv/nii/nXEX+0rzD2HxngD0mfG0ZEwBRbtRQ6DFauH7+V7P1aZzQs8N5CKBWP11uQxgHZuSB/l8iU2BBBF0kSBhz8EiRmU970tjH5uytq7NUapxZeIJ0TpEmdY4Zy29N5KF7MpLg1NpsDKE4PdAlzA96w39mODTI/9yn/CuB3zLUo5HGugiEOvr29IgRqO75Qr0qFS34MbE/jHtRJXDXlKlJH4P8kExwirCwSACJ7/PB3I1Y8y6FVXCFemNV+65qTebG2WX6nuXMrCVOLRGeFvFnjJcRPVAM8Dn9+V+pu6oJHCtWYe5UyR9EI3t25fIz2qkfxnIvXiiiKv7bYQ0/EKn47m4jWCtU0ziyJTJU9Spzkk= x-forefront-antispam-report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM;PTR:;CAT:NONE;SFS:(366004)(6506007)(4326008)(5660300002)(71200400001)(9686003)(83380400001)(2906002)(38070700005)(52536014)(86362001)(54906003)(33656002)(508600001)(8676002)(966005)(786003)(110136005)(316002)(8936002)(76116006)(66476007)(66446008)(66946007)(66556008)(64756008)(122000001)(38100700002)(186003)(4001150100001)(55016003)(7696005)(91956017);DIR:OUT;SFP:1101; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?iso-8859-1?Q?zYkh2pIntxRs+DjTM6wcvGnlQEDEV5geXxDc2cRS5Y4//q3BIIRNKLi1No?= =?iso-8859-1?Q?wX1nLDy8HfOJF4zz/4DNXknBRujXMOXYGQ2w+Y7PzquHgRheJS53H4/NBY?= =?iso-8859-1?Q?eWjbdk7j9NMd6vdjC0fgx2x9kBVqAoKNfble4hGVoAq4P+K1OvbzVk/gXu?= =?iso-8859-1?Q?zaac5NF9odQ+glyAPed31jxZ1WyuEfqAmCEyaQKU7F907XBPJRC6y2uJii?= =?iso-8859-1?Q?7kxbpD7gPMSPkftC8KoBKOKwIV+gsBZ3dzG22c3wqkdnm99Ah6kA1D+YQn?= =?iso-8859-1?Q?QFsDvHKtHVozNVqT0kT9+puqMc83pLY40E0+Ecg390qyHErd5uexqE42Yg?= =?iso-8859-1?Q?7kV0a1PWLz0BaE3eo9QhrAG9xCw2V/pSrXO5hcuo8wUyZzZFkSZfMrD4e8?= =?iso-8859-1?Q?ZpfarQ4FCt2QqEqNheQ1BQ0sbt5erbF2XFkzzY6MOaTO11QCcZNO0n2oTu?= =?iso-8859-1?Q?qETZZ1grgc3oP8zyOjMGYXXLBjXx498QxZ/D1X26hpYQ3uGtzdqXp8KvQW?= =?iso-8859-1?Q?VSsRLzh8N/jDGFdjF4z4aFIjCGODfb/20AehGK/DOS3vKlNpMk8r+E9q+g?= =?iso-8859-1?Q?B2T0yl5Cnu2+IaXso5rVtNBhwQqbfh55uz+fkYoXh5A4FBcdzbIL6s53mP?= =?iso-8859-1?Q?DFh+DS1V1U4nuBpCkYksYynQn53P4gHKEONZXYLsSDnNuNOQso7C4y/t2u?= =?iso-8859-1?Q?m+G+ZJhzX90haFLT+41HJpON2GQzLT6OTgBAKkXDQwmJ2jKjojBiN/YDE+?= =?iso-8859-1?Q?lwctmEGXZni+Jv5m+3QD87n8DlxDTQgm279gwJuAkMeiiNBnKebcc13CS6?= =?iso-8859-1?Q?D6Od+jzKjJy3o5za+kIMKvaAMLYcUtBJflwZbdS9mYLjplIh/nuJlk2KrX?= =?iso-8859-1?Q?xE8HCRtueIza07F64E8hgo0au2Xc4KSt5wSEyLq49tJDTHciANAgR9ciGl?= =?iso-8859-1?Q?lM90MBR3YVOOB0k2UxAknFsl/GvcXaDec85FudeYIedOI2YYocFqJczeuW?= =?iso-8859-1?Q?iuo8b9/zy14ziSkfPjxP/5KvxII0vru9mHSpj4h+oVcmARR4S9wNEWK4p0?= =?iso-8859-1?Q?2dIZhSG2uY0zFlODZQwFEz3XIXfFxhlty8icjl+J5IKvm2NrafS/WmQenf?= =?iso-8859-1?Q?UybD7OZlHUMMawRlLk6dczJeOAHG3AtwrTQMNhSOqDlxiJrtXhiZZhagNx?= =?iso-8859-1?Q?nloWCkAW5V7eFiW/7B1bogrJyv1XzmmOMYbgt2U51MVjaaiblu+ZHQBjtj?= =?iso-8859-1?Q?f7m4uzE7t9lWoGze9QEfjQWXjrRJIaWf0VhD5JCoEuGqWcy6kBNlzOJCPH?= =?iso-8859-1?Q?i0JqCf2I1LqJL9Ns8j9eWWefYTT1RHXzJJi+USfq8CaAbuzY2BPw7jdRNG?= =?iso-8859-1?Q?amQ7joVEsvI2kyY3rKmOOSKdN3VjssDyV6vuTWB81BWuC4gNNzwn4F4i8H?= =?iso-8859-1?Q?zdXcZRvXgBfT+HXCVvZ2j60wd0t37IppYymGPvFI4PKcahnklMR27RkZpF?= =?iso-8859-1?Q?7uQ+8lt38K/YSqaGhIN4qFLN7hp8g39HZaOw2wc5wdBQPI/eyZ0QudOZCJ?= =?iso-8859-1?Q?PdUTcKm0VA2ZWJHRlxw0SV2vvCoDN56uFILhFzxpiMnCUuxXvTZ8WC00pM?= =?iso-8859-1?Q?W6n7lCXHXHVyhs6MtNzx5qsSanojnn4AJ+Hp8pclCMcm4iwsGYeLrZrSfP?= =?iso-8859-1?Q?HxalWMy/Sxt2JpNr+gFsFkiOefx+xA8MlxNBUARRumKkxhO64/VPS03jjg?= =?iso-8859-1?Q?ImNg=3D=3D?= Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable List-Id: Commit messages for all branches of the src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-all List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-dev-commits-src-all@freebsd.org X-BeenThere: dev-commits-src-all@freebsd.org MIME-Version: 1.0 X-OriginatorOrg: uoguelph.ca X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM X-MS-Exchange-CrossTenant-Network-Message-Id: fca862d8-d994-443e-1002-08d9c0a48225 X-MS-Exchange-CrossTenant-originalarrivaltime: 16 Dec 2021 14:58:23.9063 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: be62a12b-2cad-49a1-a5fa-85f4f3156a7d X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: svgyr2Qyx452BrjyX0ITnPuHQJZtx9x3CzpTxRTs/y44rs1kRjwMpRziWNanlwcspMQCFh/G4kYb6+ENAlGvyA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: YQXPR0101MB1624 X-Rspamd-Queue-Id: 4JFFc82VnTz3D3K X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[] X-ThisMailContainsUnwantedMimeParts: N Kostik wrote:=0A= >On Wed, Dec 15, 2021 at 04:39:28PM +0000, Rick Macklem wrote:=0A= >> The branch main has been updated by rmacklem:=0A= >>=0A= >> URL: https://cgit.FreeBSD.org/src/commit/?id=3D867c27c23a5c469b27611cf53= cc2390b5a193fa5=0A= >>=0A= >> commit 867c27c23a5c469b27611cf53cc2390b5a193fa5=0A= >> Author: Rick Macklem =0A= >> AuthorDate: 2021-12-15 16:35:48 +0000=0A= >> Commit: Rick Macklem =0A= >> CommitDate: 2021-12-15 16:35:48 +0000=0A= >>=0A= >> nfscl: Change IO_APPEND writes to direct I/O=0A= >>=0A= >> IO_APPEND writes have always been very slow over NFS, due to=0A= >> the need to acquire an up to date file size after flushing=0A= >> all writes to the NFS server.=0A= >>=0A= >> This patch switches the IO_APPEND writes to use direct I/O,=0A= >> bypassing the buffer cache. As such, flushing of writes=0A= >> normally only occurs when the open(..O_APPEND..) is done.=0A= >> It does imply that all writes must be done synchronously=0A= >> and must be committed to stable storage on the file server=0A= >> (NFSWRITE_FILESYNC).=0A= >>=0A= >> For a simple test program that does 10,000 IO_APPEND writes=0A= >> in a loop, performance improved significantly with this patch.=0A= >>=0A= >> For a UFS exported file system, the test ran 12x faster.=0A= >> This drops to 3x faster when the open(2)/close(2) are done=0A= >> for each loop iteration.=0A= >> For a ZFS exported file system, the test ran 40% faster.=0A= >>=0A= >> The much smaller improvement may have been because the ZFS=0A= >> file system I tested against does not have a ZIL log and=0A= >> does have "sync" enabled.=0A= >>=0A= >> Note that IO_APPEND write performance is still much slower=0A= >> than when done on local file systems.=0A= >>=0A= >> Although this is a simple patch, it does result in a=0A= >> significant semantics change, so I have given it a=0A= >> large MFC time.=0A= >=0A= >How is the buffer cache coherency is handled then?=0A= >Imagine that other process either reads from this file, or even have it=0A= >mapped. What does ensure that reads and page cache see the data written= =0A= >by direct path?=0A= =0A= Well, for the buffer cache case, there is code near the beginning of=0A= ncl_write() (the NFS VOP_WRITE()) that calls ncl_vinvalbuf() for the=0A= IO_APPEND case. As such, any data in the buffer cache gets invalidated=0A= whenever an Append write occurs.=0A= =0A= But, now that I look at it, it does not do anything w.r.t. mmap'd files.=0A= (The direct I/O stuff has been there for a long time, but it isn't enabled= =0A= by default, so it probably doesn't get tested much. Also, it has a sysctl= =0A= that allows mmap for direct I/O, which is enabled by default. It causes=0A= getpage/putpage to fail if it is not enabled.)=0A= =0A= So, it looks like code to invalidate pages needs to be done along with=0A= the ncl_vinvalbuf()?=0A= --> I'll come up with a patch and then get you to review it.=0A= =0A= Thanks for pointing this out, rick=0A=