From nobody Wed Nov 22 07:27:39 2023 X-Original-To: stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4SZtBD1VSXz522RN for ; Wed, 22 Nov 2023 07:27:48 +0000 (UTC) (envelope-from mirror176@hotmail.com) Received: from NAM12-DM6-obe.outbound.protection.outlook.com (mail-dm6nam12acsn20807.outbound.protection.outlook.com [IPv6:2a01:111:f400:fe59::807]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "DigiCert Cloud Services CA-1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4SZtBC22Xxz4cZq for ; Wed, 22 Nov 2023 07:27:47 +0000 (UTC) (envelope-from mirror176@hotmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=hotmail.com header.s=selector1 header.b=IwsDjY1T; arc=pass ("microsoft.com:s=arcselector9901:i=1"); spf=pass (mx1.freebsd.org: domain of mirror176@hotmail.com designates 2a01:111:f400:fe59::807 as permitted sender) smtp.mailfrom=mirror176@hotmail.com; dmarc=pass (policy=none) header.from=hotmail.com ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=UQ+FEhJsiUMlIWvddiLLxSbLAlvb2bHufgSvrCJ4Z+pDM2jWvVVRKaPAGPeXtIpuX+BJgnh3jOUmKklR9GqIflcIM/Nwz6wzZ0ob1NUYoOWuUnq47dIumCVoBaZbtCydCxxWDNdhRC2wH8S/IzwF1tomBs+0PuwRptASFbq8Axmkr0lQdkWVjRs29TgwWrllZVXvlP//SiNXF/d7Tk5HTwzqG218ZA9d56EWvpRM8Y0TDUJWwkt7CsHhROxCkLFfYPxGcjpOfgCaKER4Wo4lx1eqI1PjA2omlMADmKdn5F411Q7+zNPSIMIHokRSrYk6n8xD1Vb90kvnoCXwtF/R/g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=pr4i97wBp6up+99C0BOykzKsyDoqJYoYggO2yT9Bv+s=; b=mwdY1Vq/jopoVoRwutTsY2L+XR9u17fHMpQXKM15k3qB27cbvoGby0NtuY2ePwQB8I0tJY383pJFSQiDebm5gtkBWgBxvCepSQZhAfZHOccnRHAZLC3/XFNBYSOZAU+n5uJV5FLp0YabMa/vQ6TYdrlCPx4oxIUR5CD8+79VPaIIOmg65pOknXsWVBYPbp5HJuydLYr4WtApUjb9yXX5wINZhuO7TezS0RJ3Wk1D4RGuUgR0vQZSj8hFkREg8EMtMuZdRNi7t6BqatP7E4flaMspbSvSegn1RcpMLwgG3tI/Xzd2EPbKzIvpt9GNwphexCUDnz1H+aex/QKiVFJ2Hg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=none; dmarc=none; dkim=none; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=hotmail.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=pr4i97wBp6up+99C0BOykzKsyDoqJYoYggO2yT9Bv+s=; b=IwsDjY1TCIT/L/sTpZk6C71ECN/YHXK0d8yMxCdHGbrboLdOTHrSg2AkUt7GIh5kGj4lmEG0p3EkpI8S4QZsS8MdiAJ85byluzVqpX9q3T4h2xwhxYZmm6C/z8gRdGRLLtABhS4sSS5AY01skTEu08hEWXF3WIu6pOojxx8Bv14P5uXRJimRqBhLrPzlC82LComMfOFf8zf/0J4zhtG/EJEpQI6XkK1vWbSGJAA8KP41A12orFujUVlT6jZSJPyk0AFO4dFKUnp0ed3E8BW1NsL0kdVB4HSYma3OIjuj313bWxT7vxW6pN/c1aFWBuLVGigMFQ0PkvfW6xa3y99++w== Received: from CO1PR11MB4770.namprd11.prod.outlook.com (2603:10b6:303:94::19) by PH7PR11MB8570.namprd11.prod.outlook.com (2603:10b6:510:2ff::7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7025.18; Wed, 22 Nov 2023 07:27:44 +0000 Received: from CO1PR11MB4770.namprd11.prod.outlook.com ([fe80::10a1:33d7:1d06:31ea]) by CO1PR11MB4770.namprd11.prod.outlook.com ([fe80::10a1:33d7:1d06:31ea%4]) with mapi id 15.20.7002.028; Wed, 22 Nov 2023 07:27:44 +0000 Message-ID: Date: Wed, 22 Nov 2023 00:27:39 -0700 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:102.0) Gecko/20100101 Thunderbird/102.14.0 Subject: Re: Unusual ZFS behaviour Content-Language: en-US To: stable@freebsd.org References: From: "Edward Sanford Sutton, III" In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-TMN: [nYCax6lyfAVqdpC0jSLUZeZkVjIximOg] X-ClientProxiedBy: BLAPR03CA0130.namprd03.prod.outlook.com (2603:10b6:208:32e::15) To CO1PR11MB4770.namprd11.prod.outlook.com (2603:10b6:303:94::19) X-Microsoft-Original-Message-ID: <5231735e-c767-d989-993f-4f1b1c7156d0@hotmail.com> List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CO1PR11MB4770:EE_|PH7PR11MB8570:EE_ X-MS-Office365-Filtering-Correlation-Id: 54c624de-a698-4516-aae6-08dbeb2c845d X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: ZNW733T1BtBMePKRiLJmmZ4k9vMSvP5P/+58CgZt7UdFpD2YsBE9mXCsmWhhGNNM6VZDxEqtsU35yGkowURLMKFbeC9BXut57TZK57OljEBiqHrf8SriMDMaRUzMq8codR+EGkOef1YQzdtleAh594IDwIjTG3Xf2uxZfko4pW2OJzs78ieMyehabB+V6UuqIcQjRLCzGyC6QQz8JTfjIzbTqYB6Ir221aJabvOmHw64XIRO+dGbDeorEKJBo3RfIPh/E5GxG0TaKfQjs4IP1oYWNvPISbI5VkbBw84IE3yZXCicyJK/cCd6UZ3U21XNoRq5NzemLQdWHs5VWA2jvLwfbYTuu/7h7kyh4l8TeSpYdrtkINmemmLJS+m2d4IQCpEDDvg0MVxsQUCS5/KGbPPewuvBhDS29Nskvi2nm2pQtQ9IWixo7bZ2msihDawLc0pgnuSAema7Af2impAVbqxox+kk0uZPUExJSDDqsopoC4Zc8onZFtpoZDzK2AqqIcDlPE3Hw4ASHYqDWKYpf3d6n1U0MAF7RrDAiDJWRF4fPbx8LPvhlZVDb3QYrqZ1 X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?RkxDZ1d6V3Bmc0NIK000NUI2aDJpYktkdnl6c2tObEZ0cDNnR0QzRjVVaGJr?= =?utf-8?B?cHdpdUdWalhURWNBeEw2blZYQnptVzE3NklZYUdoWFlZbFQ3K3pDcFJVTVlT?= =?utf-8?B?TDVpZlNxcnJjVzQ2UVFzTXg2dVRJZS9ya0EwWks3czE0NEpLOUtwVTh3cmJB?= =?utf-8?B?MGttMzhJUTUyalpmQXRHeFRLUDVnN3RFdU1DSGZRbUxNMG5ZRUZoczJRVEdx?= =?utf-8?B?bjNpQTUwQWZZM1JJL0FlMW5iWkNRamZJTmFDaUo3TjBjbjVBZlJ0dVduVllK?= =?utf-8?B?OUQvSHZFUGxuaWFyNEloek00V1Vvb01nVVhWRjNCTFB5RXplV3V4c3NOVHB5?= =?utf-8?B?a2JyN1hYdVc0NVRHQTU2VndtSHoyS0duaWhoM25pQmQ4ZCtwbDRPYVUrUjl5?= =?utf-8?B?azhZc3BmR203bGIyLzZZNTdZMXViZE1icGpxV213dkZuNGV5UVJRU1BBL0VI?= =?utf-8?B?OXAvNVUzZkFDUUN2Znc0SkxrNDJOVWh3ZXZjdnFIb1F2bGtaMlhuVFdJdkRt?= =?utf-8?B?azJXeVJVZ0dmQU5GNG5iRjRETHYxL2p2UitFR3F3QS90QzM4MkFaKzdySXJr?= =?utf-8?B?bm5Nc3N6V25FbytBc0xjb0VOYXdXWWFaQkNxaWZIY2E1eGs4b2hMbzQ0WHdT?= =?utf-8?B?Nm1hbGpmYit2ZGsyY3BCZXp4ZGNpV1A1Z1VEam15dUdkUEhvbkdnZGd4b1h5?= =?utf-8?B?WGROaDhFSzJtbm1qVTM3em95cloxaHVqQVBab0grbFZMbnVWeVMzaG9TRmdH?= =?utf-8?B?bXJtbkxveXg0WUdpZW5OOFBBQmZlNzdNVDhzTDBIejhscndiNnRjMVMwZENN?= =?utf-8?B?bFpmaG5VSjVHcTRPUFNubWRRS1NHb1dydU1BazVtejVUZzRFb2xHWEJIRzFm?= =?utf-8?B?bkVVVVA1VE5aWHNWRmNJMTZaQ0lIMkhwRjJ4b0pkTEoyc1NVSXNBWlVGUXRk?= =?utf-8?B?OFFQN0lEZFVPMjVNK21pZHptNFZzbW9HOGFqRUFLS3pvTDVrNmFyZEhlSFlX?= =?utf-8?B?KytwU0VoSXRjSWU4VVZUVjlIckx5QSs5Zk5ERHdrOEE3cTlZS0lVeVB2K1hC?= =?utf-8?B?a0l6Uk1xZWJvUC9wb21wSmJ6YkJURDYzb1BlRFZBamV5b2VYVElucklUclFo?= =?utf-8?B?SU0zZDhEb040UEUzd3l0aWtWa2ZUL0l0cjhhVi91WWtHTGsxNjE0T0hKdVdw?= =?utf-8?B?WjVpZ2JrRUdBV1phcHRMcHpNMTRNL2tGaktzclpkbzhkUjBEazdQUW8rVVF1?= =?utf-8?B?TXZza1NEazhNS3h4TXdvalN6TVUzdHIzRmxoSEdsMUtHRi9vRVNKdkRyVksr?= =?utf-8?B?RncxeVFDdHBySEZSZlNiZFNTbjNpc1V6bEhvMTNOSWZYd1NuUEdqakhJM1JK?= =?utf-8?B?NWRRNU54VmFRVVdzbXp2VUlOT2x5aGxvSFQ0YStYNW80SmJSQUw5UmUyQmJr?= =?utf-8?B?elhZQml6YWRtQmtJcmppNnE5bmhrcDl0OENmMEhFQjR3Ui9sc0pCT0IydStP?= =?utf-8?B?OUNUOC9lNnBxOGpwMXpIS0lwNmtmL2IxU2lGYWY5NUVVandXU0htVjdxdlhR?= =?utf-8?B?aE9WYVRpNjlKdkppVE1uUFZwdG9jaTFnVFB3RWZPL1JVOE44UFRQTU5qSzFn?= =?utf-8?Q?DZERGyetPN5LLNKqRsYilaMXxo+2cVph4uU8KD1fHYsM=3D?= X-OriginatorOrg: sct-15-20-4755-11-msonline-outlook-e8f36.templateTenant X-MS-Exchange-CrossTenant-Network-Message-Id: 54c624de-a698-4516-aae6-08dbeb2c845d X-MS-Exchange-CrossTenant-AuthSource: CO1PR11MB4770.namprd11.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 Nov 2023 07:27:44.0025 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 84df9e7f-e9f6-40af-b435-aaaaaaaaaaaa X-MS-Exchange-CrossTenant-RMS-PersistedConsumerOrg: 00000000-0000-0000-0000-000000000000 X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH7PR11MB8570 X-Spamd-Result: default: False [-2.50 / 15.00]; FORGED_MUA_THUNDERBIRD_MSGID_UNKNOWN(2.50)[]; ARC_ALLOW(-1.00)[microsoft.com:s=arcselector9901:i=1]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; DMARC_POLICY_ALLOW(-0.50)[hotmail.com,none]; R_DKIM_ALLOW(-0.20)[hotmail.com:s=selector1]; R_SPF_ALLOW(-0.20)[+ip6:2a01:111:f400::/48]; MIME_GOOD(-0.10)[text/plain]; FROM_HAS_DN(0.00)[]; DWL_DNSWL_NONE(0.00)[hotmail.com:dkim]; RCPT_COUNT_ONE(0.00)[1]; TO_MATCH_ENVRCPT_ALL(0.00)[]; FREEMAIL_FROM(0.00)[hotmail.com]; MLMMJ_DEST(0.00)[stable@freebsd.org]; DKIM_TRACE(0.00)[hotmail.com:+]; TO_DN_NONE(0.00)[]; RCVD_TLS_LAST(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; FREEMAIL_ENVFROM(0.00)[hotmail.com]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:8075, ipnet:2a01:111:f000::/36, country:US]; RCVD_COUNT_TWO(0.00)[2] X-Rspamd-Queue-Id: 4SZtBC22Xxz4cZq X-Spamd-Bar: -- On 11/22/23 00:04, Eugene Grosbein wrote: > 22.11.2023 13:49, Jonathan Chen wrote: >> Hi, >> >> I'm running a somewhat recent version of STABLE-13/amd64: stable/13-n256681-0b7939d725ba: Fri Nov 10 08:48:36 NZDT 2023, and I'm seeing some unusual behaviour with ZFS. >> >> To reproduce: >> 1. one big empty disk, GPT scheme, 1 freebsd-zfs partition. >> 2. create a zpool, eg: tank >> 3. create 2 sub-filesystems, eg: tank/one, tank/two >> 4. fill each sub-filesystem with large files until the pool is ~80% full. In my case I had 200 10Gb files in each. >> 5. in one session run 'md5 tank/one/*' >> 6. in another session run 'md5 tank/two/*' >> >> For most of my runs, one of the sessions against a sub-filesystem will be starved of I/O, while the other one is performant. >> >> Is anyone else seeing this? More details of the disk, disk controller, and FreeBSD version may be helpful. If it is SATA, maybe there is impact form its own organizing of when to handle the queue of tasks it is given in addition to what FreeBSD+ZFS have assigned and OS (+ZFS if not its default) version could say what IO balancing is currently present/available. > > Please try repeating the test with atime updates disabled: > > zfs set atime=off tank/one > zfs set atime=off tank/two atime's impact is a write and writes get priority so if anything there would be 'little' breaks in the reads to write such data. I doubt the scenario+hardware in discussion is bottlenecking on writing atime data for the access of these 10GB files but it would be interesting. On the other hand, I think it is atime that trashes a smooth disk of freshly created file structure with many files after default cronjobs pass over it due to it + COW fragmenting the data structure to do basic things like list disk contents; I have not tested properly what the source of that repeatable issue was yet. Accessing data within the file doesn't seem impacted the same way as the directory listing though. I was thinking maybe an impact with sysctl settings involving prefetch may impact the sequence. vfs.zfs.prefetch_disable=1 is probably the one I am thinking of but I normally don't tweak zfs and related settings if I don't have to as I usually later find the tweaks are problems themselves. I keep my system running smoother when I put it under excessive load with idprio and nice being used on the heavier noninteractive load. > Does it make any difference? > Does it make any difference, if you import the pool with readonly=on instead? > > Writing to ~80% pool is almost always slow for ZFS. Been there, done that. It is painful but I think it is more complicated than just a total free space counter before that issue shows up. Other performance issues also exist as I've had horrible I/O on disks that never exceeded 20% used since being formatted.