From nobody Sun Mar 03 12:33:02 2024 X-Original-To: questions@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Tnh7X1n9Sz5Chrt for ; Sun, 3 Mar 2024 12:33:12 +0000 (UTC) (envelope-from mirror176@hotmail.com) Received: from NAM11-CO1-obe.outbound.protection.outlook.com (mail-co1nam11olkn20801.outbound.protection.outlook.com [IPv6:2a01:111:f403:2c16::801]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "DigiCert Cloud Services CA-1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Tnh7V66jGz4rQs for ; Sun, 3 Mar 2024 12:33:10 +0000 (UTC) (envelope-from mirror176@hotmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=hotmail.com header.s=selector1 header.b=fzxSFx56; dmarc=pass (policy=none) header.from=hotmail.com; spf=pass (mx1.freebsd.org: domain of mirror176@hotmail.com designates 2a01:111:f403:2c16::801 as permitted sender) smtp.mailfrom=mirror176@hotmail.com; arc=pass ("microsoft.com:s=arcselector9901:i=1") ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=gT7rkVod6/7HtGqCUZRvS86wrxYA9t0THsUeSId0wfZPmhj6fcUvbhlr6ZgQyEhUzDfD5d3GV1CKbjKVuIhD+CMYITBbO+tYjyHjxKbY18pdJcmGsFzre18N/T10NfypdRJDjUL7IZNuKBDaZEmnruBwkoXRYXAGJ06QIycPIcYZ/KU4HV3di73HGwMSAhnnU1ka5PGc1mBHdnISDMG4mtGXN2K122fP8BhNrA3lHRRjT8pwKKBzH0huC8tQMWopce+Zmriw0bDH2jWf5y9PCctuASLf4fPgpfI2tlBSgC/PaFAe4F3Afz3Xu5E4DcDjnGGbycyJ+t7zcSaUVLDG3A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=+1D0wczGavHlJn3sSrmZTfIEW1VXmGFnul8zuCmB/rA=; b=oZFnYc9bvVjZbD/pyrPrsA7kWYSc+GpkX+MBRFwdGptR+UcV4Cq/CmWjFaJHriqxZymMCdbO7bCnrKVWDd5zi8twKDqUXrzjONFgC9w9OF5YFKjAHM3lJWTpI81UQBfH8Vy4YqYEx/CjuAJ3X/tBAhrrHIVAnKfIsm9UgU8K+T0zatLoTfc0tGVtw74d0lliGDrM3jv9w2M7eK4LHS8goUgHSLap7z1SvLeQTyAbmQL1aeQYn1DQCMf4y/1ir+w7iX/ueWZ0Oa3ZE9qdn9fb9b5dGyZ29BWoBqiQyvt6FOp59aBSGXRANgo8ctEuuGgpq8cehH2WzBFc6yC4rJhxAQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=none; dmarc=none; dkim=none; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=hotmail.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=+1D0wczGavHlJn3sSrmZTfIEW1VXmGFnul8zuCmB/rA=; b=fzxSFx56W6d17wyfgWkKqh6V4DNV0nku5dL/uAt+WiKNOFrMp5o2aNRmvzBr0nfXPyBK5L8AlKgvS8OIp9UnililV/0Rl6MWJw94o8ubIKy459ZuQE4ORVtklGcE/cF+gtNisQaBVBrv9Jv9W7NeBrgltRWS10lOdOglD04ZsuE20sGPJqwkigSGw0ahiQaejvzFve7iR7AedFgs/IsxbNL+0ViwQVLuQyt3z5FNi1LMdWWzMbzG7h//0sjzmMeNptPlShhSlEkBFOo4T4Ymcx6xbE0MQty+RlzHbhJdfb1tvsggGWHtyaxCY000towfVZ+97yEshJOnhqimLRgOpw== Received: from CO1PR11MB4770.namprd11.prod.outlook.com (2603:10b6:303:94::19) by IA1PR11MB7175.namprd11.prod.outlook.com (2603:10b6:208:419::5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7362.18; Sun, 3 Mar 2024 12:33:05 +0000 Received: from CO1PR11MB4770.namprd11.prod.outlook.com ([fe80::e526:b74c:4798:1295]) by CO1PR11MB4770.namprd11.prod.outlook.com ([fe80::e526:b74c:4798:1295%4]) with mapi id 15.20.7362.019; Sun, 3 Mar 2024 12:33:05 +0000 Message-ID: Date: Sun, 3 Mar 2024 05:33:02 -0700 User-Agent: Mozilla Thunderbird Subject: Re: ZFS pool/filesystem space oddity Content-Language: en-US To: questions@freebsd.org References: <20240303115859.922d34bb2f47b6df9093315a@sohara.org> From: "Edward Sanford Sutton, III" In-Reply-To: <20240303115859.922d34bb2f47b6df9093315a@sohara.org> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-TMN: [ZptPeNuD/BO6QMl8tLddYRzeQzpJu+5S] X-ClientProxiedBy: BLAPR03CA0126.namprd03.prod.outlook.com (2603:10b6:208:32e::11) To CO1PR11MB4770.namprd11.prod.outlook.com (2603:10b6:303:94::19) X-Microsoft-Original-Message-ID: <90cce8fb-ff06-4abb-9b51-8b8b9d0a50d3@hotmail.com> List-Id: User questions List-Archive: https://lists.freebsd.org/archives/freebsd-questions List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-questions@freebsd.org X-BeenThere: freebsd-questions@freebsd.org MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CO1PR11MB4770:EE_|IA1PR11MB7175:EE_ X-MS-Office365-Filtering-Correlation-Id: 9f0869b3-102b-49ce-d376-08dc3b7e131d X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: dndk2fx3+zGNJi6z/loTX86ETxApNOV2VZ6ps+RJFBHJueVg8H21a8wZbxoU4uJAC/WauZXxFSVO1iBvUpTbppLcCgjwURzhJhByH/0FposqnJx5MXTIH+8REvpTsS9oEeaosyu1uDaYA17X8UV7GAIBtH5/m+7g48Jh3oRZP2z1WbOJUDjbRrZzwOJcW31tMAJEtoZs7yEsHExtz8EreKdJ1QlXj7UvDajul44HSG43F5ISgo2iuVDLjNTdSm4iPlP8K7zfACznL886ez2wsjsVXImzBET42+mCvmqj3ptYU+HWV7dwKTvbhoOeGb4foQPaOAnb/BClucQsHunY9dAfecFMmudlW08Lt49XnugAjBOIXCEZBYK82MDzBZp9wZmDgVjB0vPu5XdEKdTRXGi6rwB3pbvj8fbGGCFT/bMTlRj9Ci9dqFbx16ysZcIkZsORGBlLl93cb2CCL8zXAuRO4NIZxeDuUiCz03j0jo7bGYq8cKUFzP0UfAv+ihRgwvifI5DMx5GkQpRxnI2IQ6skk2I7SLNpOv6UEAyMGlMj3wDa9ePY/lCHoszAM2cq X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?d3pYUlhJSmx2SDBjaWM2dUxhS0UvZEpSNWF6Q3FpL0V1dDF5aklVRWM4M3JF?= =?utf-8?B?ZWNwT1hrZFpaMjkyMGVpNGRmRi9adFFsZ1RiQ0xWQUtJV0FVMjNLZm8wUm5j?= =?utf-8?B?djREeFU5eVhlOEVjWCswYlU2cENSbjMzQVZBeFRpdUdNdlV6MG9TQUtXTlpV?= =?utf-8?B?RkdSc3hSdDBkQ3RLT0NYQzhWZUJaTk9mMTRTaDRpa0g3M0VwOUV4UGJOaHVz?= =?utf-8?B?OGdZWDNUc2cyNU5UN2xYS1B5QlF2NVBGRGxQUGpqTjVCZGtVZ2RxMU1ZQXU1?= =?utf-8?B?ZnFqd0kwMjRacTVoQnlXNlBNbXN5ZmhwQjc2VVlWV01xVFRMK0RyMFR4dWdT?= =?utf-8?B?NHdNbU9CRVFvSXVQY1l1TWFwQWhmQlFMRkFEUzIycFRwdGdrNExxM3MrVFVQ?= =?utf-8?B?bG9lZ1BzQVhyNWdSakdOVjZSR0htdE5uWHRNK045K0ZaeG45OFl6Sit4MmVy?= =?utf-8?B?WURlaTNCNERBZUp5a1k2bVE3bHhjb1VhUnVQanBpRlRNY2MyRXJEQ3BMMUcw?= =?utf-8?B?R0lKbkRYZnVKcGZTc3I0ZXJwaFlZSzM1eEdXSU9Ic2xMb1g1M1NaQThvbnNI?= =?utf-8?B?U29jUVBBWkNVdkFpNUhwZXRHaGJ5WjNrR0Z5TmozbGtoa2crWkoxa1JiMDhO?= =?utf-8?B?NFNiUHo1QVF3UDlLMzZjWnZNNmZITWNIcERpa2dsSkd4Zyt3eHlxQWpsRDlz?= =?utf-8?B?QzZSU1pqTlZ3NDk2WWxESXV6VGRMbG0wN00wQytDK2M4Q3FORDNKU0c1cjNh?= =?utf-8?B?ZHFxNUxVWWtJWUczajlDdXJiQzliOWR6amFUanFtQTBwcWwvbHpDaTlPd2Zv?= =?utf-8?B?VEw4Qm8vNFZMWXhJMGFiOWVTTDFFTUpZdHFiRC9xZ2dXSGlDTGVjOGF2NTF4?= =?utf-8?B?Q29SOHlXU3dqOWp0MU9jWnQ2QVNxdkNDM05oYnFMVVFybFhVNENFVGYyVXFR?= =?utf-8?B?MFJnZW9WS2VFRWVzQ2NZNFBMZWV6WHcrSUhyQ2JEUU1NSEJlM3lVam9hd2xx?= =?utf-8?B?RGNybzJHRGRCdUlHN29UZ3ZlVTg5Wll4ckRITlJIU1pNNFUrWWhJUEg2bWxy?= =?utf-8?B?ZGJianN5M3NNakUzWmd4VUlYY2tpdjRUMkZxRklFUWhJN21qekNUQXNibDRz?= =?utf-8?B?ZGxSdlA2REMrYmpzWXlpd2pFb2xHVGRwQ3ZDa2dobjhEdG9Vb0JzQjlaV2h2?= =?utf-8?B?Zk05RzBIeXlDSGV3YmVEemdNUDlGZmVoYlNTTkI4NEdKSTF0ZzZsZXNXYmFN?= =?utf-8?B?OVdPblo4dkxkQTN3VzdBbFJaQm5OUFpSZFBPYkNacTNodTdyMlRQbkZkeUNr?= =?utf-8?B?MnB1L1crVm85OTVZUDVYRkJMaUdmN0tTQWM5L0dVTE5qNm5pN2g4c1hKaGtx?= =?utf-8?B?eUNURXhCUUt2eDZqOStYNjNoV3dvODkvSkVTanIrVHhqNWdENHE5YWZUZVFT?= =?utf-8?B?YThaWFYxbGZwUzFwL1NUdktpaHFlakt1Mjk2emdXNFFTVHBPbXFkK2kxYjYr?= =?utf-8?B?KzNwM09SUVNXRFJFQU8ydzZLeGNqUE03QkNBYjdmeEJKTTRvUFd0bk44aVJl?= =?utf-8?B?NGY3MWxPTS9IVlBldGM0Nmg4S1gySVZVNFJEdDFCZkxtcmUva0VtQ3pjWnFH?= =?utf-8?Q?1WzXpXDDJ04MdOFBpgoyFEz52foaNTDw8f2LdF+gRqQM=3D?= X-OriginatorOrg: sct-15-20-4755-11-msonline-outlook-e8f36.templateTenant X-MS-Exchange-CrossTenant-Network-Message-Id: 9f0869b3-102b-49ce-d376-08dc3b7e131d X-MS-Exchange-CrossTenant-AuthSource: CO1PR11MB4770.namprd11.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 03 Mar 2024 12:33:05.5646 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 84df9e7f-e9f6-40af-b435-aaaaaaaaaaaa X-MS-Exchange-CrossTenant-RMS-PersistedConsumerOrg: 00000000-0000-0000-0000-000000000000 X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA1PR11MB7175 X-Spamd-Bar: -- X-Spamd-Result: default: False [-2.31 / 15.00]; FORGED_MUA_THUNDERBIRD_MSGID_UNKNOWN(2.50)[]; ARC_ALLOW(-1.00)[microsoft.com:s=arcselector9901:i=1]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.82)[-0.822]; DMARC_POLICY_ALLOW(-0.50)[hotmail.com,none]; R_DKIM_ALLOW(-0.20)[hotmail.com:s=selector1]; R_SPF_ALLOW(-0.20)[+ip6:2a01:111:f403::/49]; MIME_GOOD(-0.10)[text/plain]; XM_UA_NO_VERSION(0.01)[]; RCVD_IN_DNSWL_NONE(0.00)[2a01:111:f403:2c16::801:from]; FREEMAIL_ENVFROM(0.00)[hotmail.com]; FREEMAIL_FROM(0.00)[hotmail.com]; MIME_TRACE(0.00)[0:+]; RCPT_COUNT_ONE(0.00)[1]; ASN(0.00)[asn:8075, ipnet:2a01:111:f000::/36, country:US]; MLMMJ_DEST(0.00)[questions@freebsd.org]; RCVD_TLS_LAST(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; FROM_HAS_DN(0.00)[]; DWL_DNSWL_NONE(0.00)[hotmail.com:dkim]; RCVD_COUNT_TWO(0.00)[2]; TO_MATCH_ENVRCPT_ALL(0.00)[]; TO_DN_NONE(0.00)[]; DKIM_TRACE(0.00)[hotmail.com:+] X-Rspamd-Queue-Id: 4Tnh7V66jGz4rQs On 3/3/24 04:58, Steve O'Hara-Smith wrote: > Hi, > > My NAS runs a ZFS pool called archive striped over two mirrored > pairs of 4TB drives. It all works fine but I had a space eater recently and > so have been paying closer attention to space reports than normal - which > is why I noticed the oddity that the filesystems in the pool are reported > as having more free space than the pool. > > Why is this ? Which is right ? > > From zpool iostat -v > capacity operations bandwidth > pool alloc free read write read write > ----------------- ----- ----- ----- ----- ----- ----- > archive 4.38T 2.87T 59 52 880K 1.14M > > One of the filesystems in the archive pool: > > ✓ steve@holdall ~ $ df -H /data > Filesystem Size Used Avail Capacity Mounted on > archive/data 7.0T 4.0T 3.0T 57% /data ZFS filesystems do not report space in a way that df expects and will result in it giving oddities like a size that appears to adjust as size=used+available and avail=realsize-. Space is further complicated with topics like snapshots, refreservation, compression, etc. zpool also shows space including reserved space that zfs and other tools would not see/report as free while also understanding pool redundancy that is outside the scope of the filesystem. I use `zfs list -ro space archive` a lot though customized output is easily available such as `zfs list -t snapshot -ro name,used -s used`. These keep you looking at what is expected of the filesystem instead of the pool and does so with more knowledge of how ZFS is working.