From nobody Sun Mar 03 12:53:03 2024 X-Original-To: questions@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TnhZj0nPXz5ClBY for ; Sun, 3 Mar 2024 12:53:17 +0000 (UTC) (envelope-from 4250.82.1d52800017a01b6.7e2a1dadfc7663ac5cf0881d6e2b1141@email-od.com) Received: from s1-b0c6.socketlabs.email-od.com (s1-b0c6.socketlabs.email-od.com [142.0.176.198]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4TnhZh3KSVz3x1M for ; Sun, 3 Mar 2024 12:53:16 +0000 (UTC) (envelope-from 4250.82.1d52800017a01b6.7e2a1dadfc7663ac5cf0881d6e2b1141@email-od.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=email-od.com header.s=dkim header.b=F4xKOeoX; dmarc=none; spf=pass (mx1.freebsd.org: domain of 4250.82.1d52800017a01b6.7e2a1dadfc7663ac5cf0881d6e2b1141@email-od.com designates 142.0.176.198 as permitted sender) smtp.mailfrom=4250.82.1d52800017a01b6.7e2a1dadfc7663ac5cf0881d6e2b1141@email-od.com DKIM-Signature: v=1; a=rsa-sha256; d=email-od.com;i=@email-od.com;s=dkim; c=relaxed/relaxed; q=dns/txt; t=1709470396; x=1712062396; h=content-transfer-encoding:content-type:mime-version:references:in-reply-to:message-id:subject:to:from:date:x-thread-info:subject:to:from:cc:reply-to; bh=6z3jlAgtIlwhayoaRwQ6me90Ie2+aRRXE7/ehHqugw0=; b=F4xKOeoXVqIICgL5mKZV2HlqkcopeeVWkNANeaLdeMruhShRsWIga/0lTPZB2r1D9J33C8O0OhoPK7wZUkxsJ5wgCAexwO6nanAtpKcFvdrvGutT1lvkmyhenc4KplsJkFDHrJw1aYqjZZhUkVUUE/URwc2MuveHzfnEBPexsYI= X-Thread-Info: NDI1MC4xMi4xZDUyODAwMDE3YTAxYjYucXVlc3Rpb25zPWZyZWVic2Qub3Jn Received: from r3.h.in.socketlabs.com (r3.h.in.socketlabs.com [142.0.180.13]) by mxsg2.email-od.com with ESMTP(version=Tls12 cipher=Aes256 bits=256); Sun, 3 Mar 2024 07:53:05 -0500 Received: from smtp.lan.sohara.org (EMTPY [185.202.17.215]) by r3.h.in.socketlabs.com with ESMTP(version=Tls12 cipher=Aes256 bits=256); Sun, 3 Mar 2024 07:53:05 -0500 Received: from [192.168.63.1] (helo=steve.lan.sohara.org) by smtp.lan.sohara.org with smtp (Exim 4.95 (FreeBSD)) (envelope-from ) id 1rglKx-00017w-Gw for questions@freebsd.org; Sun, 03 Mar 2024 12:53:03 +0000 Date: Sun, 3 Mar 2024 12:53:03 +0000 From: Steve O'Hara-Smith To: questions@freebsd.org Subject: Re: ZFS pool/filesystem space oddity Message-Id: <20240303125303.98948d590354735d30f89d9a@sohara.org> In-Reply-To: References: <20240303115859.922d34bb2f47b6df9093315a@sohara.org> X-Mailer: Sylpheed 3.7.0 (GTK+ 2.24.33; amd64-portbld-freebsd13.1) X-Clacks-Overhead: "GNU Terry Pratchett" List-Id: User questions List-Archive: https://lists.freebsd.org/archives/freebsd-questions List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-questions@freebsd.org X-BeenThere: freebsd-questions@freebsd.org Mime-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Spamd-Bar: -- X-Spamd-Result: default: False [-2.65 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.95)[-0.952]; MV_CASE(0.50)[]; FORGED_SENDER(0.30)[steve@sohara.org,4250.82.1d52800017a01b6.7e2a1dadfc7663ac5cf0881d6e2b1141@email-od.com]; R_DKIM_ALLOW(-0.20)[email-od.com:s=dkim]; R_SPF_ALLOW(-0.20)[+ip4:142.0.176.0/20]; MIME_GOOD(-0.10)[text/plain]; ARC_NA(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:7381, ipnet:142.0.176.0/22, country:US]; DMARC_NA(0.00)[sohara.org]; RCVD_COUNT_THREE(0.00)[3]; DWL_DNSWL_NONE(0.00)[email-od.com:dkim]; FROM_NEQ_ENVFROM(0.00)[steve@sohara.org,4250.82.1d52800017a01b6.7e2a1dadfc7663ac5cf0881d6e2b1141@email-od.com]; FROM_HAS_DN(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; TO_DN_NONE(0.00)[]; RCVD_TLS_LAST(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MLMMJ_DEST(0.00)[questions@freebsd.org]; DKIM_TRACE(0.00)[email-od.com:+] X-Rspamd-Queue-Id: 4TnhZh3KSVz3x1M On Sun, 3 Mar 2024 05:33:02 -0700 "Edward Sanford Sutton, III" wrote: > On 3/3/24 04:58, Steve O'Hara-Smith wrote: > > Hi, > > > > My NAS runs a ZFS pool called archive striped over two mirrored > > pairs of 4TB drives. It all works fine but I had a space eater recently > > and so have been paying closer attention to space reports than normal - > > which is why I noticed the oddity that the filesystems in the pool are > > reported as having more free space than the pool. > > > > Why is this ? Which is right ? > > > > From zpool iostat -v > > capacity operations bandwidth > > pool alloc free read write read write > > ----------------- ----- ----- ----- ----- ----- ----- > > archive 4.38T 2.87T 59 52 880K 1.14M > > > > One of the filesystems in the archive pool: > > > > ✓ steve@holdall ~ $ df -H /data > > Filesystem Size Used Avail Capacity Mounted on > > archive/data 7.0T 4.0T 3.0T 57% /data > > ZFS filesystems do not report space in a way that df expects and will > result in it giving oddities like a size that appears to adjust as > size=used+available and avail=realsize-. Space is further complicated > with topics like snapshots, refreservation, compression, etc. zpool also > shows space including reserved space that zfs and other tools would not > see/report as free while also understanding pool redundancy that is > outside the scope of the filesystem. All of these would lead me to expect zpool to show more space than df rather than the other way round - hence my surprise. -- Steve O'Hara-Smith