zfs raidz overhead

[Eric A. Borisch]

On Wed, Feb 22, 2017 at 3:50 PM, Wiktor Niesiobedzki <bsd at vink.pl> wrote:

> I can add to this, that this is not only seen on raidz, but also on
> mirror pools, such as this:
> # zpool status tank
>   pool: tank
>  state: ONLINE
>   scan: scrub repaired 0 in 3h22m with 0 errors on Thu Feb  9 06:47:07 2017
> config:
>
>         NAME               STATE     READ WRITE CKSUM
>         tank               ONLINE       0     0     0
>           mirror-0         ONLINE       0     0     0
>             gpt/tank1.eli  ONLINE       0     0     0
>             gpt/tank2.eli  ONLINE       0     0     0
>
> errors: No known data errors
>
>
> When I createted test zvols:
> # zfs create -V10gb -o volblocksize=8k tank/tst-8k
> # zfs create -V10gb -o volblocksize=16k tank/tst-16k
> # zfs create -V10gb -o volblocksize=32k tank/tst-32k
> # zfs create -V10gb -o volblocksize=64k tank/tst-64k
> # zfs create -V10gb -o volblocksize=128k tank/tst-128k
>
> # zfs get used tank/tst-8k
> NAME         PROPERTY  VALUE  SOURCE
> tank/tst-8k  used      10.3G  -
> root at kadlubek:~ # zfs get used tank/tst-16k
> NAME          PROPERTY  VALUE  SOURCE
> tank/tst-16k  used      10.2G  -
> root at kadlubek:~ # zfs get used tank/tst-32k
> NAME          PROPERTY  VALUE  SOURCE
> tank/tst-32k  used      10.1G  -
> root at kadlubek:~ # zfs get used tank/tst-64k
> NAME          PROPERTY  VALUE  SOURCE
> tank/tst-64k  used      10.0G  -
> root at kadlubek:~ # zfs get used tank/tst-128k
> NAME           PROPERTY  VALUE  SOURCE
> tank/tst-128k  used      10.0G  -


Nope, that all looks correct. There is space reserved for metadata
(checksum for example) when you create a zvol. Since checksums are computed
by block, it makes sense that more metadata is required for the 8k
volblocksize.


> I also noted, that snapshots impact used stats far much, than

usedbysnapshot value:
> zfs get volsize,used,referenced,compressratio,volblocksize,
> usedbysnapshots,usedbydataset,usedbychildren
> tank/dkr-thinpool
> NAME               PROPERTY         VALUE      SOURCE
> tank/dkr-thinpool  volsize          10G        local
> tank/dkr-thinpool  used             12.0G      -
> tank/dkr-thinpool  referenced       1.87G      -
> tank/dkr-thinpool  compressratio    1.91x      -
> tank/dkr-thinpool  volblocksize     64K        -
> tank/dkr-thinpool  usedbysnapshots  90.4M      -
> tank/dkr-thinpool  usedbydataset    1.87G      -
> tank/dkr-thinpool  usedbychildren   0          -
>
>
> On a 10G volume, filled with 2G of data, and 90M used by snapshosts,
> used is 2G. When I destroy the snapshots, used will drop to 10.0G.
>

It makes sense when you just remember that 'used' is enforcing a contract
to the user that 10G of new data can be written to the zvol
(refreservation, I'm sure it's set to 10G here), and still have all of the
current snapshots will still be available.

Breaking that down, usedbysnapshots refers to the data that is no longer
referenced by the dataset (overwritten, in the case of a zvol.) Used is the
space the the dataset is consuming (i.e. not available for other datasets).
This means that 10G (refreservation)+ the referenced property of your
earliest snapshot + the written value of all but your latest snapshot + the
aforementioned metadata overhead needs to be reserved, or 'used' by the
dataset. (I think I got that calculation right.)

So even though usedbysnapshots is 90M (90M of blocks have been
overwritten), the referenced and written properties of the snapshot
accumulate to the space you're seeing here. Once all the snapshots are
gone, used drops to (almost) the volsize.

Hope that helps. If you want thin provisioning, you can unset the
refreservation, or just create the volume with the '-s' option.

 - Eric