Re: ZFS Root size keeps going down after upgrade to 13.2-release

From: Kaya Saman <kayasaman_at_optiplex-networks.com>
Date: Mon, 21 Aug 2023 14:25:25 UTC
On 8/21/23 15:15, freebsd@vanderzwan.org wrote:
>
>> On 21 Aug 2023, at 15:52, Kaya Saman <kayasaman@optiplex-networks.com> wrote:
>>
>>
>> On 8/21/23 14:49, freebsd@vanderzwan.org wrote:
>>>> On 21 Aug 2023, at 15:41, Kaya Saman <kayasaman@optiplex-networks.com> wrote:
>>>>
>>>>
>>>> On 8/21/23 14:38, freebsd@vanderzwan.org wrote:
>>>>>> On 21 Aug 2023, at 15:34, Kaya Saman <kayasaman@optiplex-networks.com> wrote:
>>>>>>
>>>>>>
>>>>>> On 8/21/23 14:23, freebsd@vanderzwan.org wrote:
>>>>>>>> On 21 Aug 2023, at 15:14, Kaya Saman <kayasaman@optiplex-networks.com> wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>> On 8/21/23 14:07, freebsd@vanderzwan.org wrote:
>>>>>>>>>> On 21 Aug 2023, at 14:21, Kaya Saman <kayasaman@optiplex-networks.com> wrote:
>>>>>>>>>> # zfs list -ro space zroot/ROOT
>>>>>>>>>> NAME                                          AVAIL   USED USEDSNAP  USEDDS  USEDREFRESERV  USEDCHILD
>>>>>>>>>> zroot/ROOT                                     720M  88.5G 0B     88K             0B      88.5G
>>>>>>>>>> zroot/ROOT/13.1-RELEASE-p5_2023-08-20_220010   720M     8K 0B      8K             0B         0B
>>>>>>>>>> zroot/ROOT/13.2-RELEASE-p2_2023-08-20_224534   720M     8K 0B      8K             0B         0B
>>>>>>>>>> zroot/ROOT/default                             720M  88.5G 52.0G   36.5G             0B         0B
>>>>>>>>>>
>>>>>>>>> That last line seems to indicate there is 52.0 G used in snapshots of zroot/ROOT/default and 36.5G in the dataset itself.
>>>>>>>>> Output of zfs list -o  space -t snapshot |grep zroot/ROOT/default should give a list of the snapshots and the sizes.
>>>>>>>>>
>>>>>>>>> 	Paul
>>>>>>>>>
>>>>>>>>>
>>>>>>>> Thanks so much Paul.... I feel like I'm spinning around in circles currently...
>>>>>>>>
>>>>>>>>
>>>>>>>> There doesn't seem to be anything indicative of that space being used up?
>>>>>>>>
>>>>>>>>
>>>>>>>>   zfs list -o  space -t snapshot |grep zroot/ROOT/defaul
>>>>>>>> zroot/ROOT/default@2022-11-28-06:08:26-0      -   583M -       -              -          -
>>>>>>>> zroot/ROOT/default@2022-11-28-06:38:48-0      -  11.4M -       -              -          -
>>>>>>>> zroot/ROOT/default@2022-11-28-07:26:20-0      -  9.41M -       -              -          -
>>>>>>>> zroot/ROOT/default@2023-08-20-22:00:10-0      -  34.5M -       -              -          -
>>>>>>>> zroot/ROOT/default@2023-08-20-22:45:34-0      -  9.56M -       -              -          -
>>>>>>>>
>>>>>>>>
>>>>>>>> certainly not 52GB?
>>>>>>>>
>>>>>>>>
>>>>>>> That’s definitely confusing.
>>>>>>> What’s the full output of 'zfs list -o space -r zroot’  and of 'zpool list  -v zroot’
>>>>>>>
>>>>>>> 	Paul
>>>>>>>
>>>>>>>
>>>>>> Here are the outputs:
>>>>>>
>>>>>>
>>>>>> zfs list -o space -r zroot
>>>>>> NAME                                          AVAIL   USED USEDSNAP  USEDDS  USEDREFRESERV  USEDCHILD
>>>>>> zroot                                          720M   107G 0B   2.20G             0B       105G
>>>>>> zroot/ROOT                                     720M  88.5G 0B     88K             0B      88.5G
>>>>>> zroot/ROOT/13.1-RELEASE-p5_2023-08-20_220010   720M     8K 0B      8K             0B         0B
>>>>>> zroot/ROOT/13.2-RELEASE-p2_2023-08-20_224534   720M     8K 0B      8K             0B         0B
>>>>>> zroot/ROOT/default                             720M  88.5G 52.0G   36.5G             0B         0B
>>>>>> zroot/swap                                    3.78G  6.19G 0B   3.11G          3.08G         0B
>>>>>> zroot/tmp                                      720M   121M 0B    121M             0B         0B
>>>>>> zroot/usr                                      720M  7.09G 0B     88K             0B      7.09G
>>>>>> zroot/usr/home                                 720M    88K 0B     88K             0B         0B
>>>>>> zroot/usr/obj                                  720M  4.75G 0B   4.75G             0B         0B
>>>>>> zroot/usr/ports                                720M   432K 0B    176K             0B       256K
>>>>>> zroot/usr/ports/distfiles                      720M   168K 0B    168K             0B         0B
>>>>>> zroot/usr/ports/packages                       720M    88K 0B     88K             0B         0B
>>>>>> zroot/usr/src                                  720M  2.35G 0B   2.35G             0B         0B
>>>>>> zroot/var                                      720M  2.60G 0B     88K             0B      2.60G
>>>>>> zroot/var/audit                                720M    88K 0B     88K             0B         0B
>>>>>> zroot/var/backups                              720M   141M 0B    141M             0B         0B
>>>>>> zroot/var/crash                                720M    88K 0B     88K             0B         0B
>>>>>> zroot/var/db                                   720M  1.69G 0B   1.59G             0B      94.8M
>>>>>> zroot/var/db/pkg                               720M  94.8M 0B   94.8M             0B         0B
>>>>>> zroot/var/empty                                720M    88K 0B     88K             0B         0B
>>>>>> zroot/var/log                                  720M   792M 0B    792M             0B         0B
>>>>>> zroot/var/mail                                 720M  3.96M 0B   3.96M             0B         0B
>>>>>> zroot/var/run                                  720M   248K 0B    248K             0B         0B
>>>>>> zroot/var/tmp                                  720M   672K 0B    672K             0B         0B
>>>>>>
>>>>>>
>>>>>>   zpool list  -v zroot
>>>>>> NAME         SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP DEDUP    HEALTH  ALTROOT
>>>>>> zroot        111G   104G  7.22G        -         -    82%    93% 1.00x    ONLINE  -
>>>>>>    mirror-0   111G   104G  7.22G        -         -    82% 93.5%      -    ONLINE
>>>>>>      ada0p3   111G      -      -        -         -      - -      -    ONLINE
>>>>>>      ada1p3   111G      -      -        -         -      - -      -    ONLINE
>>>>>>
>>>>> Those look quite normal. What about 'zfs list -r -t all -o space zroot/ROOT/default’  ?
>>>>>
>>>>> 	Paul
>>>>>
>>>>>
>>>>>
>>>> zfs list -r -t all -o space zroot/ROOT/default
>>>> NAME                                      AVAIL   USED  USEDSNAP USEDDS  USEDREFRESERV  USEDCHILD
>>>> zroot/ROOT/default                         720M  88.5G     52.0G 36.5G             0B         0B
>>>> zroot/ROOT/default@2022-11-28-06:08:26-0      -   583M -       -              -          -
>>>> zroot/ROOT/default@2022-11-28-06:38:48-0      -  11.4M -       -              -          -
>>>> zroot/ROOT/default@2022-11-28-07:26:20-0      -  9.41M -       -              -          -
>>>> zroot/ROOT/default@2023-08-20-22:00:10-0      -  34.5M -       -              -          -
>>>> zroot/ROOT/default@2023-08-20-22:45:34-0      -  9.56M -       -              -          -
>>>>
>>>>
>>>> Strange! It shows a 52GB 'claimed' snapshot but where is it?
>>>>
>>>>
>>> Can you give output of 'zfs get all zroot/ROOT/default’ as well ? Maybe that gives a hint.
>>>
>>> 	Paul
>>>
>> Sure:
>>
>>
>> zfs get all zroot/ROOT/default
>> NAME                PROPERTY              VALUE SOURCE
>> zroot/ROOT/default  type                  filesystem             -
>> zroot/ROOT/default  creation              Tue Aug  7  2:53 2018  -
>> zroot/ROOT/default  used                  88.5G                  -
>> zroot/ROOT/default  available             720M                   -
>> zroot/ROOT/default  referenced            36.5G                  -
>> zroot/ROOT/default  compressratio         1.29x                  -
>> zroot/ROOT/default  mounted               yes                    -
>> zroot/ROOT/default  quota                 none default
>> zroot/ROOT/default  reservation           none default
>> zroot/ROOT/default  recordsize            128K default
>> zroot/ROOT/default  mountpoint            / local
>> zroot/ROOT/default  sharenfs              off default
>> zroot/ROOT/default  checksum              on default
>> zroot/ROOT/default  compression           lz4 inherited from zroot
>> zroot/ROOT/default  atime                 on default
>> zroot/ROOT/default  devices               on default
>> zroot/ROOT/default  exec                  on default
>> zroot/ROOT/default  setuid                on default
>> zroot/ROOT/default  readonly              off default
>> zroot/ROOT/default  jailed                off default
>> zroot/ROOT/default  snapdir               hidden default
>> zroot/ROOT/default  aclmode               discard default
>> zroot/ROOT/default  aclinherit            restricted default
>> zroot/ROOT/default  createtxg             93                     -
>> zroot/ROOT/default  canmount              noauto local
>> zroot/ROOT/default  xattr                 on default
>> zroot/ROOT/default  copies                1 default
>> zroot/ROOT/default  version               5                      -
>> zroot/ROOT/default  utf8only              off                    -
>> zroot/ROOT/default  normalization         none                   -
>> zroot/ROOT/default  casesensitivity       sensitive              -
>> zroot/ROOT/default  vscan                 off default
>> zroot/ROOT/default  nbmand                off default
>> zroot/ROOT/default  sharesmb              off default
>> zroot/ROOT/default  refquota              none default
>> zroot/ROOT/default  refreservation        none default
>> zroot/ROOT/default  guid                  15509736917917171623   -
>> zroot/ROOT/default  primarycache          all default
>> zroot/ROOT/default  secondarycache        all default
>> zroot/ROOT/default  usedbysnapshots       52.0G                  -
>> zroot/ROOT/default  usedbydataset         36.5G                  -
>> zroot/ROOT/default  usedbychildren        0B                     -
>> zroot/ROOT/default  usedbyrefreservation  0B                     -
>> zroot/ROOT/default  logbias               latency default
>> zroot/ROOT/default  objsetid              58                     -
>> zroot/ROOT/default  dedup                 off default
>> zroot/ROOT/default  mlslabel              none default
>> zroot/ROOT/default  sync                  standard default
>> zroot/ROOT/default  dnodesize             legacy default
>> zroot/ROOT/default  refcompressratio      1.32x                  -
>> zroot/ROOT/default  written               1.53G                  -
>> zroot/ROOT/default  logicalused           108G                   -
>> zroot/ROOT/default  logicalreferenced     46.0G                  -
>> zroot/ROOT/default  volmode               default default
>> zroot/ROOT/default  filesystem_limit      none default
>> zroot/ROOT/default  snapshot_limit        none default
>> zroot/ROOT/default  filesystem_count      none default
>> zroot/ROOT/default  snapshot_count        none default
>> zroot/ROOT/default  snapdev               hidden default
>> zroot/ROOT/default  acltype               nfsv4 default
>> zroot/ROOT/default  context               none default
>> zroot/ROOT/default  fscontext             none default
>> zroot/ROOT/default  defcontext            none default
>> zroot/ROOT/default  rootcontext           none default
>> zroot/ROOT/default  relatime              off default
>> zroot/ROOT/default  redundant_metadata    all default
>> zroot/ROOT/default  overlay               on default
>> zroot/ROOT/default  encryption            off default
>> zroot/ROOT/default  keylocation           none default
>> zroot/ROOT/default  keyformat             none default
>> zroot/ROOT/default  pbkdf2iters           0 default
>> zroot/ROOT/default  special_small_blocks  0 default
>>
>>
> That looks consistent with the rest of the data.
>
> Man page for zfsprops mentions this for snapshots;
>                             The used space of a snapshot (see the Snapshots
>                             section of zfsconcepts(7)) is space that is
>                             referenced exclusively by this snapshot.  If this
>                             snapshot is destroyed, the amount of used space
>                             will be freed.  Space that is shared by multiple
>                             snapshots isn't accounted for in this metric.  When
>                             a snapshot is destroyed, space that was previously
>                             shared with this snapshot can become unique to
>                             snapshots adjacent to it, thus changing the used
>                             space of those snapshots.  The used space of the
>                             latest snapshot can also be affected by changes in
>                             the file system.  Note that the used space of a
>                             snapshot is a subset of the written space of the
>                             snapshot.
> Note : Space that is shared by multiple  snapshots isn't accounted for in this metric.
>
> My theory is that the snapshots/boot environments share about  52G of with each other but not with the parent dataset.
> As there is no data unique to one snapshot it does not show
> What you could do is delete all except one bootenvironment.
> The snapshot data used should then be unique to that last snapshot and  appear in the lists.
>
> 	Paul
>
>
>

I just responded to Olivier above but I have managed to free up the 
necessary space by deleting all the snapshots:


df -h
Filesystem                              Size    Used   Avail Capacity  
Mounted on
zroot/ROOT/default                       89G     36G     53G 41%    /


For some reason some of the snapshots were automatically 'cloned' and 
exactly as you suggested above there was a fair amount of space sharing 
between snapshots. This of course resulted in data being removed from 
the main snapshot but still remained on disk due another snapshot 
referencing that data....

Very confusing for an automated process but anyway, I have space again :-)


Thank you so much everyone for your help and input!


Best Regards,


Kaya