Make ZFS auto-destroy snapshots when the out of space?

[Kirk Strauser]

On May 30, 2010, at 3:09 PM, Matthew Dillon wrote:

>    It is actually a security issue to automatically destroy  
> snapshots based
>    on whether a filesystem is full, even automatically generated  
> snapshots.
>    Since one usually implements snapshots to perform a function you  
> wish
>    to rely on, such as to retain backups of historical data for  
> auditing
>    or other purposes, you do not want an attacker to be able to  
> indirectly
>    destroy snapshots simply by filling up the filesystem.
>
>    Instead what you want to do is to treat both the automatic and  
> the manual
>    snapshots as an integrated part of the filesystem's operation.   
> Just as
>    we have to deal with a nominal non-snapshotted filesystem-full  
> condition
>    today we also want to treat a filesystem with multiple snapshots  
> in the
>    same vein.  So, for example, you might administratively desire 60  
> 1-day
>    snapshots plus 10 minute snapshots for the most recent 3 days to be
>    retained at all times.  The automatic maintainance of the snapshots
>    would then administratively delete snapshots over 60 days old and  
> prune
>    to a coarser grain past 3 days.

First, I agree wholeheartedly with everything you've said. But in the  
interest of keeping the low-level stuff as simple as possible, how  
about a compromise: a cronjob or periodic script that regularly marks  
snapshots as  "expendable" based on user-configurable criteria. For  
example, you could say "snapshots_keep_hourly=72" and  
"snapshots_keep_monthly=6", and anything older than those would be  
marked as *eligible* for automatic deletion, but would not be  
*actually* destroyed unless necessary. Then you could meet both goals  
of clearing old data to make room for new content while establishing  
policies of how much you want to guarantee having on hand (to the  
extend that a filesystem snapshot is a guarantee).

>    The use of snapshots on modern filesystem capable of managing large
>    numbers of snapshots relatively pain-free, particularly on large  
> storage
>    systems and/or on modern multi-terrabyte HDs, requires a bit of a  
> change
>    in thinking.  You have to stop thinking of the snapshots as  
> optional and
>    start thinking of them as mandatory.

I'll grant you that. But first let's establish them as possible, even  
for people who don't want to actively monitor their drive space  
availability or actively, manually prune the old ones. Note: this is  
exactly what OS X's Time Machine does. If they can do it, and we have  
all the components in place to have something at least as nice  
ourselves, then I think we should go for it.

>    When snapshot availability is an assumed condition and not an
>    exceptional or special-case condition it opens up a whole new arena
>    in how filesystems can be managed, backed-up, audited, and used in
>    every-day work.  Once your thinking processes change you'll never
>    go back to non-snapshotted or nontrivially-snapshotted filesystems.

I couldn't agree more. I see it as programming with and without  
version control. Before you have reliable VC in your arsenal, changes  
are risky and difficult. What if you screw up? Will you remember  
everything that you need to revert to get back to a known-working  
state, or make periodic tarballs before you start on an experimental  
branch?

Once you have VC, you're so... liberated. If you try something and it  
doesn't work out, it's no big deal - you just go back to the last good  
version and try again another day. Knowing that, you're much more  
likely to try those grand experiments because the price of being wrong  
is drastically reduced.

To give a more pragmatic example, I wanted to try a KDE port from  
area51. I knew it was experimental, so I made a snapshot of /usr/local  
and /usr/ports, built the new version, and tested it. It wasn't fully  
working that day and I didn't have time to troubleshoot it, so I just  
rolled back to the snapshot I'd taken before starting. Voila! Back to  
a working system.

>    And you will certainly not want to allow a filesystem being  
> mistakenly
>    filled up to destroy your precious snapshots :-)


Very true, but I don't mind if it destroys the ones I told it that I  
don't care about anymore. :-)
-- 
Kirk Strauser