ZFS snapdir readability (Crosspost)

[Peter Eriksson]

Yes, and no. We use ‘refquotas’ on the user filesystems to prevent a user from using up all the free space in the zpool. 

However, what happens is that ZFS will decrease the “transaction size” of writes when a filesystem is nearly at quota in order to prevent a transaction to write more than the assigned quota (since it might write more due to compression/snapshots etc (or whatever)). Now this is (apparently) by design and it works. 

The effect is that a user trying to write to a nearly full (at quota) filesystem will “see” an extremely slow fileserver (from MB/s to KB/s (or B/s if _really_ full) which makes for interesting bug reports and problem diagnosing (since other users at the same time will see no such slow-downs).

_However_ - other effects are that creating snapshots, or destroying snapshots _also_ can take longer than usual. Or if you delete a lot of snapshots at the same time as a user has deleted a lot of files and you then reboot the server - then when it starts up again and attempts to mount all filesystems you will notice that it might take “forever” and it might look like “zfs mount -a” is “stuck”. Luckily for us when this happened we had previously modified the system startup scripts so all zfs mounting is done in the background and in parallell - so we got a login prompt and could login and run some commands (normally all filesystems are mounted first before allowing system logins….)

So we logged in and noticed that it was doing a lot of writes (zpool iostat) and was “stuck” at attempting to mount a single specific filesystem. A filesystem that happened to have 0 bytes free refquota. When I added some 10G more ‘refquota’ to that filesystem things speeded up a lot :-).

(First time this happened we let the server finish mounting by itself - which took about 2.5 days…)

- Peter

> On 8 Nov 2019, at 01:31, Chris Watson <bsdunix44 at gmail.com> wrote:
> 
> Peter, on your last point about 100% utilization, don’t you use quotas/user quotas to prevent that?
> 
> Chris
> 
> Sent from my iPhone
> 
>> On Nov 7, 2019, at 4:06 PM, Peter Eriksson <pen at lysator.liu.se> wrote:
>> 
>> The “easy” solution is to give each user (or group / project) their own ZFS filesystem. Then the “.zfs” directory would be inside the users own $HOME and you can set $HOME to 0700….
>> 
>> That is what we are doing. Granted it generates a “few” filesystems (like some 20000 per server (we have around 120k users), and then add hourly snapshots to each as “icing” on the cake). Mounting all those takes a bit of time - but luckily with the latest FreeBSD release things are much faster these days :-)
>> 
>> There are some other issues with that - like 100% full filesystems causing severe system slowdown during writes… So you really wanna have some monitoring system that warns for that.
>> 
>> - Peter
>> 
>> 
>>> 
>>> I recently noticed that all ZFS filesystems in FreeBSD allow access to
>>> the .zfs directory (snapdir) for all users of the system. It is
>>> possible to hide that directory using the snapdir option:
>> 
>> 
>> _______________________________________________
>> freebsd-fs at freebsd.org mailing list
>> https://lists.freebsd.org/mailman/listinfo/freebsd-fs
>> To unsubscribe, send any mail to "freebsd-fs-unsubscribe at freebsd.org"