Snapdir of Nullfs-mounted Dataset incomplete in Jail

From: Matthias Petermann <mp_at_petermann-it.de>
Date: Thu, 16 Nov 2023 14:22:20 UTC
Hello everyone,

I have just made an observation regarding the interaction of ZFS 
snapshots, Jails, and Nullfs mounts.

I have a FreeBSD 13.2-p5 system with a ZFS Root Filesystem. I am using 
Bastille to manage Jails. There is a Jail whose root filesystem is 
mounted from the dataset zroot/bastille/jails/dc/root to 
/usr/local/bastille/jails/dc/root.

Furthermore, there is a dataset on the host named zroot/data/shares. I 
mount this dataset to the Jail's root filesystem using a Nullfs mount at 
/usr/local/bastille/jails/dc/root/shares.

This works so far – I can access /shares within the Jail and effectively 
use the underlying dataset from both the host and the Jail.

Now, I wanted to access the snapshots of zroot/data/shares from within 
the Jail. So, I quickly set the Snapdir property to visible. Within the 
Jail, I can now list the existing snapshots via /shares/.zfs/snapshot.

Here's the problem: Snapshots are continuously created by Sanoid from 
the host. Over time, I now get different directory listings of snapdir 
depending on where I list it from: when listing from the host at 
/zroot/data/shares/.zfs/snapshot, everything is complete. In the Jail at 
/shares/.zfs/snapshot, the newer snapshots are missing.

If I then go to the host and execute the following:

```
% doas ls -lah /zroot/data/shares/.zfs/snapshot/*
```

...I get all the snapshots with sub-directories listed as expected.

Surprisingly, after that, I can also see the snapshots from within the Jail.

A similar issue is discussed in [1] and [2].

Where is my mistake in thinking?

Best regards,
Matthias

[1] 
https://forums.freebsd.org/threads/zfs-snapshot-directory-access-operation-not-permitted-in-jail-on-freebsd13.80713/

[2] https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=260160