[Bug 274263] Access to zfs snapshots within a jail return EPERM after a while of operation

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=274263

            Bug ID: 274263
           Summary: Access to zfs snapshots within a jail return EPERM
                    after a while of operation
           Product: Base System
           Version: 13.2-RELEASE
          Hardware: Any
                OS: Any
            Status: New
          Severity: Affects Only Me
          Priority: ---
         Component: kern
          Assignee: bugs@FreeBSD.org
          Reporter: freebsd-bugs@virtualtec.ch

Overview:
- host-system with jails, that run hosting environments (webs etc)
- jails run in their own zfs datasets, but filesystems are managed by the host,
not by the jails
- daily zfs snapshots provide jail content of the previous days
- customers are taught to, and use these snapshots for easy access to previous
content
- some time after a reboot (I'm not aware of the specific trigger), access to
/.zfs/snapshot/{SPECIFICSNAP}
  from within the jail fails with EPERM, no matter whether the user is
unprivileged or root. Listing of
  the snapshot directory itself (/.zfs/snapshot) works fine.
- I found that if I list that snapshot directory from the host system (even as
unprivileged user), access
  from within the jail works fine again (for a while)

Specifics (possibly too verbose, sorry):
- current host system: 13.1-RELEASE-p6
- sample zfs layout:
  tank/services/jails/reseller123
  tank/services/jails/reseller123/chrimg/someweb.com
  tank/services/jails/reseller123/chrimg/someweb.com/tmp
- corresponding zfs filesystem mountpoints:
  /services/jails/reseller123
  /services/jails/reseller123/services/webs/someweb.com/system
  /services/jails/reseller123/services/webs/someweb.com/system/tmp
- we clone specific php-version based template filesystems to a system
directory within a web root to implement different php environments.
  these clones are all read-only
- there's an additional devfs and a nullfs mount per web:
  devfs on /services/jails/reseller123/services/webs/someweb.com/dev
  /services/addons nullfs read-only to
/services/jails/reseller123/services/webs/someweb.com/system/services/addons
  the nullfs mount grants access to common typo3 version templates the customer
uses within webs
- customer accesses /.zfs/snapshot via a symlink we provide in
/services/webs/backup, but the observed behavior
  happens via direct access as well as via symlink, so I don't think this is
relevant

In the failure case, I get this:
$ ls /services/webs/backup/2023-10-04-00:00:04
ls: /services/webs/backup/2023-10-04-00:00:04: Operation not permitted

or with ktrace:
...
 34393 ls       CALL  fstatat(AT_FDCWD,0x8006df098,0x7fffffffe2b0,0)
 34393 ls       NAMI  "/services/webs/backup/2023-10-04-00:00:04"
 34393 ls       RET   fstatat -1 errno 1 Operation not permitted
 34393 ls       CALL 
fstatat(AT_FDCWD,0x8006df098,0x7fffffffe2b0,0x200<AT_SYMLINK_NOFOLLOW>)
 34393 ls       NAMI  "/services/webs/backup/2023-10-04-00:00:04"
 34393 ls       RET   fstatat -1 errno 1 Operation not permitted


Now, if I do the same ls on the host system (also as unprivileged user), it
works fine. And as soon as I've
done that, the jail access works again.

Could there be permission issues with nullfs in jails, such as in this setup? 
Unfortunately, I've so far not found a way to forcibly cause the failure.

-- 
You are receiving this mail because:
You are the assignee for the bug.