ZFS bootfs vs vfs.root.mountfrom

From: Peter Jeremy <peterj_at_freebsd.org>
Date: Sat, 27 Aug 2022 04:01:09 UTC
When booting from ZFS was first introduced, the recommended approach
was to configure the root filesystem via vfs.root.mountfrom.  See (e.g.)
 https://lists.freebsd.org/pipermail/freebsd-fs/2011-September/012482.html
 https://lists.freebsd.org/pipermail/svn-src-head/2011-October/030641.html
and people wanted to change that to use the zpool bootfs property - e.g.
 https://lists.freebsd.org/pipermail/freebsd-current/2009-October/012933.html
 https://lists.freebsd.org/pipermail/freebsd-fs/2010-March/008010.html
resulting in it becoming optional in SVN r235330 in May 2012:
 https://lists.freebsd.org/pipermail/svn-src-head/2012-May/036902.html

The recommendation to use vfs.root.mountfrom remains in parts of the
wiki (e.g. https://wiki.freebsd.org/RootOnZFS/GPTZFSBoot/Mirror step
2.6) and on mailing list posts (e.g. the thread starting at
https://lists.freebsd.org/pipermail/freebsd-fs/2020-July/028351.html).
The loader(8) and loader.conf(5) man pages state that the root
filesystem is set by vfs.root.mountfrom, with a default derived from
/etc/fstab or, for ZFS only, currdev (the boot filesystem).

OTOH, https://wiki.freebsd.org/BootEnvironments states "/etc/fstab
must be purged of any / mount, and /boot/loader.conf must not be
setting vfs.root.mountfrom directly."  I can't find any other
reference to this requirement.

As for the BootEnvironment management tools themselves:
* Neither beadm(8) nor bectl(8) mention vfs.root.mountfrom.
* beadm(8) refers to https://forums.freebsd.org/showthread.php?t=31662,
  which recommends setting vfs.root.mountfrom
* beadm(8) will update vfs.root.mountfrom in /boot/loader.conf if it
  exists but will not add it if it doesn't exist.
* bectl(8) ignores /boot/loader.conf

The last point means that switching an existing system from beadm to
bectl is very likely to result in the system mounting the wrong root
filesystem: An existing system is likely to have vfs.root.mountfrom
set, since that used to be the only approach and is still widely
recommended, whilst bectl ignores it and will therefore leave the
old vfs.root.mountfrom value present.

IMO, if setting vfs.root.mountfrom in /boot/loader.conf is no longer
supported for ZFS, we need to do a better job of documenting that and
removing documentation to the contrary.  I also believe that bectl
should at least warn if /boot/loader.conf sets vfs.root.mountfrom.

-- 
Peter Jeremy