Re: ZFS bootfs vs vfs.root.mountfrom

On 27/08/2022 05:15, Warner Losh wrote:
> 
> 
> On Fri, Aug 26, 2022 at 10:01 PM Peter Jeremy <peterj@freebsd.org 
> <mailto:peterj@freebsd.org>> wrote:
> 
>     When booting from ZFS was first introduced, the recommended approach
>     was to configure the root filesystem via vfs.root.mountfrom.  See (e.g.)
>     https://lists.freebsd.org/pipermail/freebsd-fs/2011-September/012482.html <https://lists.freebsd.org/pipermail/freebsd-fs/2011-September/012482.html>
>     https://lists.freebsd.org/pipermail/svn-src-head/2011-October/030641.html <https://lists.freebsd.org/pipermail/svn-src-head/2011-October/030641.html>
>     and people wanted to change that to use the zpool bootfs property - e.g.
>     https://lists.freebsd.org/pipermail/freebsd-current/2009-October/012933.html <https://lists.freebsd.org/pipermail/freebsd-current/2009-October/012933.html>
>     https://lists.freebsd.org/pipermail/freebsd-fs/2010-March/008010.html <https://lists.freebsd.org/pipermail/freebsd-fs/2010-March/008010.html>
>     resulting in it becoming optional in SVN r235330 in May 2012:
>     https://lists.freebsd.org/pipermail/svn-src-head/2012-May/036902.html <https://lists.freebsd.org/pipermail/svn-src-head/2012-May/036902.html>
> 
> 
> Yes. This was a total hack at the time. The boot loader normally sets 
> this variable
> w/o users intervening and this hack propagated into loader.conf files, 
> which has
> always been a bad idea.
> 
>     The recommendation to use vfs.root.mountfrom remains in parts of the
>     wiki (e.g. https://wiki.freebsd.org/RootOnZFS/GPTZFSBoot/Mirror
>     <https://wiki.freebsd.org/RootOnZFS/GPTZFSBoot/Mirror> step
>     2.6) and on mailing list posts (e.g. the thread starting at
>     https://lists.freebsd.org/pipermail/freebsd-fs/2020-July/028351.html
>     <https://lists.freebsd.org/pipermail/freebsd-fs/2020-July/028351.html>).
>     The loader(8) and loader.conf(5) man pages state that the root
>     filesystem is set by vfs.root.mountfrom, with a default derived from
>     /etc/fstab or, for ZFS only, currdev (the boot filesystem).
> 
> 
> Yes. That's how it is supposed to work. The boot loader sets this based on
> where it loaded the kernel. This variable should *NEVER* be set in a
> loader.conf file unless you have a 'very special snowflake' system that
> can't determine it automatically. ZFS root isn't special.
> 
>     OTOH, https://wiki.freebsd.org/BootEnvironments
>     <https://wiki.freebsd.org/BootEnvironments> states "/etc/fstab
>     must be purged of any / mount, and /boot/loader.conf must not be
>     setting vfs.root.mountfrom directly."  I can't find any other
>     reference to this requirement.
> 
> 
> /etc/fstab isn't used for zfs root. I believe that BootEnvironments is 100%
> right here.
> 
>     As for the BootEnvironment management tools themselves:
>     * Neither beadm(8) nor bectl(8) mention vfs.root.mountfrom.
> 
> 
> This is correct.
> 
>     * beadm(8) refers to
>     https://forums.freebsd.org/showthread.php?t=31662
>     <https://forums.freebsd.org/showthread.php?t=31662>,
>        which recommends setting vfs.root.mountfrom
> 
> 
> This is bad advice.
> 
>     * beadm(8) will update vfs.root.mountfrom in /boot/loader.conf if it
>        exists but will not add it if it doesn't exist.
> 
> 
> This is terrible. It's part of the continuing papering over of bugs that 
> have
> long ago been fixed and should, at most, warn that's its set and that it
> really shouldn't be.
> 
>     * bectl(8) ignores /boot/loader.conf
> 
> 
> This is proper behavior.
> 
>     The last point means that switching an existing system from beadm to
>     bectl is very likely to result in the system mounting the wrong root
>     filesystem: An existing system is likely to have vfs.root.mountfrom
>     set, since that used to be the only approach and is still widely
>     recommended, whilst bectl ignores it and will therefore leave the
>     old vfs.root.mountfrom value present.
> 
> 
> Yes. bectl likely should, at most, warn that it's set in loader.conf as an
> antifootshooting measure, but only due to the all-too-long history.
> 
>     IMO, if setting vfs.root.mountfrom in /boot/loader.conf is no longer
>     supported for ZFS, we need to do a better job of documenting that and
>     removing documentation to the contrary.  I also believe that bectl
>     should at least warn if /boot/loader.conf sets vfs.root.mountfrom.
> 
> 
> It really never had been the proper way. One should almost never do
> it today, except in the above 'special snowflake' systems that still crop up
> from time to time (we have one at work since we still have some systems
> that use gmirror roots, and the boot loader doesn't set it right for 
> them since
> we can't for logistical reasons set /etc/fstab, but that's not ZFS 
> related). We
> should actively militate against it with extreme prejudice, though, for 
> routine
> uses. We should rip out all references in the docs related to ZFS. We should
> warn if we detect it today. I agree completely with what you are saying 
> here.
> I would be extraordinarily surprised to find anybody with a contrary 
> opinion.

Isn't a legitimate use-case where /boot and / are on different 
devices/pools? That's what I'm using it for. bootfs property says where 
to find the kernel, and the accompanying vfs.root.mountfrom in 
loader.conf says where the matching rootfs is (in my case on a different 
pool).

> 
> Now, who is going to do the work? :)
> 
> Warner