Re: ZFS bootfs vs vfs.root.mountfrom

On Sat, Aug 27, 2022, 7:25 PM Johannes Totz <jo@bruelltuete.com> wrote:

> On 27/08/2022 05:15, Warner Losh wrote:
> >
> >
> > On Fri, Aug 26, 2022 at 10:01 PM Peter Jeremy <peterj@freebsd.org
> > <mailto:peterj@freebsd.org>> wrote:
> >
> >     When booting from ZFS was first introduced, the recommended approach
> >     was to configure the root filesystem via vfs.root.mountfrom.  See
> (e.g.)
> >
> https://lists.freebsd.org/pipermail/freebsd-fs/2011-September/012482.html
> <https://lists.freebsd.org/pipermail/freebsd-fs/2011-September/012482.html
> >
> >
> https://lists.freebsd.org/pipermail/svn-src-head/2011-October/030641.html
> <https://lists.freebsd.org/pipermail/svn-src-head/2011-October/030641.html
> >
> >     and people wanted to change that to use the zpool bootfs property -
> e.g.
> >
> https://lists.freebsd.org/pipermail/freebsd-current/2009-October/012933.html
> <
> https://lists.freebsd.org/pipermail/freebsd-current/2009-October/012933.html
> >
> >
> https://lists.freebsd.org/pipermail/freebsd-fs/2010-March/008010.html <
> https://lists.freebsd.org/pipermail/freebsd-fs/2010-March/008010.html>
> >     resulting in it becoming optional in SVN r235330 in May 2012:
> >
> https://lists.freebsd.org/pipermail/svn-src-head/2012-May/036902.html <
> https://lists.freebsd.org/pipermail/svn-src-head/2012-May/036902.html>
> >
> >
> > Yes. This was a total hack at the time. The boot loader normally sets
> > this variable
> > w/o users intervening and this hack propagated into loader.conf files,
> > which has
> > always been a bad idea.
> >
> >     The recommendation to use vfs.root.mountfrom remains in parts of the
> >     wiki (e.g. https://wiki.freebsd.org/RootOnZFS/GPTZFSBoot/Mirror
> >     <https://wiki.freebsd.org/RootOnZFS/GPTZFSBoot/Mirror> step
> >     2.6) and on mailing list posts (e.g. the thread starting at
> >     https://lists.freebsd.org/pipermail/freebsd-fs/2020-July/028351.html
> >     <
> https://lists.freebsd.org/pipermail/freebsd-fs/2020-July/028351.html>).
> >     The loader(8) and loader.conf(5) man pages state that the root
> >     filesystem is set by vfs.root.mountfrom, with a default derived from
> >     /etc/fstab or, for ZFS only, currdev (the boot filesystem).
> >
> >
> > Yes. That's how it is supposed to work. The boot loader sets this based
> on
> > where it loaded the kernel. This variable should *NEVER* be set in a
> > loader.conf file unless you have a 'very special snowflake' system that
> > can't determine it automatically. ZFS root isn't special.
> >
> >     OTOH, https://wiki.freebsd.org/BootEnvironments
> >     <https://wiki.freebsd.org/BootEnvironments> states "/etc/fstab
> >     must be purged of any / mount, and /boot/loader.conf must not be
> >     setting vfs.root.mountfrom directly."  I can't find any other
> >     reference to this requirement.
> >
> >
> > /etc/fstab isn't used for zfs root. I believe that BootEnvironments is
> 100%
> > right here.
> >
> >     As for the BootEnvironment management tools themselves:
> >     * Neither beadm(8) nor bectl(8) mention vfs.root.mountfrom.
> >
> >
> > This is correct.
> >
> >     * beadm(8) refers to
> >     https://forums.freebsd.org/showthread.php?t=31662
> >     <https://forums.freebsd.org/showthread.php?t=31662>,
> >        which recommends setting vfs.root.mountfrom
> >
> >
> > This is bad advice.
> >
> >     * beadm(8) will update vfs.root.mountfrom in /boot/loader.conf if it
> >        exists but will not add it if it doesn't exist.
> >
> >
> > This is terrible. It's part of the continuing papering over of bugs that
> > have
> > long ago been fixed and should, at most, warn that's its set and that it
> > really shouldn't be.
> >
> >     * bectl(8) ignores /boot/loader.conf
> >
> >
> > This is proper behavior.
> >
> >     The last point means that switching an existing system from beadm to
> >     bectl is very likely to result in the system mounting the wrong root
> >     filesystem: An existing system is likely to have vfs.root.mountfrom
> >     set, since that used to be the only approach and is still widely
> >     recommended, whilst bectl ignores it and will therefore leave the
> >     old vfs.root.mountfrom value present.
> >
> >
> > Yes. bectl likely should, at most, warn that it's set in loader.conf as
> an
> > antifootshooting measure, but only due to the all-too-long history.
> >
> >     IMO, if setting vfs.root.mountfrom in /boot/loader.conf is no longer
> >     supported for ZFS, we need to do a better job of documenting that and
> >     removing documentation to the contrary.  I also believe that bectl
> >     should at least warn if /boot/loader.conf sets vfs.root.mountfrom.
> >
> >
> > It really never had been the proper way. One should almost never do
> > it today, except in the above 'special snowflake' systems that still
> crop up
> > from time to time (we have one at work since we still have some systems
> > that use gmirror roots, and the boot loader doesn't set it right for
> > them since
> > we can't for logistical reasons set /etc/fstab, but that's not ZFS
> > related). We
> > should actively militate against it with extreme prejudice, though, for
> > routine
> > uses. We should rip out all references in the docs related to ZFS. We
> should
> > warn if we detect it today. I agree completely with what you are saying
> > here.
> > I would be extraordinarily surprised to find anybody with a contrary
> > opinion.
>
> Isn't a legitimate use-case where /boot and / are on different
> devices/pools? That's what I'm using it for. bootfs property says where
> to find the kernel, and the accompanying vfs.root.mountfrom in
> loader.conf says where the matching rootfs is (in my case on a different
> pool).
>

Yea... that's a legitimate setup, but it is in the special snowflake
category... it is a setup we can't otherwise know and configure
automatically, but that works nonetheless with the right tweaks of expert
knobs. I think you could also get the same effect with the zfs pool in
/etc/fstab, bit I haven't tried it.

Warner

> >
> > Now, who is going to do the work? :)
> >
> > Warner
>
>