From nobody Sat Aug 27 04:15:55 2022 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4MF3Kh446dz4ZQbl for ; Sat, 27 Aug 2022 04:16:08 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mail-vs1-xe2c.google.com (mail-vs1-xe2c.google.com [IPv6:2607:f8b0:4864:20::e2c]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4MF3Kg5hfYz3NJX for ; Sat, 27 Aug 2022 04:16:07 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: by mail-vs1-xe2c.google.com with SMTP id p6so3417929vsr.9 for ; Fri, 26 Aug 2022 21:16:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bsdimp-com.20210112.gappssmtp.com; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc; bh=JikUl2LaMm/29qMeEkFLywcqnwINh+fJtt1z0LllJnE=; b=JZYOLCOfYETkcp8q4PyElTlfQr7xE5eV4vyjyheiCSkccmwZtptUz51RNDJJoTL7XT cBoaadw1K4NwIuxhkkzq62/uYLntWiVOhjC0mgQAKLm4KXHeZnqVAgS1DUOvFXtiqNFI 8DUJTB4zRqDyUveztnse9hw97dITI3bjBGcxHecgV+iWpXVzcoMzZt18tXPx7ipf0tmu IYndoVa4Pg2UoMCzwTNIQzCVGP6KPlKNgscAIaj2Y/OdiWk1sc5DuJAXrB9zPqG+wj7y gjzQHRHILzz0nBq4RkI55c7OLcwBDKKN14PIO9JzJbVQ7NcG3a3SKViWqHQi0ZYDHLOh 55ZA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc; bh=JikUl2LaMm/29qMeEkFLywcqnwINh+fJtt1z0LllJnE=; b=N2SW/a36+1ZjqgDfPw6ang7FeRF6j2/d5vr7S3o6Pnb6iO2mInqIAoN7vdaEoc89Hz Wa6a4dqluT6TNH1ozP+OT9HujnPD78eJtaCAbI3fYgLdLyey61YGRH74fKWfvS1vFXPJ atW25e/3JNn0bHistDOQKELH6yux/mVQ3qm+yV5LIgMlepyf6Kr83qac7Ox2q/WQa8Bk wdlvFmMblfTj78ZZMowKmzU46WTTg0o+eoc2PsldRV3ND4bicVUFriPtVH4z6a5Fv4nL ZvLr9FmY9ii9GCiglCnvgahZUUDrRPiAKHK8aDXZm4fYGRl36QLNBzswuU47AH2eXJp0 fQwQ== X-Gm-Message-State: ACgBeo2L4jMGNGEqwM4iC87v0Y/XzEQPaoQKqdN9BJYcnVOqwwrt+ufX 1YumldgAaZIfQJe7g8kqxvdKfs9f/xwqf19YtafzB/uMP7e1oA== X-Google-Smtp-Source: AA6agR4sgpA1X48xv1/8IdWn2pUPpFfd+BBmfnERc18hdbWqDggAJihjPwB3qvrWKIFs9HJ/4CZolxpqkiHnKe2DbA4= X-Received: by 2002:a05:6102:2146:b0:38f:f3d6:51da with SMTP id h6-20020a056102214600b0038ff3d651damr867003vsg.38.1661573766829; Fri, 26 Aug 2022 21:16:06 -0700 (PDT) List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 References: In-Reply-To: From: Warner Losh Date: Fri, 26 Aug 2022 22:15:55 -0600 Message-ID: Subject: Re: ZFS bootfs vs vfs.root.mountfrom To: Peter Jeremy , Kyle Evans Cc: FreeBSD FS Content-Type: multipart/alternative; boundary="00000000000023f0ef05e7314a51" X-Rspamd-Queue-Id: 4MF3Kg5hfYz3NJX X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=bsdimp-com.20210112.gappssmtp.com header.s=20210112 header.b=JZYOLCOf; dmarc=none; spf=none (mx1.freebsd.org: domain of wlosh@bsdimp.com has no SPF policy when checking 2607:f8b0:4864:20::e2c) smtp.mailfrom=wlosh@bsdimp.com X-Spamd-Result: default: False [-3.00 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; FORGED_SENDER(0.30)[imp@bsdimp.com,wlosh@bsdimp.com]; R_DKIM_ALLOW(-0.20)[bsdimp-com.20210112.gappssmtp.com:s=20210112]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::e2c:from]; MLMMJ_DEST(0.00)[freebsd-fs@freebsd.org]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; MIME_TRACE(0.00)[0:+,1:+,2:~]; R_SPF_NA(0.00)[no SPF record]; ARC_NA(0.00)[]; TO_DN_ALL(0.00)[]; FROM_NEQ_ENVFROM(0.00)[imp@bsdimp.com,wlosh@bsdimp.com]; RCPT_COUNT_THREE(0.00)[3]; FROM_HAS_DN(0.00)[]; DKIM_TRACE(0.00)[bsdimp-com.20210112.gappssmtp.com:+]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DMARC_NA(0.00)[bsdimp.com]; RCVD_COUNT_TWO(0.00)[2] X-ThisMailContainsUnwantedMimeParts: N --00000000000023f0ef05e7314a51 Content-Type: text/plain; charset="UTF-8" On Fri, Aug 26, 2022 at 10:01 PM Peter Jeremy wrote: > When booting from ZFS was first introduced, the recommended approach > was to configure the root filesystem via vfs.root.mountfrom. See (e.g.) > https://lists.freebsd.org/pipermail/freebsd-fs/2011-September/012482.html > https://lists.freebsd.org/pipermail/svn-src-head/2011-October/030641.html > and people wanted to change that to use the zpool bootfs property - e.g. > > https://lists.freebsd.org/pipermail/freebsd-current/2009-October/012933.html > https://lists.freebsd.org/pipermail/freebsd-fs/2010-March/008010.html > resulting in it becoming optional in SVN r235330 in May 2012: > https://lists.freebsd.org/pipermail/svn-src-head/2012-May/036902.html Yes. This was a total hack at the time. The boot loader normally sets this variable w/o users intervening and this hack propagated into loader.conf files, which has always been a bad idea. The recommendation to use vfs.root.mountfrom remains in parts of the > wiki (e.g. https://wiki.freebsd.org/RootOnZFS/GPTZFSBoot/Mirror step > 2.6) and on mailing list posts (e.g. the thread starting at > https://lists.freebsd.org/pipermail/freebsd-fs/2020-July/028351.html). > The loader(8) and loader.conf(5) man pages state that the root > filesystem is set by vfs.root.mountfrom, with a default derived from > /etc/fstab or, for ZFS only, currdev (the boot filesystem). > Yes. That's how it is supposed to work. The boot loader sets this based on where it loaded the kernel. This variable should *NEVER* be set in a loader.conf file unless you have a 'very special snowflake' system that can't determine it automatically. ZFS root isn't special. > OTOH, https://wiki.freebsd.org/BootEnvironments states "/etc/fstab > must be purged of any / mount, and /boot/loader.conf must not be > setting vfs.root.mountfrom directly." I can't find any other > reference to this requirement. > /etc/fstab isn't used for zfs root. I believe that BootEnvironments is 100% right here. > As for the BootEnvironment management tools themselves: > * Neither beadm(8) nor bectl(8) mention vfs.root.mountfrom. > This is correct. > * beadm(8) refers to https://forums.freebsd.org/showthread.php?t=31662, > which recommends setting vfs.root.mountfrom > This is bad advice. > * beadm(8) will update vfs.root.mountfrom in /boot/loader.conf if it > exists but will not add it if it doesn't exist. > This is terrible. It's part of the continuing papering over of bugs that have long ago been fixed and should, at most, warn that's its set and that it really shouldn't be. > * bectl(8) ignores /boot/loader.conf > This is proper behavior. > The last point means that switching an existing system from beadm to > bectl is very likely to result in the system mounting the wrong root > filesystem: An existing system is likely to have vfs.root.mountfrom > set, since that used to be the only approach and is still widely > recommended, whilst bectl ignores it and will therefore leave the > old vfs.root.mountfrom value present. > Yes. bectl likely should, at most, warn that it's set in loader.conf as an antifootshooting measure, but only due to the all-too-long history. > IMO, if setting vfs.root.mountfrom in /boot/loader.conf is no longer > supported for ZFS, we need to do a better job of documenting that and > removing documentation to the contrary. I also believe that bectl > should at least warn if /boot/loader.conf sets vfs.root.mountfrom. > It really never had been the proper way. One should almost never do it today, except in the above 'special snowflake' systems that still crop up from time to time (we have one at work since we still have some systems that use gmirror roots, and the boot loader doesn't set it right for them since we can't for logistical reasons set /etc/fstab, but that's not ZFS related). We should actively militate against it with extreme prejudice, though, for routine uses. We should rip out all references in the docs related to ZFS. We should warn if we detect it today. I agree completely with what you are saying here. I would be extraordinarily surprised to find anybody with a contrary opinion. Now, who is going to do the work? :) Warner --00000000000023f0ef05e7314a51 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable


=
On Fri, Aug 26, 2022 at 10:01 PM Pete= r Jeremy <peterj@freebsd.org&g= t; wrote:
When b= ooting from ZFS was first introduced, the recommended approach
was to configure the root filesystem via vfs.root.mountfrom.=C2=A0 See (e.g= .)
=C2=A0https://lists.freebsd= .org/pipermail/freebsd-fs/2011-September/012482.html
=C2=A0https://lists.freebsd= .org/pipermail/svn-src-head/2011-October/030641.html
and people wanted to change that to use the zpool bootfs property - e.g. =C2=A0https://lists.free= bsd.org/pipermail/freebsd-current/2009-October/012933.html
=C2=A0https://lists.freebsd.org= /pipermail/freebsd-fs/2010-March/008010.html
resulting in it becoming optional in SVN r235330 in May 2012:
=C2=A0https://lists.freebsd.org= /pipermail/svn-src-head/2012-May/036902.html

Yes. This was a total hack at the time. The boot loader normally sets= this variable
w/o users intervening and this hack propagated=C2= =A0into loader.conf files, which has
always been a bad idea.

The recommendation to use vfs.root.mountfrom remains in parts of the
wiki (e.g. https://wiki.freebsd.org/RootOnZFS/GP= TZFSBoot/Mirror step
2.6) and on mailing list posts (e.g. the thread starting at
https://lists.freebsd.org/piperm= ail/freebsd-fs/2020-July/028351.html).
The loader(8) and loader.conf(5) man pages state that the root
filesystem is set by vfs.root.mountfrom, with a default derived from
/etc/fstab or, for ZFS only, currdev (the boot filesystem).

Yes. That's how it is supposed to work. The boot l= oader sets this based on
where it loaded the kernel. This variabl= e should *NEVER* be set in a
loader.conf file unless you have a &= #39;very special snowflake' system that
can't determine i= t automatically. ZFS root isn't special.
=C2=A0
OTOH, https://wiki.freebsd.org/BootEnvironments states = "/etc/fstab
must be purged of any / mount, and /boot/loader.conf must not be
setting vfs.root.mountfrom directly."=C2=A0 I can't find any other=
reference to this requirement.

/etc/fst= ab isn't used for zfs root. I believe that BootEnvironments is 100%
right here.
=C2=A0
As for the BootEnvironment management tools themselves:
* Neither beadm(8) nor bectl(8) mention vfs.root.mountfrom.

This is correct.
=C2=A0
* beadm(8) refers to https://forums.freebsd.org/s= howthread.php?t=3D31662,
=C2=A0 which recommends setting vfs.root.mountfrom
This is bad advice.
=C2=A0
* beadm(8) will update vfs.root.mountfrom in /boot/loader.conf if it
=C2=A0 exists but will not add it if it doesn't exist.
=

This is terrible. It's part of the continuing paper= ing over of bugs that have
long ago been fixed and should, at mos= t, warn that's its set and that it
really shouldn't be.
=C2=A0
* bectl(8) ignores /boot/loader.conf

Th= is is proper behavior.
=C2=A0
The last point means that switching an existing system from beadm to
bectl is very likely to result in the system mounting the wrong root
filesystem: An existing system is likely to have vfs.root.mountfrom
set, since that used to be the only approach and is still widely
recommended, whilst bectl ignores it and will therefore leave the
old vfs.root.mountfrom value present.

Y= es. bectl=C2=A0likely should, at most, warn that it's set in loader.con= f as an
antifootshooting=C2=A0measure, but only due to the all-to= o-long history.
=C2=A0
IMO, if setting vfs.root.mountfrom in /boot/loader.conf is no longer
supported for ZFS, we need to do a better job of documenting that and
removing documentation to the contrary.=C2=A0 I also believe that bectl
should at least warn if /boot/loader.conf sets vfs.root.mountfrom.

It really never had been the proper way. One sh= ould almost never do
it today, except in the above 'special s= nowflake' systems that=C2=A0still crop up
from time to time (= we have one at work since we still have=C2=A0some systems
that us= e=C2=A0gmirror roots,=C2=A0and the boot loader doesn't set it right for= them since
we can't for logistical reasons set /etc/fstab,= =C2=A0but that's not ZFS related). We
should actively militat= e against it with extreme prejudice,=C2=A0though,=C2=A0for routine
uses. We should rip out all references in the docs related to ZFS. We sho= uld
warn if we detect it today. I agree completely with what you = are saying here.
I would be extraordinarily surprised to find any= body with a contrary opinion.

Now, who is going to= do the work? :)

Warner
--00000000000023f0ef05e7314a51--