From nobody Sun Aug 28 01:39:27 2022 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4MFbpk3MW1z4ZRKn for ; Sun, 28 Aug 2022 01:39:42 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mail-vk1-xa35.google.com (mail-vk1-xa35.google.com [IPv6:2607:f8b0:4864:20::a35]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4MFbpj4RGzz3SNk for ; Sun, 28 Aug 2022 01:39:41 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: by mail-vk1-xa35.google.com with SMTP id q14so2325831vke.9 for ; Sat, 27 Aug 2022 18:39:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bsdimp-com.20210112.gappssmtp.com; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc; bh=MzcAvg7NCYP6ptReIXbiXcERv3iXWjiQ8lVDHLqrHp0=; b=X30pVrC8WiVhXQKtE3BmeIpP3krnrvfXmNgkSNDQTbr+qcM2TnWPKQGmO2R7qIkkgk aKvlRxKUVXRqfgYpnbDHepW32C2xJXS7EtdECWUsgMvt3GIoWThSOCdioiad4GOpz1V9 1a+KQesRTFVpdqh+vTAXCejtZndD6NDVuwaQI6LJI4B29QZzQet4FmyQnqW20zjs5Ds2 PoV7LIEiqfUMUfkrhIP3LdksqfY/bwzSZ6H+SaLJ8nC/A6TB4C8npbMo63lgXb8SuSaT jhfO3FrLqpQckwqBPZVHPpiiuK+TL5XfeJeVsn8GxUa6/k8yJc1WV5+mjz0fhUI/3Va7 5ovQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc; bh=MzcAvg7NCYP6ptReIXbiXcERv3iXWjiQ8lVDHLqrHp0=; b=CHGvh1Em3nC3FKkq5zZBqdeD2N00NCPnhBp6fCUc0OCH5YtZET03Ulcd6/Q4JpfsCF ddK+Yn1dl8TEaROVt1mm3YatMotZfAm2ySO8tnswd36dzxIrvyr91Rr9Y7bf28u1MWVG 2U9y2X6P1ZzuKfzwqcFIpgP/ZL2kGFWIER0/dsCp3NAAoiXv1taPFQWyxzhCG363p9xs BXVh2hIaG5WIIUhcY3AEBa2iDOKrPNezcDonLxxguoJ+0ILYDQjzCkaJIaqUVqD501Km gs7puSlv8Eey8x2/vb2dKS5Fc8/BuIn9BMQake4kHoTqI/BSxmcKbbtjp8A5lvNTLFM+ fcQA== X-Gm-Message-State: ACgBeo3OhM60mhe6yVklS8XXhGQ0EEwX1aPxkcKfRAL4e+RbMZn6Dbbr 7xpm+LA8zTadGDOuUr5hqOBbycuovIG/ZwkVe9ly+w== X-Google-Smtp-Source: AA6agR666NFaWDOLtP2Re70BoOakIeRbx8ITBRMcFmFpMu9/iLV4qjJUF4Fe1IZxyk/4fIKLgCqJinhnnycgT3vrPZ4= X-Received: by 2002:a1f:5ec7:0:b0:394:4ef0:d583 with SMTP id s190-20020a1f5ec7000000b003944ef0d583mr698409vkb.37.1661650780389; Sat, 27 Aug 2022 18:39:40 -0700 (PDT) List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 References: In-Reply-To: From: Warner Losh Date: Sat, 27 Aug 2022 19:39:27 -0600 Message-ID: Subject: Re: ZFS bootfs vs vfs.root.mountfrom To: Johannes Totz Cc: Peter Jeremy , Kyle Evans , FreeBSD FS Content-Type: multipart/alternative; boundary="00000000000081963805e7433802" X-Rspamd-Queue-Id: 4MFbpj4RGzz3SNk X-Spamd-Bar: - Authentication-Results: mx1.freebsd.org; dkim=pass header.d=bsdimp-com.20210112.gappssmtp.com header.s=20210112 header.b=X30pVrC8; dmarc=none; spf=none (mx1.freebsd.org: domain of wlosh@bsdimp.com has no SPF policy when checking 2607:f8b0:4864:20::a35) smtp.mailfrom=wlosh@bsdimp.com X-Spamd-Result: default: False [-2.00 / 15.00]; URI_COUNT_ODD(1.00)[69]; NEURAL_HAM_SHORT(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-0.999]; NEURAL_HAM_LONG(-1.00)[-0.998]; FORGED_SENDER(0.30)[imp@bsdimp.com,wlosh@bsdimp.com]; R_DKIM_ALLOW(-0.20)[bsdimp-com.20210112.gappssmtp.com:s=20210112]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; MIME_TRACE(0.00)[0:+,1:+,2:~]; MLMMJ_DEST(0.00)[freebsd-fs@freebsd.org]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::a35:from]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; R_SPF_NA(0.00)[no SPF record]; RCVD_TLS_LAST(0.00)[]; ARC_NA(0.00)[]; TO_DN_ALL(0.00)[]; FROM_NEQ_ENVFROM(0.00)[imp@bsdimp.com,wlosh@bsdimp.com]; RCPT_COUNT_THREE(0.00)[4]; FROM_HAS_DN(0.00)[]; DKIM_TRACE(0.00)[bsdimp-com.20210112.gappssmtp.com:+]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DMARC_NA(0.00)[bsdimp.com]; RCVD_COUNT_TWO(0.00)[2] X-ThisMailContainsUnwantedMimeParts: N --00000000000081963805e7433802 Content-Type: text/plain; charset="UTF-8" On Sat, Aug 27, 2022, 7:25 PM Johannes Totz wrote: > On 27/08/2022 05:15, Warner Losh wrote: > > > > > > On Fri, Aug 26, 2022 at 10:01 PM Peter Jeremy > > wrote: > > > > When booting from ZFS was first introduced, the recommended approach > > was to configure the root filesystem via vfs.root.mountfrom. See > (e.g.) > > > https://lists.freebsd.org/pipermail/freebsd-fs/2011-September/012482.html > > > > > https://lists.freebsd.org/pipermail/svn-src-head/2011-October/030641.html > > > > and people wanted to change that to use the zpool bootfs property - > e.g. > > > https://lists.freebsd.org/pipermail/freebsd-current/2009-October/012933.html > < > https://lists.freebsd.org/pipermail/freebsd-current/2009-October/012933.html > > > > > https://lists.freebsd.org/pipermail/freebsd-fs/2010-March/008010.html < > https://lists.freebsd.org/pipermail/freebsd-fs/2010-March/008010.html> > > resulting in it becoming optional in SVN r235330 in May 2012: > > > https://lists.freebsd.org/pipermail/svn-src-head/2012-May/036902.html < > https://lists.freebsd.org/pipermail/svn-src-head/2012-May/036902.html> > > > > > > Yes. This was a total hack at the time. The boot loader normally sets > > this variable > > w/o users intervening and this hack propagated into loader.conf files, > > which has > > always been a bad idea. > > > > The recommendation to use vfs.root.mountfrom remains in parts of the > > wiki (e.g. https://wiki.freebsd.org/RootOnZFS/GPTZFSBoot/Mirror > > step > > 2.6) and on mailing list posts (e.g. the thread starting at > > https://lists.freebsd.org/pipermail/freebsd-fs/2020-July/028351.html > > < > https://lists.freebsd.org/pipermail/freebsd-fs/2020-July/028351.html>). > > The loader(8) and loader.conf(5) man pages state that the root > > filesystem is set by vfs.root.mountfrom, with a default derived from > > /etc/fstab or, for ZFS only, currdev (the boot filesystem). > > > > > > Yes. That's how it is supposed to work. The boot loader sets this based > on > > where it loaded the kernel. This variable should *NEVER* be set in a > > loader.conf file unless you have a 'very special snowflake' system that > > can't determine it automatically. ZFS root isn't special. > > > > OTOH, https://wiki.freebsd.org/BootEnvironments > > states "/etc/fstab > > must be purged of any / mount, and /boot/loader.conf must not be > > setting vfs.root.mountfrom directly." I can't find any other > > reference to this requirement. > > > > > > /etc/fstab isn't used for zfs root. I believe that BootEnvironments is > 100% > > right here. > > > > As for the BootEnvironment management tools themselves: > > * Neither beadm(8) nor bectl(8) mention vfs.root.mountfrom. > > > > > > This is correct. > > > > * beadm(8) refers to > > https://forums.freebsd.org/showthread.php?t=31662 > > , > > which recommends setting vfs.root.mountfrom > > > > > > This is bad advice. > > > > * beadm(8) will update vfs.root.mountfrom in /boot/loader.conf if it > > exists but will not add it if it doesn't exist. > > > > > > This is terrible. It's part of the continuing papering over of bugs that > > have > > long ago been fixed and should, at most, warn that's its set and that it > > really shouldn't be. > > > > * bectl(8) ignores /boot/loader.conf > > > > > > This is proper behavior. > > > > The last point means that switching an existing system from beadm to > > bectl is very likely to result in the system mounting the wrong root > > filesystem: An existing system is likely to have vfs.root.mountfrom > > set, since that used to be the only approach and is still widely > > recommended, whilst bectl ignores it and will therefore leave the > > old vfs.root.mountfrom value present. > > > > > > Yes. bectl likely should, at most, warn that it's set in loader.conf as > an > > antifootshooting measure, but only due to the all-too-long history. > > > > IMO, if setting vfs.root.mountfrom in /boot/loader.conf is no longer > > supported for ZFS, we need to do a better job of documenting that and > > removing documentation to the contrary. I also believe that bectl > > should at least warn if /boot/loader.conf sets vfs.root.mountfrom. > > > > > > It really never had been the proper way. One should almost never do > > it today, except in the above 'special snowflake' systems that still > crop up > > from time to time (we have one at work since we still have some systems > > that use gmirror roots, and the boot loader doesn't set it right for > > them since > > we can't for logistical reasons set /etc/fstab, but that's not ZFS > > related). We > > should actively militate against it with extreme prejudice, though, for > > routine > > uses. We should rip out all references in the docs related to ZFS. We > should > > warn if we detect it today. I agree completely with what you are saying > > here. > > I would be extraordinarily surprised to find anybody with a contrary > > opinion. > > Isn't a legitimate use-case where /boot and / are on different > devices/pools? That's what I'm using it for. bootfs property says where > to find the kernel, and the accompanying vfs.root.mountfrom in > loader.conf says where the matching rootfs is (in my case on a different > pool). > Yea... that's a legitimate setup, but it is in the special snowflake category... it is a setup we can't otherwise know and configure automatically, but that works nonetheless with the right tweaks of expert knobs. I think you could also get the same effect with the zfs pool in /etc/fstab, bit I haven't tried it. Warner > > > > Now, who is going to do the work? :) > > > > Warner > > --00000000000081963805e7433802 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable


On Sat, Aug 27, 2022, 7:25 PM Johannes Totz <jo@bruelltuete.com> wrote:
=
On 27/08/2022 05:15, Warner Losh wrote:
>
>
> On Fri, Aug 26, 2022 at 10:01 PM Peter Jeremy <peterj@freebsd.org
> <mailto:peterj@freebsd.org>> wrote:
>
>=C2=A0 =C2=A0 =C2=A0When booting from ZFS was first introduced, the rec= ommended approach
>=C2=A0 =C2=A0 =C2=A0was to configure the root filesystem via vfs.root.m= ountfrom.=C2=A0 See (e.g.)
>=C2=A0 =C2=A0 =C2=A0https://lists.freebsd.org/pipermail/freebsd-fs/2011-September/01248= 2.html <h= ttps://lists.freebsd.org/pipermail/freebsd-fs/2011-September/012482.html>
>=C2=A0 =C2=A0 =C2=A0https://lists.freebsd.org/pipermail/svn-src-head/2011-October/03064= 1.html <h= ttps://lists.freebsd.org/pipermail/svn-src-head/2011-October/030641.html>
>=C2=A0 =C2=A0 =C2=A0and people wanted to change that to use the zpool b= ootfs property - e.g.
>=C2=A0 =C2=A0 =C2=A0https://lists.freebsd.org/pipermail/freebsd-current/2009-Octobe= r/012933.html <https://lists.freebsd.org/pipermail/freebsd-current/2009-October/0= 12933.html>
>=C2=A0 =C2=A0 =C2=A0https://lists.freebsd.org/pipermail/freebsd-fs/2010-March/008010.html <https://lists= .freebsd.org/pipermail/freebsd-fs/2010-March/008010.html>
>=C2=A0 =C2=A0 =C2=A0resulting in it becoming optional in SVN r235330 in= May 2012:
>=C2=A0 =C2=A0 =C2=A0https://lists.freebsd.org/pipermail/svn-src-head/2012-May/036902.html <https://lists= .freebsd.org/pipermail/svn-src-head/2012-May/036902.html>
>
>
> Yes. This was a total hack at the time. The boot loader normally sets =
> this variable
> w/o users intervening and this hack propagated=C2=A0into loader.conf f= iles,
> which has
> always been a bad idea.
>
>=C2=A0 =C2=A0 =C2=A0The recommendation to use vfs.root.mountfrom remain= s in parts of the
>=C2=A0 =C2=A0 =C2=A0wiki (e.g. ht= tps://wiki.freebsd.org/RootOnZFS/GPTZFSBoot/Mirror
>=C2=A0 =C2=A0 =C2=A0<https://w= iki.freebsd.org/RootOnZFS/GPTZFSBoot/Mirror> step
>=C2=A0 =C2=A0 =C2=A02.6) and on mailing list posts (e.g. the thread sta= rting at
>=C2=A0 =C2=A0 =C2=A0https://lists.freebsd.org/pipermail/freebsd-fs/2020-July/028351.html=
>=C2=A0 =C2=A0 =C2=A0<https://lists.freebsd.org/pipermail/freebsd-fs/2020-July/028351.html= >).
>=C2=A0 =C2=A0 =C2=A0The loader(8) and loader.conf(5) man pages state th= at the root
>=C2=A0 =C2=A0 =C2=A0filesystem is set by vfs.root.mountfrom, with a def= ault derived from
>=C2=A0 =C2=A0 =C2=A0/etc/fstab or, for ZFS only, currdev (the boot file= system).
>
>
> Yes. That's how it is supposed to work. The boot loader sets this = based on
> where it loaded the kernel. This variable should *NEVER* be set in a > loader.conf file unless you have a 'very special snowflake' sy= stem that
> can't determine it automatically. ZFS root isn't special.
>
>=C2=A0 =C2=A0 =C2=A0OTOH, https://wiki.freeb= sd.org/BootEnvironments
>=C2=A0 =C2=A0 =C2=A0<https://wiki.freebsd= .org/BootEnvironments> states "/etc/fstab
>=C2=A0 =C2=A0 =C2=A0must be purged of any / mount, and /boot/loader.con= f must not be
>=C2=A0 =C2=A0 =C2=A0setting vfs.root.mountfrom directly."=C2=A0 I = can't find any other
>=C2=A0 =C2=A0 =C2=A0reference to this requirement.
>
>
> /etc/fstab isn't used for zfs root. I believe that BootEnvironment= s is 100%
> right here.
>
>=C2=A0 =C2=A0 =C2=A0As for the BootEnvironment management tools themsel= ves:
>=C2=A0 =C2=A0 =C2=A0* Neither beadm(8) nor bectl(8) mention vfs.root.mo= untfrom.
>
>
> This is correct.
>
>=C2=A0 =C2=A0 =C2=A0* beadm(8) refers to
>=C2=A0 =C2=A0 =C2=A0https://forums= .freebsd.org/showthread.php?t=3D31662
>=C2=A0 =C2=A0 =C2=A0<https://fo= rums.freebsd.org/showthread.php?t=3D31662>,
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 which recommends setting vfs.root.mountfrom=
>
>
> This is bad advice.
>
>=C2=A0 =C2=A0 =C2=A0* beadm(8) will update vfs.root.mountfrom in /boot/= loader.conf if it
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 exists but will not add it if it doesn'= t exist.
>
>
> This is terrible. It's part of the continuing papering over of bug= s that
> have
> long ago been fixed and should, at most, warn that's its set and t= hat it
> really shouldn't be.
>
>=C2=A0 =C2=A0 =C2=A0* bectl(8) ignores /boot/loader.conf
>
>
> This is proper behavior.
>
>=C2=A0 =C2=A0 =C2=A0The last point means that switching an existing sys= tem from beadm to
>=C2=A0 =C2=A0 =C2=A0bectl is very likely to result in the system mounti= ng the wrong root
>=C2=A0 =C2=A0 =C2=A0filesystem: An existing system is likely to have vf= s.root.mountfrom
>=C2=A0 =C2=A0 =C2=A0set, since that used to be the only approach and is= still widely
>=C2=A0 =C2=A0 =C2=A0recommended, whilst bectl ignores it and will there= fore leave the
>=C2=A0 =C2=A0 =C2=A0old vfs.root.mountfrom value present.
>
>
> Yes. bectl=C2=A0likely should, at most, warn that it's set in load= er.conf as an
> antifootshooting=C2=A0measure, but only due to the all-too-long histor= y.
>
>=C2=A0 =C2=A0 =C2=A0IMO, if setting vfs.root.mountfrom in /boot/loader.= conf is no longer
>=C2=A0 =C2=A0 =C2=A0supported for ZFS, we need to do a better job of do= cumenting that and
>=C2=A0 =C2=A0 =C2=A0removing documentation to the contrary.=C2=A0 I als= o believe that bectl
>=C2=A0 =C2=A0 =C2=A0should at least warn if /boot/loader.conf sets vfs.= root.mountfrom.
>
>
> It really never had been the proper way. One should almost never do > it today, except in the above 'special snowflake' systems that= =C2=A0still crop up
> from time to time (we have one at work since we still have=C2=A0some s= ystems
> that use=C2=A0gmirror roots,=C2=A0and the boot loader doesn't set = it right for
> them since
> we can't for logistical reasons set /etc/fstab,=C2=A0but that'= s not ZFS
> related). We
> should actively militate against it with extreme prejudice,=C2=A0thoug= h,=C2=A0for
> routine
> uses. We should rip out all references in the docs related to ZFS. We = should
> warn if we detect it today. I agree completely with what you are sayin= g
> here.
> I would be extraordinarily surprised to find anybody with a contrary <= br> > opinion.

Isn't a legitimate use-case where /boot and / are on different
devices/pools? That's what I'm using it for. bootfs property says w= here
to find the kernel, and the accompanying vfs.root.mountfrom in
loader.conf says where the matching rootfs is (in my case on a different pool).

Yea... that's a legitimate setup, but it is in the special snowfl= ake category... it is a setup we can't otherwise know and configure aut= omatically, but that works nonetheless with the right tweaks of expert knob= s. I think you could also get the same effect with the zfs pool in /etc/fst= ab, bit I haven't tried it.

Warner=C2=A0
>
> Now, who is going to do the work? :)
>
> Warner

--00000000000081963805e7433802--