From nobody Tue Sep 06 16:13:04 2022 X-Original-To: freebsd-arm@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4MMVmX5znwz4c7M1 for ; Tue, 6 Sep 2022 16:13:16 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mail-vs1-xe30.google.com (mail-vs1-xe30.google.com [IPv6:2607:f8b0:4864:20::e30]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4MMVmX131Gz3GR9 for ; Tue, 6 Sep 2022 16:13:16 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: by mail-vs1-xe30.google.com with SMTP id o123so12213214vsc.3 for ; Tue, 06 Sep 2022 09:13:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bsdimp-com.20210112.gappssmtp.com; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date; bh=FqkbuO42xWfsyAMrYOu7oMYybQWuCoaSmuP48pWyLpE=; b=f+oOV/ngBM53QBW/thCtLOv6wJDaSy6/H/NJ7aQ8gt6A4R85C30NjxjB07OUIZLKBL JdpLjSRchO+ZoCAxjq48khP5UwGoY50OlxBgEwdsRtVyCCfkQFGCw9DAwNElu7pyQF6q CaOiw435N1rhtrtBNd7Qi0PYcexdx0q14QbsX35v69JGKw3WE0y98fgO+V7mVm00K0R1 S3Q6OfdQRpjtoLamwetj25sVp0sTYtpU/l/Vy3K9c+4xNyb7p97oRTpN7YtvOtZh8NJs WXxvyUnJrzsjcwd1NE7FKBMZvtdbk0fUqld6sn20Ha8XjKtgfrG3CemasYeNwlq6QGZ6 VayQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date; bh=FqkbuO42xWfsyAMrYOu7oMYybQWuCoaSmuP48pWyLpE=; b=4Pvpd19TNW6bTV4iQVoaWv8gyR0W4nxUnFDrqU45FVJmy01FxHDoTI7Y8IYGXk7PB1 o+bflHymHqQfyjrg/GK6b6wXskBqEjJ2O+eIA4ZIr5M79VHDv21ABmmoLHf2LvAwDsF4 xSiTtF0jeZYDl7IrPLDLhS8La3JK9O1PJq73Vt0uXYtLooem0wffuU2ER2Z3tXk90vT0 RMpeo98+0hq8TTlZHNOWIMZzuAjf9ca6JFNagUhpTzINNA88HEfi9FAmUp/zrFjYwWJ7 ttRQL1cQp++80vCHpcsrYHtI11BAVZbez6Z2u8lqrywV0Km8rBCuGKhhSwLwMr/9dADW hmRw== X-Gm-Message-State: ACgBeo0lQSvmZcAswnZpHTE1A2+L/ieDy/zLUZwpgPUs58h1lif+hSfU jdFsGrzETyKeGUMUHHvIwYrp/nsJ7y/pXWKeTo0MOg== X-Google-Smtp-Source: AA6agR4+ByNu1r2/Lqipvjo7Od7TcGII0sgXF7f3gQPpDOM/wzbsKFTf+RrgLni+obqD47bA9RHelEppd7tqk9252E4= X-Received: by 2002:a05:6102:345:b0:390:9ebf:3d2c with SMTP id e5-20020a056102034500b003909ebf3d2cmr15272383vsa.11.1662480795295; Tue, 06 Sep 2022 09:13:15 -0700 (PDT) List-Id: Porting FreeBSD to ARM processors List-Archive: https://lists.freebsd.org/archives/freebsd-arm List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-arm@freebsd.org MIME-Version: 1.0 References: <79278D35-F3CE-43F9-BB85-1D1798B6B1D4.ref@yahoo.com> <79278D35-F3CE-43F9-BB85-1D1798B6B1D4@yahoo.com> In-Reply-To: <79278D35-F3CE-43F9-BB85-1D1798B6B1D4@yahoo.com> From: Warner Losh Date: Tue, 6 Sep 2022 10:13:04 -0600 Message-ID: Subject: Re: kernel update broke -current To: Mark Millard Cc: freebsd-arm Content-Type: multipart/alternative; boundary="0000000000003ff7c405e804796f" X-Rspamd-Queue-Id: 4MMVmX131Gz3GR9 X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=bsdimp-com.20210112.gappssmtp.com header.s=20210112 header.b="f+oOV/ng"; dmarc=none; spf=none (mx1.freebsd.org: domain of wlosh@bsdimp.com has no SPF policy when checking 2607:f8b0:4864:20::e30) smtp.mailfrom=wlosh@bsdimp.com X-Spamd-Result: default: False [-3.00 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; FORGED_SENDER(0.30)[imp@bsdimp.com,wlosh@bsdimp.com]; R_DKIM_ALLOW(-0.20)[bsdimp-com.20210112.gappssmtp.com:s=20210112]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; FREEMAIL_TO(0.00)[yahoo.com]; R_SPF_NA(0.00)[no SPF record]; RCVD_TLS_LAST(0.00)[]; MLMMJ_DEST(0.00)[freebsd-arm@freebsd.org]; MIME_TRACE(0.00)[0:+,1:+,2:~]; RCVD_COUNT_TWO(0.00)[2]; ARC_NA(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; PREVIOUSLY_DELIVERED(0.00)[freebsd-arm@freebsd.org]; FROM_HAS_DN(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::e30:from]; DMARC_NA(0.00)[bsdimp.com]; TO_DN_ALL(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[bsdimp-com.20210112.gappssmtp.com:+]; FROM_NEQ_ENVFROM(0.00)[imp@bsdimp.com,wlosh@bsdimp.com] X-ThisMailContainsUnwantedMimeParts: N --0000000000003ff7c405e804796f Content-Type: text/plain; charset="UTF-8" On Mon, Sep 5, 2022 at 4:21 PM Mark Millard wrote: > Warner Losh wrote on > Date: Mon, 05 Sep 2022 20:07:33 UTC : > > > On Sun, Sep 4, 2022 at 4:51 PM void wrote: > > > > > On Sun, Sep 04, 2022 at 07:31:59PM +0000, void wrote: > > > > > > > >On Sun, 4 Sep 2022, at 19:07, Warner Losh wrote: > > > >> You need a newer boot loader... > > > > > > > >I was thinking - getting latest -current image, booting to it then > > > >copying the contents of /boot from the image onto the mounted zpool > ... > > > > > > > >feasible? > > > > > > Seems only EFI/ needed replacing from a recent snapshot. I thought it > might > > > be all of /boot but I was wrong. Thank you Mark! > > > > > > > Yes. You'll need to update EFI/BOOT on your ESP. The rest of /boot is > > updated > > when you do an installworld. > > One of the oddities of the update sequence instructions is > the lack of coverage of the likes of: > > Load Path: /efi\boot\bootaa64.efi > > What step of the sequencing for the overall system update? > When is such an update required? (Here the example would > be: Before rebooting when the ZFS pool(s) possibly used to > boot gain new features?) When is it not required to update > the loader in the ESP (or analogous)? Even just knowing > the stage at which one should do the update indicates some > about when to figure out if an update is needed and so > prompts to be ready. > Today: make installworld installkernel mount -t msdos /dev/ /boot/efi and then one of three scenarios (1) You have the old boot1.efi. This will *always* be in ESP:EFI\BOOT\BOOT${ARCH}.EFI. (see uefi(8) for the values of ARCH). You can automatically detect this for most installations that were done since about FreeBSD 12: % sudo efivar | grep Boot1 cfee69ad-a0de-47a9-93a8-f63106f8ae99-Boot1Path cfee69ad-a0de-47a9-93a8-f63106f8ae99-Boot1Dev In this case you would do: % sudo cp /boot/boot1.efi /boot/efi/efi/boot/boot${ARCH}.efi and you are done. Please note: If your system was installed a long time ago, you should also check to see if you have a LoaderPath variable set: % sudo efivar --utf cfee69ad-a0de-47a9-93a8-f63106f8ae99-LoaderPath /boot/loader.efi When Boot1 isn't set and it is set to the above value (or something similar), then you are booting with a really old copy of boot1.efi. You will need to update it and may need to arrange for a larger ESP since FreeBSD's boot1.efifat was a tiny image that was hard to grow. Steal space from swap for a larger ESP, etc. Documenting that is beyond the scope of this email. (2) You are booting with loader.efi and it is in the bug-compatibility place. Some UEFI BIOSes don't respect or can't set BootXXXX variables set by efibootmgr(8). In that case, many people are booting from the 'compatibiltiy' location for removable devices. This will almost always be a single boot scenario because you can't easily boot other systems like this (well, unless 'quit' works from the OK prompt to go to the next one on the list, but I digress). You will see something like: % sudo efivar --utf cfee69ad-a0de-47a9-93a8-f63106f8ae99-LoaderPath cfee69ad-a0de-47a9-93a8-f63106f8ae99-LoaderPath \EFI\BOOT\BOOTX64.EFI (or AA64) when you are booting this way. In this case the update command is: % sudo cp /boot/loader.efi /boot/efi/efi/boot/boot${ARCH}.efi to upgrade. Some people may be doing this already on systems that can support efibootmgr(8) and they can of course upgrade to using that, though that's also beyond the scope of this email. (3) You are booting on a working system with a FreeBSD installed by the boot loader, or some other custom arrangement. In that case, you'll see something like: sudo efivar --utf cfee69ad-a0de-47a9-93a8-f63106f8ae99-LoaderPath cfee69ad-a0de-47a9-93a8-f63106f8ae99-LoaderPath \EFI\FREEBSD\LOADER.EFI (or some other place for custom setups: I assume if you are doing that you know enough to update my instructions as appropriate). To update, you'd do % sudo cp /boot/loader.efi /boot/efi/efi/freebsd/loader.efi Automating all these cases is, needless to say, tricky. > Part of the sequencing gets into having needed to do a > installworld to have a from-same-build content replacement > for the likes of /efi\boot\bootaa64.efi . So installkernel > already done, a reboot already done, and an installworld > having been done as well in order to have something to > copy over. (There are no pointers to alternate places to > get copies that I know of. One can find copies in the > build tree when one builds from source locally. So waiting > is not really required for that context.) > Yea, I have a branch that has an 'installboot' target that updates the boot loader regardless, but given the 50-odd combinations of ways we can boot, it's a bit bogged down. > This also make it seem that updating ZFS pool features > should wait until after a system upgrade that spans both > the loader and kernel being ready for the new features, > even if compatibility with other systems is not a worry. > Yes. ALWAYS update your boot blocks before zpool update. > Do any of the system upgrade instructions cover such > relationships? Do any of the ZFS pool upgrade instructions > cover such? Does zpool or the like suggest such issues > when it reports there are new features that could be > enabled? > See above. :) > Part of what I expect happened here was contributed to by > a lack of being prompted to even think about the relevant > issues, leading to a pool feature upgrade that had not > been prepared for. > Yea, so far 100% of the 'help, I upgraded and now the boot loader can't see zfs pool' issues have been 'I didn't upgrade my loader.efi properly after zpool upgrade.' It was mandatory when we had the OpenZFS rebase, but it is also necessary every few OpenZFS updates from upstream as nnew features are enabled. I'd love for someone to add this information to the handbook, and I'd happily review such an effort. I have time to do a brain dump, but not to make it pretty / formatted for asciidoctor and won't for some time. Warner > === > Mark Millard > marklmi at yahoo.com > > --0000000000003ff7c405e804796f Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable


=
On Mon, Sep 5, 2022 at 4:21 PM Mark M= illard <marklmi@yahoo.com> w= rote:
Warner Los= h <imp_at_bsdimp.com> wrote on
Date: Mon, 05 Sep 2022 20:07:33 UTC :

> On Sun, Sep 4, 2022 at 4:51 PM void <void@f-m.fm> wrote:
>
> > On Sun, Sep 04, 2022 at 07:31:59PM +0000, void wrote:
> > >
> > >On Sun, 4 Sep 2022, at 19:07, Warner Losh wrote:
> > >> You need a newer boot loader...
> > >
> > >I was thinking - getting latest -current image, booting to it= then
> > >copying the contents of /boot from the image onto the mounted= zpool ...
> > >
> > >feasible?
> >
> > Seems only EFI/ needed replacing from a recent snapshot. I though= t it might
> > be all of /boot but I was wrong. Thank you Mark!
> >
>
> Yes. You'll need to update EFI/BOOT on your ESP. The rest of /boot= is
> updated
> when you do an installworld.

One of the oddities of the update sequence instructions is
the lack of coverage of the likes of:

Load Path: /efi\boot\bootaa64.efi

What step of the sequencing for the overall system update?
When is such an update required? (Here the example would
be: Before rebooting when the ZFS pool(s) possibly used to
boot gain new features?) When is it not required to update
the loader in the ESP (or analogous)? Even just knowing
the stage at which one should do the update indicates some
about when to figure out if an update is needed and so
prompts to be ready.

Today:
<= br>
make installworld installkernel
mount -t msdos /dev= /<mumble-esp> /boot/efi

and then one of thre= e scenarios

(1) You have the old boot1.efi. This w= ill *always* be in ESP:EFI\BOOT\BOOT${ARCH}.EFI.
(see uefi(8) for= the values of ARCH). You can automatically detect this for most installati= ons
that were done since about FreeBSD 12:
% sudo efiva= r | grep Boot1
cfee69ad-a0de-47a9-93a8-f63106f8ae99-Boot1Path
= cfee69ad-a0de-47a9-93a8-f63106f8ae99-Boot1Dev

= In this case you would do:

% sudo cp /boot/boot1.e= fi /boot/efi/efi/boot/boot${ARCH}.efi and you are done.

Please note: If your system was installed a long time ago, you should= also check to see if
you have a LoaderPath variable set:
% sudo efivar --utf cfee69ad-a0de-47a9-93a8-f63106f8ae99-LoaderPath
<= /div>
/boot/loader.efi

When Boot1 isn't se= t and it is set to the above value (or something similar), then you are
booting with a really old copy of boot1.efi. You will need to update= it and may need to arrange
for a larger ESP since FreeBSD's = boot1.efifat was a tiny image that was hard to grow. Steal
space = from swap for a larger ESP, etc. Documenting that is beyond the scope of th= is email.

(2) You are booting with loader.efi and = it is in the bug-compatibility place. Some UEFI BIOSes
don't = respect or can't set BootXXXX variables set by efibootmgr(8). In that c= ase, many people
are booting from the 'compatibiltiy' loc= ation for removable=C2=A0devices. This will almost always be
a si= ngle boot scenario because you can't easily boot other systems like thi= s (well, unless 'quit'
works from the OK prompt to go to = the next one on the list, but I digress). You will see something like:

% sudo efivar --utf cfee69ad-a0de-47a9-93a8-f63106f8ae= 99-LoaderPath
cfee69ad-a0de-47a9-93a8-f63106f8ae99-LoaderPath
\EFI\= BOOT\BOOTX64.EFI
(or AA64)

when you are booting = this way. In this case the update command is:

% sudo cp /boot/loader.efi /boot/ef= i/efi/boot/boot${ARCH}.efi

to upgrade. Some people may be doing this already on s= ystems that can support efibootmgr(8) and
t= hey can of course upgrade to using that, though that's also beyond the = scope of this email.

(3) You are booting on a working system with a FreeBSD ins= talled by the boot loader, or some other
cu= stom arrangement. In that case, you'll see something like:
sudo efivar --utf cfee69ad-a0de-47a9-93a8-f63106f8ae99-= LoaderPath
cfee69ad-a0de-47a9-93a8-f63106f8ae99-LoaderPath
\EFI\FREEB= SD\LOADER.EFI

(or some other place for custom setups: I assume if you are doing t= hat you know enough to
update my instructio= ns as appropriate). To update, you'd do

% sudo cp /boot/loader.efi /boot/efi/= efi/freebsd/loader.efi

Automating all these cases is, needless to say, tricky.
=C2=A0
Part of the sequencing gets into having needed to do a
installworld to have a from-same-build content replacement
for the likes of /efi\boot\bootaa64.efi . So installkernel
already done, a reboot already done, and an installworld
having been done as well in order to have something to
copy over. (There are no pointers to alternate places to
get copies that I know of. One can find copies in the
build tree when one builds from source locally. So waiting
is not really required for that context.)

Yea, I have a branch that has an 'installboot' target that updat= es the boot
loader regardless, but given the 50-odd combinations = of ways we can
boot, it's a bit bogged down.
=C2=A0=
This also make it seem that updating ZFS pool features
should wait until after a system upgrade that spans both
the loader and kernel being ready for the new features,
even if compatibility with other systems is not a worry.

Yes. ALWAYS update your boot blocks before zpool update.<= /div>
=C2=A0
Do any of the system upgrade instructions cover such
relationships? Do any of the ZFS pool upgrade instructions
cover such? Does zpool or the like suggest such issues
when it reports there are new features that could be
enabled?

See above. :)
=C2=A0=
Part of what I expect happened here was contributed to by
a lack of being prompted to even think about the relevant
issues, leading to a pool feature upgrade that had not
been prepared for.

Yea, so far 100% of = the 'help, I upgraded and now the boot loader
can't see z= fs pool' issues have been 'I didn't upgrade my loader.efi
=
properly after zpool upgrade.' It was mandatory when we had the
OpenZFS rebase, but it is also necessary every few OpenZFS
updates from upstream as nnew features are enabled.

<= div>I'd love for someone to add this information to the handbook, and I= 'd
happily=C2=A0review such an effort. I have time to do a br= ain dump, but not
to make it pretty / formatted for asciidoctor a= nd won't for some time.

Warner
=C2= =A0
=3D=3D=3D
Mark Millard
marklmi at yahoo.com

--0000000000003ff7c405e804796f--