From nobody Tue Sep 06 19:52:28 2022 X-Original-To: freebsd-arm@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4MMbdh6TWnz4bYHV for ; Tue, 6 Sep 2022 19:52:40 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mail-ua1-x929.google.com (mail-ua1-x929.google.com [IPv6:2607:f8b0:4864:20::929]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4MMbdh0r28z3jyJ for ; Tue, 6 Sep 2022 19:52:40 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: by mail-ua1-x929.google.com with SMTP id d14so4711206ual.9 for ; Tue, 06 Sep 2022 12:52:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bsdimp-com.20210112.gappssmtp.com; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date; bh=e7bJnSSkMJ9j3TsjzngxeoQm2LXOqqtJD553quTWWz4=; b=C7qSdRVV3MQhkUClCOhLU9WG1ktpGKGVBeQ8HKMTOn43nVaMk2Iuvu05mDiO+O66as /dFSL09d++D2aKsO1s9ELOo75qv/NZOvqwbKDRem/3XkWWW3dyo0pynEeOi8DSS/tkPy R2NaC3Fytl+Eutstk9AhaK5C290JdETeuAnTt1y0m4Og7AS1+d6a2qZYByBhCn+dVZhN fLn2r9wAsWMUqk2x/Z351jXhqBDcft2h71YhH2n44BmjaK7AWjnlcTLe32pFNTjFni7V 1Jirv2kbrChjIheqNsJf02DwzD6O4S2P5B9hIwafKdFT19C0pwm6NbPy8j83Q92/TG2Z v2HA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date; bh=e7bJnSSkMJ9j3TsjzngxeoQm2LXOqqtJD553quTWWz4=; b=OfV56wASg3ee1Ug4mGKuKWL0e6JrMEl4gdNjAs6BWoyClZ/0tbj3cVrp9iPE8XtYz8 KMk4Nf2uN9EjpweurOzoFPX9dI4ooaOyUKB4VAsa6U8K0ujGwbBe3ISOUT9OK47C9MiH KMsKotB351DBgLTejfvFCkGgloSjgOXtoUFnHo0JA7O8FVkKDmsNoWOFelgoDkO5i0rr gA/b0cm/0DbgaHSO6CPbvEhHEMhFNXQlq8oAxyRVK6iyCaUufVsMI5TfTDplgOPaxGEj RgvKi9fcmlIiFdpGZHSMkvp7q0kSu/6XsV1kzTPSVN8P7HZMC5rGUsMlt2LrFhMTdqr2 2+1w== X-Gm-Message-State: ACgBeo1RgHeRCnD/YoBGCspkqVER+ZFq2mKFy1GYTR8DDQki+ki3Cgrz 3SAwpYwUOUGOrsYr/BNlYm+SQ0Goq9Wp+Q9XymuvPg== X-Google-Smtp-Source: AA6agR6xbZVbElvcX+dyH8uNl9fqbVwNoCAyrZNRwSfGzi3vsJjezk0Ftaq+VBlSh+QYl6dW8yI/n/xR8VIDA0QBVcY= X-Received: by 2002:ab0:15ed:0:b0:365:f250:7384 with SMTP id j42-20020ab015ed000000b00365f2507384mr69473uae.44.1662493959284; Tue, 06 Sep 2022 12:52:39 -0700 (PDT) List-Id: Porting FreeBSD to ARM processors List-Archive: https://lists.freebsd.org/archives/freebsd-arm List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-arm@freebsd.org MIME-Version: 1.0 References: <79278D35-F3CE-43F9-BB85-1D1798B6B1D4.ref@yahoo.com> <79278D35-F3CE-43F9-BB85-1D1798B6B1D4@yahoo.com> In-Reply-To: From: Warner Losh Date: Tue, 6 Sep 2022 13:52:28 -0600 Message-ID: Subject: Re: kernel update broke -current To: Mike Karels Cc: Mark Millard , freebsd-arm Content-Type: multipart/alternative; boundary="000000000000e2818c05e80789c5" X-Rspamd-Queue-Id: 4MMbdh0r28z3jyJ X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=bsdimp-com.20210112.gappssmtp.com header.s=20210112 header.b=C7qSdRVV; dmarc=none; spf=none (mx1.freebsd.org: domain of wlosh@bsdimp.com has no SPF policy when checking 2607:f8b0:4864:20::929) smtp.mailfrom=wlosh@bsdimp.com X-Spamd-Result: default: False [-3.00 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; FORGED_SENDER(0.30)[imp@bsdimp.com,wlosh@bsdimp.com]; R_DKIM_ALLOW(-0.20)[bsdimp-com.20210112.gappssmtp.com:s=20210112]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; MLMMJ_DEST(0.00)[freebsd-arm@freebsd.org]; R_SPF_NA(0.00)[no SPF record]; MIME_TRACE(0.00)[0:+,1:+,2:~]; ARC_NA(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::929:from]; DKIM_TRACE(0.00)[bsdimp-com.20210112.gappssmtp.com:+]; TO_DN_ALL(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; FROM_HAS_DN(0.00)[]; FROM_NEQ_ENVFROM(0.00)[imp@bsdimp.com,wlosh@bsdimp.com]; PREVIOUSLY_DELIVERED(0.00)[freebsd-arm@freebsd.org]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DMARC_NA(0.00)[bsdimp.com]; FREEMAIL_CC(0.00)[yahoo.com,freebsd.org] X-ThisMailContainsUnwantedMimeParts: N --000000000000e2818c05e80789c5 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Tue, Sep 6, 2022 at 10:41 AM Mike Karels wrote: > On 6 Sep 2022, at 11:13, Warner Losh wrote: > > > On Mon, Sep 5, 2022 at 4:21 PM Mark Millard wrote: > > > >> Warner Losh wrote on > >> Date: Mon, 05 Sep 2022 20:07:33 UTC : > >> > >>> On Sun, Sep 4, 2022 at 4:51 PM void wrote: > >>> > >>>> On Sun, Sep 04, 2022 at 07:31:59PM +0000, void wrote: > >>>>> > >>>>> On Sun, 4 Sep 2022, at 19:07, Warner Losh wrote: > >>>>>> You need a newer boot loader... > >>>>> > >>>>> I was thinking - getting latest -current image, booting to it then > >>>>> copying the contents of /boot from the image onto the mounted zpool > >> ... > >>>>> > >>>>> feasible? > >>>> > >>>> Seems only EFI/ needed replacing from a recent snapshot. I thought i= t > >> might > >>>> be all of /boot but I was wrong. Thank you Mark! > >>>> > >>> > >>> Yes. You'll need to update EFI/BOOT on your ESP. The rest of /boot is > >>> updated > >>> when you do an installworld. > >> > >> One of the oddities of the update sequence instructions is > >> the lack of coverage of the likes of: > >> > >> Load Path: /efi\boot\bootaa64.efi > >> > >> What step of the sequencing for the overall system update? > >> When is such an update required? (Here the example would > >> be: Before rebooting when the ZFS pool(s) possibly used to > >> boot gain new features?) When is it not required to update > >> the loader in the ESP (or analogous)? Even just knowing > >> the stage at which one should do the update indicates some > >> about when to figure out if an update is needed and so > >> prompts to be ready. > >> > > > > Today: > > > > make installworld installkernel > > The procedure documented in the Handbook (section 24.6), and which > I have always followed, is to install the kernel, reboot, and then > make installworld. Yes. And that reboot is tricky. Do not zpool upgrade at this stage. That's the only reason you *MUST* update boot blocks that we don't put in updating= . And so far, there's no reason to put something specific in UPDATING, though we did with the OpenZFS import since there was the zpool upgrade issue then for sure. Well, the other reason you'd need to upgrade your boot loader is if you create a new zpool to put root on... I think we likely need a generic note in UPDATING... > The reason is to add any new kernel facilities > used by libc or sysadmin utilities before installing components > that require them. I think that loader updates need to be an > exception to the rule, not the rule. This means that notes in > UPDATING are crucial for required loader updates (e.g. when zfs is > updated in a non-backward-compatible way. Of course, the handbook > does not mention upgrading loader.efi in the efi partition in the > section on upgrading from source. > Correct. Generally, we try very hard to not make incompatible updates to the boot loader. Generally, it can boot old and new kernels, can cope with the different scripts that might be present, etc. The zpool upgrade thing is a new wrinkle that we might want to, honestly, revisit policy around (eg, have a treat unsupported features of the zpool as a warning and give it your best shot). While we've had ZFS support forever, upstream OpenZFS seems to accumulate more new features than we did at a faster rate than before we switched. > Fwiw, I thought that incompatible zpool updates were not supposed > to be enabled until =E2=80=9Czpool upgrade=E2=80=9D is run, so that older= kernels > could access them. This seems like it show go for loaders as well. > This may be difficult during a development cycle, hence the need for > UPDATING information. > To date, that's been 100% true of every instance of this that I've investigated. UPDATING likely needs to only be updated to admonish against zpool upgrade without updating the boot blocks. Warner > Mike > > > mount -t msdos /dev/ /boot/efi > > > > and then one of three scenarios > > > > (1) You have the old boot1.efi. This will *always* be in > > ESP:EFI\BOOT\BOOT${ARCH}.EFI. > > (see uefi(8) for the values of ARCH). You can automatically detect this > for > > most installations > > that were done since about FreeBSD 12: > > % sudo efivar | grep Boot1 > > cfee69ad-a0de-47a9-93a8-f63106f8ae99-Boot1Path > > cfee69ad-a0de-47a9-93a8-f63106f8ae99-Boot1Dev > > > > In this case you would do: > > > > % sudo cp /boot/boot1.efi /boot/efi/efi/boot/boot${ARCH}.efi and you ar= e > > done. > > > > Please note: If your system was installed a long time ago, you should > also > > check to see if > > you have a LoaderPath variable set: > > % sudo efivar --utf cfee69ad-a0de-47a9-93a8-f63106f8ae99-LoaderPath > > /boot/loader.efi > > > > When Boot1 isn't set and it is set to the above value (or something > > similar), then you are > > booting with a really old copy of boot1.efi. You will need to update it > and > > may need to arrange > > for a larger ESP since FreeBSD's boot1.efifat was a tiny image that was > > hard to grow. Steal > > space from swap for a larger ESP, etc. Documenting that is beyond the > scope > > of this email. > > > > (2) You are booting with loader.efi and it is in the bug-compatibility > > place. Some UEFI BIOSes > > don't respect or can't set BootXXXX variables set by efibootmgr(8). In > that > > case, many people > > are booting from the 'compatibiltiy' location for removable devices. Th= is > > will almost always be > > a single boot scenario because you can't easily boot other systems like > > this (well, unless 'quit' > > works from the OK prompt to go to the next one on the list, but I > digress). > > You will see something like: > > > > % sudo efivar --utf cfee69ad-a0de-47a9-93a8-f63106f8ae99-LoaderPath > > cfee69ad-a0de-47a9-93a8-f63106f8ae99-LoaderPath > > \EFI\BOOT\BOOTX64.EFI > > (or AA64) > > > > when you are booting this way. In this case the update command is: > > > > % sudo cp /boot/loader.efi /boot/efi/efi/boot/boot${ARCH}.efi > > > > to upgrade. Some people may be doing this already on systems that can > > support efibootmgr(8) and > > they can of course upgrade to using that, though that's also beyond the > > scope of this email. > > > > (3) You are booting on a working system with a FreeBSD installed by the > > boot loader, or some other > > custom arrangement. In that case, you'll see something like: > > sudo efivar --utf cfee69ad-a0de-47a9-93a8-f63106f8ae99-LoaderPath > > cfee69ad-a0de-47a9-93a8-f63106f8ae99-LoaderPath > > \EFI\FREEBSD\LOADER.EFI > > > > (or some other place for custom setups: I assume if you are doing that > you > > know enough to > > update my instructions as appropriate). To update, you'd do > > > > % sudo cp /boot/loader.efi /boot/efi/efi/freebsd/loader.efi > > > > Automating all these cases is, needless to say, tricky. > > > > > >> Part of the sequencing gets into having needed to do a > >> installworld to have a from-same-build content replacement > >> for the likes of /efi\boot\bootaa64.efi . So installkernel > >> already done, a reboot already done, and an installworld > >> having been done as well in order to have something to > >> copy over. (There are no pointers to alternate places to > >> get copies that I know of. One can find copies in the > >> build tree when one builds from source locally. So waiting > >> is not really required for that context.) > >> > > > > Yea, I have a branch that has an 'installboot' target that updates the > boot > > loader regardless, but given the 50-odd combinations of ways we can > > boot, it's a bit bogged down. > > > > > >> This also make it seem that updating ZFS pool features > >> should wait until after a system upgrade that spans both > >> the loader and kernel being ready for the new features, > >> even if compatibility with other systems is not a worry. > >> > > > > Yes. ALWAYS update your boot blocks before zpool update. > > > > > >> Do any of the system upgrade instructions cover such > >> relationships? Do any of the ZFS pool upgrade instructions > >> cover such? Does zpool or the like suggest such issues > >> when it reports there are new features that could be > >> enabled? > >> > > > > See above. :) > > > > > >> Part of what I expect happened here was contributed to by > >> a lack of being prompted to even think about the relevant > >> issues, leading to a pool feature upgrade that had not > >> been prepared for. > >> > > > > Yea, so far 100% of the 'help, I upgraded and now the boot loader > > can't see zfs pool' issues have been 'I didn't upgrade my loader.efi > > properly after zpool upgrade.' It was mandatory when we had the > > OpenZFS rebase, but it is also necessary every few OpenZFS > > updates from upstream as nnew features are enabled. > > > > I'd love for someone to add this information to the handbook, and I'd > > happily review such an effort. I have time to do a brain dump, but not > > to make it pretty / formatted for asciidoctor and won't for some time. > > > > Warner > > > > > >> =3D=3D=3D > >> Mark Millard > >> marklmi at yahoo.com > >> > >> > --000000000000e2818c05e80789c5 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable


=
On Tue, Sep 6, 2022 at 10:41 AM Mike = Karels <mike@karels.net> wrote= :
On 6 Sep 2022,= at 11:13, Warner Losh wrote:

> On Mon, Sep 5, 2022 at 4:21 PM Mark Millard <marklmi@yahoo.com> wrote:
>
>> Warner Losh <imp_at_bsdimp.com> wrote on
>> Date: Mon, 05 Sep 2022 20:07:33 UTC :
>>
>>> On Sun, Sep 4, 2022 at 4:51 PM void <void@f-m.fm> wrote:
>>>
>>>> On Sun, Sep 04, 2022 at 07:31:59PM +0000, void wrote:
>>>>>
>>>>> On Sun, 4 Sep 2022, at 19:07, Warner Losh wrote:
>>>>>> You need a newer boot loader...
>>>>>
>>>>> I was thinking - getting latest -current image, bootin= g to it then
>>>>> copying the contents of /boot from the image onto the = mounted zpool
>> ...
>>>>>
>>>>> feasible?
>>>>
>>>> Seems only EFI/ needed replacing from a recent snapshot. I= thought it
>> might
>>>> be all of /boot but I was wrong. Thank you Mark!
>>>>
>>>
>>> Yes. You'll need to update EFI/BOOT on your ESP. The rest = of /boot is
>>> updated
>>> when you do an installworld.
>>
>> One of the oddities of the update sequence instructions is
>> the lack of coverage of the likes of:
>>
>> Load Path: /efi\boot\bootaa64.efi
>>
>> What step of the sequencing for the overall system update?
>> When is such an update required? (Here the example would
>> be: Before rebooting when the ZFS pool(s) possibly used to
>> boot gain new features?) When is it not required to update
>> the loader in the ESP (or analogous)? Even just knowing
>> the stage at which one should do the update indicates some
>> about when to figure out if an update is needed and so
>> prompts to be ready.
>>
>
> Today:
>
> make installworld installkernel

The procedure documented in the Handbook (section 24.6), and which
I have always followed, is to install the kernel, reboot, and then
make installworld.

Yes. And that reboot is = tricky. Do not zpool upgrade at this stage. That's
the only r= eason you *MUST* update boot blocks that we don't put in updating.
And so far, there's no reason to put something specific in UPDATI= NG,
though we did with the OpenZFS import since there was the zpo= ol upgrade
issue then for sure.

Well, th= e other reason you'd need to upgrade your boot loader is if you
create a new zpool to put root on...

I think we= likely need a generic note in UPDATING...
=C2=A0
The reason is to add any new kernel= facilities
used by libc or sysadmin utilities before installing components
that require them.=C2=A0 I think that loader updates need to be an
exception to the rule, not the rule.=C2=A0 This means that notes in
UPDATING are crucial for required loader updates (e.g. when zfs is
updated in a non-backward-compatible way.=C2=A0 Of course, the handbook
does not mention upgrading loader.efi in the efi partition in the
section on upgrading from source.

Corre= ct. Generally, we try very hard to not make incompatible updates
= to the boot loader. Generally, it can boot old and new kernels, can cope
with the different scripts that might be present, etc. The zpool up= grade
thing is a new wrinkle that we might want to, honestly, rev= isit policy
around (eg, have a treat unsupported features of the = zpool as a warning
and give it your best shot). While we've h= ad ZFS support forever, upstream
OpenZFS seems to accumulate more= new features than we did at a faster
rate than before we switche= d.
=C2=A0
Fwiw, I thought that incompatible zpool updates were not supposed
to be enabled until =E2=80=9Czpool upgrade=E2=80=9D is run, so that older k= ernels
could access them.=C2=A0 This seems like it show go for loaders as well. This may be difficult during a development cycle, hence the need for
UPDATING information.

To date, that'= ;s been 100% true of every instance of this that I've investigated.

UPDATING likely needs to only be updated to admonish = against zpool upgrade
without updating the boot blocks.

Warner
=C2=A0
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 Mike

> mount -t msdos /dev/<mumble-esp> /boot/efi
>
> and then one of three scenarios
>
> (1) You have the old boot1.efi. This will *always* be in
> ESP:EFI\BOOT\BOOT${ARCH}.EFI.
> (see uefi(8) for the values of ARCH). You can automatically detect thi= s for
> most installations
> that were done since about FreeBSD 12:
> % sudo efivar | grep Boot1
> cfee69ad-a0de-47a9-93a8-f63106f8ae99-Boot1Path
> cfee69ad-a0de-47a9-93a8-f63106f8ae99-Boot1Dev
>
> In this case you would do:
>
> % sudo cp /boot/boot1.efi /boot/efi/efi/boot/boot${ARCH}.efi and you a= re
> done.
>
> Please note: If your system was installed a long time ago, you should = also
> check to see if
> you have a LoaderPath variable set:
> % sudo efivar --utf cfee69ad-a0de-47a9-93a8-f63106f8ae99-LoaderPath > /boot/loader.efi
>
> When Boot1 isn't set and it is set to the above value (or somethin= g
> similar), then you are
> booting with a really old copy of boot1.efi. You will need to update i= t and
> may need to arrange
> for a larger ESP since FreeBSD's boot1.efifat was a tiny image tha= t was
> hard to grow. Steal
> space from swap for a larger ESP, etc. Documenting that is beyond the = scope
> of this email.
>
> (2) You are booting with loader.efi and it is in the bug-compatibility=
> place. Some UEFI BIOSes
> don't respect or can't set BootXXXX variables set by efibootmg= r(8). In that
> case, many people
> are booting from the 'compatibiltiy' location for removable de= vices. This
> will almost always be
> a single boot scenario because you can't easily boot other systems= like
> this (well, unless 'quit'
> works from the OK prompt to go to the next one on the list, but I digr= ess).
> You will see something like:
>
> % sudo efivar --utf cfee69ad-a0de-47a9-93a8-f63106f8ae99-LoaderPath > cfee69ad-a0de-47a9-93a8-f63106f8ae99-LoaderPath
> \EFI\BOOT\BOOTX64.EFI
> (or AA64)
>
> when you are booting this way. In this case the update command is:
>
> % sudo cp /boot/loader.efi /boot/efi/efi/boot/boot${ARCH}.efi
>
> to upgrade. Some people may be doing this already on systems that can<= br> > support efibootmgr(8) and
> they can of course upgrade to using that, though that's also beyon= d the
> scope of this email.
>
> (3) You are booting on a working system with a FreeBSD installed by th= e
> boot loader, or some other
> custom arrangement. In that case, you'll see something like:
> sudo efivar --utf cfee69ad-a0de-47a9-93a8-f63106f8ae99-LoaderPath
> cfee69ad-a0de-47a9-93a8-f63106f8ae99-LoaderPath
> \EFI\FREEBSD\LOADER.EFI
>
> (or some other place for custom setups: I assume if you are doing that= you
> know enough to
> update my instructions as appropriate). To update, you'd do
>
> % sudo cp /boot/loader.efi /boot/efi/efi/freebsd/loader.efi
>
> Automating all these cases is, needless to say, tricky.
>
>
>> Part of the sequencing gets into having needed to do a
>> installworld to have a from-same-build content replacement
>> for the likes of /efi\boot\bootaa64.efi . So installkernel
>> already done, a reboot already done, and an installworld
>> having been done as well in order to have something to
>> copy over. (There are no pointers to alternate places to
>> get copies that I know of. One can find copies in the
>> build tree when one builds from source locally. So waiting
>> is not really required for that context.)
>>
>
> Yea, I have a branch that has an 'installboot' target that upd= ates the boot
> loader regardless, but given the 50-odd combinations of ways we can > boot, it's a bit bogged down.
>
>
>> This also make it seem that updating ZFS pool features
>> should wait until after a system upgrade that spans both
>> the loader and kernel being ready for the new features,
>> even if compatibility with other systems is not a worry.
>>
>
> Yes. ALWAYS update your boot blocks before zpool update.
>
>
>> Do any of the system upgrade instructions cover such
>> relationships? Do any of the ZFS pool upgrade instructions
>> cover such? Does zpool or the like suggest such issues
>> when it reports there are new features that could be
>> enabled?
>>
>
> See above. :)
>
>
>> Part of what I expect happened here was contributed to by
>> a lack of being prompted to even think about the relevant
>> issues, leading to a pool feature upgrade that had not
>> been prepared for.
>>
>
> Yea, so far 100% of the 'help, I upgraded and now the boot loader<= br> > can't see zfs pool' issues have been 'I didn't upgrade= my loader.efi
> properly after zpool upgrade.' It was mandatory when we had the > OpenZFS rebase, but it is also necessary every few OpenZFS
> updates from upstream as nnew features are enabled.
>
> I'd love for someone to add this information to the handbook, and = I'd
> happily review such an effort. I have time to do a brain dump, but not=
> to make it pretty / formatted for asciidoctor and won't for some t= ime.
>
> Warner
>
>
>> =3D=3D=3D
>> Mark Millard
>> marklmi at yahoo.com
>>
>>
--000000000000e2818c05e80789c5--