Geli prompts on gptzfsboot (Was:: Serious ZFS Bootcode Problem (GPT NON-UEFI -- RESOLVED)

[Ian Lepore]

On Sun, 2019-02-10 at 12:35 -0600, Karl Denninger wrote:
> On 2/10/2019 12:01, Ian Lepore wrote:
> > On Sun, 2019-02-10 at 11:54 -0600, Karl Denninger wrote:
> > > On 2/10/2019 11:50, Ian Lepore wrote:
> > > > On Sun, 2019-02-10 at 11:37 -0600, Karl Denninger wrote:
> > > > 
> > > > > [...]
> > > > > 
> > > > > BTW am I correct that gptzfsboot did *not* get the ability to
> > > > > read
> > > > > geli-encrypted pools in 12.0?  The UEFI loader does know how
> > > > > (which I'm
> > > > > using on my laptop) but I was under the impression that for
> > > > > non-
> > > > > UEFI
> > > > > systems you still needed the unencrypted boot partition from
> > > > > which to
> > > > > load the kernel.
> > > > > 
> > > > 
> > > > Nope, that's not correct. GELI support was added to the boot
> > > > and
> > > > loader
> > > > programs for both ufs and zfs in freebsd 12. You must set the
> > > > geli
> > > > '-g' 
> > > > option to be prompted for the passphrase while booting (this is
> > > > separate from the '-b' flag that enables mounting the encrypted
> > > > partition as the rootfs). You can use "geli configure -g" to
> > > > turn
> > > > on
> > > > the flag on any existing geli partition.
> > > > 
> > > > -- Ian
> > > 
> > > Excellent - this will eliminate the need for me to run down the
> > > foot-shooting that occurred in my update script since the
> > > unencrypted
> > > kernel partition is no longer needed at all.  That also
> > > significantly
> > > reduces the attack surface on such a machine (although you could
> > > still
> > > tamper with the contents of freebsd-boot of course.)
> > > 
> > > The "-g" flag I knew about from experience in putting 12 on my X1
> > > Carbon
> > > (which works really well incidentally; the only issue I'm aware
> > > of is
> > > that there's no 5Ghz WiFi support.)
> > > 
> > 
> > One thing that is rather unfortunate... if you have multiple geli
> > encrypted partitions that all have the same passphrase, you will be
> > required to enter that passphrase twice while booting -- once in
> > gpt[zfs]boot, then again during kernel startup when the rest of the
> > drives/partitions get tasted by geom. This is because APIs within
> > the
> > boot process got changed to pass keys instead of the passphrase
> > itself
> > from one stage of booting to the next, and the fallout of that is
> > the
> > key for the rootfs is available to the kernel for mountroot, but
> > the
> > passphrase is not available to the system when geom is probing all
> > the
> > devices, so you get prompted for it again.
> > 
> > -- Ian
> 
> Let me see if I understand this before I do it then... :-)
> 
> I have the following layout:
> 
> 1. Two SSDs that contain the OS as a two-provider ZFS pool, which has
> "-b" set on both members; I get the "GELI Passphrase:" prompt from
> the
> loader and those two providers (along with encrypted swap) attach
> early
> in the boot process.  The same SSDs contain a mirrored non-encrypted
> pool that has /boot (and only /boot) on it because previously you
> couldn't boot from an EFI-encrypted pool at all.
> 
> Thus:
> 
> [\u at NewFS /root]# gpart show da1
> =>       34  468862061  da1  GPT  (224G)
>          34       2014       - free -  (1.0M)
>        2048       1024    1  freebsd-boot  (512K)
>        3072       1024       - free -  (512K)
>        4096   20971520    2  freebsd-zfs  [bootme]  (10G)
>    20975616  134217728    3  freebsd-swap  (64G)
>   155193344  313667584    4  freebsd-zfs  (150G)
>   468860928       1167       - free -  (584K)
> 
> There is of course a "da2" that is identical.  The actual encrypted
> root
> pool is on partition 4 with "-b" set at present.  I get prompted from
> loader as a result after the unencrypted partition (#2) boots.
> 
> 2. Multiple additional "user space" pools on a bunch of other disks.
> 
> Right now #2 is using geli groups.  Prior to 12.0 they were handled
> using a custom /etc/rc.d script I wrote that did basically the same
> thing that geli groups does because all use the same passphrase and
> entering the same thing over and over on a boot was a pain in the
> butt. 
> It prompted cleanly with no echo, took a password and then iterated
> over
> a list of devices attaching them one at a time.  That requirement is
> now
> gone with geli groups, which is nice since mergemaster always
> complained
> about it being a "non-standard" thing; it *had* to go in /etc/rc.d
> and
> not in /usr/etc/rc.d else I couldn't get it to run early enough --
> unfortunately.
> 
> So if I remove the non-encrypted freebsd-zfs mirror that the system
> boots from in favor of setting "-g" on the root pool (both providers)
> gptzfsboot will find and prompt for the password to boot before
> loader
> gets invoked at all, much like the EFI loader does.  That's good. 
> (My
> assumption is that the "-g" is sufficient; I don't need (or want)
> "bootme" set -- correct?)
> 
> /However, /once the kernel boots somewhere in the mishmash of boot-
> time
> messages, and probably not where it's instantly obvious nor where it
> will halt the cascade display on the console, I'm going to get asked
> for
> that passphrase again?  I assume I want to remove
> 'geom_eli_passphrase_prompt="YES"' from loader.conf as well -- or
> would
> leaving it in there save me from the prompt that's hard to find in
> the
> cascade?
> 
> Or, even better, would that situation of a double-prompt only apply
> if I
> had "-b" set on something /other than /the boot device pool vdevs (I
> don't -- those are handled by #2 for this exact reason.)
> 

I think at this point I have to ease out of the conversation, because I
know almost nothing about zfs, despite having somehow managed to add
geli support to the zfs code in loader. I did so without understanding
zfs in any way, because I added the support at a more generic "disk
drive support" layer in loader, and did all my testing using automated
scripts Alan and Warner created to test zfs booting using qemu.

-- Ian