Encrypted (GELI) root on ZFS troubles

[Ben Morrow]

Quoth Karl Denninger <karl at denninger.net>:
> 
> The problem is that when the system boots geli "finds" the raw device
> (in this case /dev/da0p4), prompts for the password and attaches there
> instead of in /dev/gpt.

Since you're using ZFS, that doesn't matter (as you noticed below). ZFS
treats the names that devices were added to the pool with as hints only,
and actually searches through all available disks every time it imports
a pool, looking for volumes it recognises.

> The gpt label is missing --- and equally bad
> the "root" pool does not appear to import at boot time either.
> 
> As a result the system tries to mount root from /zboot (even though it's
> not been told to, and HAS been told where to mount off the root pool),
> but there's no init in there (or anything else other than the boot
> filesystem itself) and as a result I get an immediate panic.
> 
> If I boot off a different (working) zfs-based system the probe still
> finds the "prompt during boot" flag on that gpt partition and asks for
> the password on the device.  I can see the pool; zpool import shows it:
> 
>  pool: root
>      id: 17719633931604198170
>   state: ONLINE
>  action: The pool can be imported using its name or numeric identifier.
>  config:
> 
>         root         ONLINE
>           da2p4.eli  ONLINE

You appear to have exported the pool? I don't think you want to that,
since, as you've found, the kernel will not (must not) automatically
import pools at boot time.

> More-interestingly if I reboot the cloned system with the root pool
> imported it does come back up, even though the device is the base
> (da2p4.eli) rather than in the /dev/gpt directory.

Is there any reason not to simply leave the pool imported when you
reboot? The geli device will detach when the system shuts down, after
ZFS has finished flushing data; when it reattaches at boot time, ZFS
will see an imported pool and make it available.

> Anyone know what's going on here?  And is there a way to have geli
> attach during boot-time off the /dev/gpt directory instead of on the
> base device partition name?

I don't have a definitive answer to that, but I strongly suspect not.
The only place for the information to come from would be
/boot/loader.conf, and there's no mention of an appropriate tunable in
geli(8). But, as you already found out, it doesn't matter.

Ben