Encrypted (GELI) root on ZFS troubles

[Karl Denninger]

So here's the fun part of what I'm trying to do (and getting frustrated
with)

I have set up a GPT disk with the following setup:

=>       34  625142381  da2  GPT  (298G)
         34          6       - free -  (3.0K)
         40       1024    1  freebsd-boot  (512K)
       1064    4194304    2  freebsd-zfs  [bootme]  (2.0G)
    4195368  134217728    3  freebsd-swap  (64G)
  138413096  486729312    4  freebsd-zfs  (232G)
  625142408          7       - free -  (3.5K)

Then on freebsd-boot I have written the bootloaders.

The "bootme" filesystem has *only* the /boot directory copied over from
the rest of the system's root directory (that is, the kernel, loadables,
/boot/loader.conf, etc); that pool is called "zboot"

Partition 4 has the label "root0" on it, and thus shows up in /dev/gpt. 
I have initialized that with geli, set the boot option flag (that is,
prompt on boot) and created a pool called "root" on the resulting .eli
device and then put the system on that.  That's all ok.

Finally, I set the bootfs on that latter pool.  There is no bootfs set
on /zboot:

# zpool get bootfs zboot
NAME   PROPERTY  VALUE   SOURCE
zboot  bootfs    -       default

It is set on the root pool to the proper filesystem:

# zpool get bootfs root
NAME  PROPERTY  VALUE              SOURCE
root  bootfs    root/R/10.1-CLEAN  local

The problem is that when the system boots geli "finds" the raw device
(in this case /dev/da0p4), prompts for the password and attaches there
instead of in /dev/gpt.  The gpt label is missing --- and equally bad
the "root" pool does not appear to import at boot time either.

As a result the system tries to mount root from /zboot (even though it's
not been told to, and HAS been told where to mount off the root pool),
but there's no init in there (or anything else other than the boot
filesystem itself) and as a result I get an immediate panic.

If I boot off a different (working) zfs-based system the probe still
finds the "prompt during boot" flag on that gpt partition and asks for
the password on the device.  I can see the pool; zpool import shows it:

 pool: root
     id: 17719633931604198170
  state: ONLINE
 action: The pool can be imported using its name or numeric identifier.
 config:

        root         ONLINE
          da2p4.eli  ONLINE

Not so good.

If I detach that the device reappears in /dev/gpt; I can then attach
geli and import the pool in either location.  Putting the cache file
from the previous imported state in the zboot/boot/zfs directory doesn't
help (nor does removing the cache file entirely)

More-interestingly if I reboot the cloned system with the root pool
imported it does come back up, even though the device is the base
(da2p4.eli) rather than in the /dev/gpt directory.

Anyone know what's going on here?  And is there a way to have geli
attach during boot-time off the /dev/gpt directory instead of on the
base device partition name?

-- 
Karl Denninger
karl at denninger.net <mailto:karl at denninger.net>
/The Market Ticker/
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 2711 bytes
Desc: S/MIME Cryptographic Signature
URL: <http://lists.freebsd.org/pipermail/freebsd-stable/attachments/20141001/74069f3f/attachment.bin>