eli encrypted providers for zfs raidz1

[Ben Woods]

On Sat, 17 Nov 2018 at 06:36, Marco Steinbach <coco at executive-computing.de>
wrote:

> I'm trying to wrap my head around on how to avoid a zpool resilver on a
> non-booting ZFS raidz1 of off four equally sized (GPT) partitions on
> four distinct drives using eli for encyption.
>
> IOW: I do struggle with finding a way to attach all the
> providers such, that ZFS does not initiate a resilver due to the
> providers being attached sequentially.
>


ZFS doesn't auto-mount by itself - there are other elements at work doing
this.
1. During boot, the /etc/rc.d/zfs script will call "zfs mount -va" to
automount any available datasets. Note this only happens once during boot
(not continuously after boot), and it happens *after* the /etc/rc.d/geli
script has already attached any geli providers.
2. The daemon zfsd(8) was newly introduced in FreeBSD 11.0 will online any
vdevs as new GEOM devices appear. It also handles other devctl(4) events.
This daemon is not enabled by default - you have to specify
zfsd_enable="YES" in /etc/rc.conf if you want this behaviour.

I suspect your problem is the second items here. Have you enabled it?


I've created and initialized the partitions as follows (generic
> notation, comments on chosen encryption algo welome, since this
> testing setup lacks AES-NI):
> # gpart create -s gpt /dev/ada[2-5]
> # gpart add -t freebsd-zfs /dev/ada[2-5]
> # geli init -e AES-CBC -l 128 /dev/ada[2-5]p1
>

I suspect you are just using the [2-5] as shorthand in your email here, and
not running this exact command on FreeBSD 11.2.
For reference, geli has been updated on FreeBSD 12.0 such that you could
run this exact command to init multiple providers in a single command (they
will all use the same password/file with unique salt).


> Then I attached the geli partitions like so:
> # geli attach /dev/ada[2-5]p1
>

Same for geli attach - this could all be done in the one command like this
on FreeBSD 12.0.


> And finally created a raidz1 spanning all four partitions:
> # zpool create u0001 raidz1 /dev/ada[2-5]p1.eli
>
> That works flawlessly. And naturally, after a reboot none of the
> encrypted devices is available to the zpool then, unless I attach them.
>

You can configure the /etc/rc.d/geli script to attach additional devices
(e.g. non-root ZFS) during userland boot. You can add the following to
/etc/rc.conf:
geli_devices="ada2p1 ada3p1 ada4p1 ada5p1"
This will prompt you to type your password in 4 times during userland boot,
and each would be attached one at a time.

In FreeBSD 12.0, you can make this only prompt you for your password once
by putting the following in /etc/rc.conf:
geli_groups="u0001"
geli_u0001_devices="ada2p1 ada3p1 ada4p1 ada5p1"


>
> Doing so using geli attach requires me to attach them sequentially,
> which then results in ZFS resilvering the pool.
>

This would only occur if there was something automatically onlining the
vdevs and mounting the datasets, such as zfsd.


> So, here's my questions:
>
> 1. Is the inavoidable resilver intended behaviour based on current
> implementation, or am I missing something ?
>

I suspect this is zfsd working against you in this instance.

2. In case I use a bootable zfsroot (cudos to allanjude@, I highly
> recommend his BSDCan presentations on the matter), is there a way to
> hand over the zfsroot passphrase to eli for automatically attaching
> other providers ?
>

The geli attach for root ZFS devices would be handled by the boot loader,
whilst the geli attach for non-root ZFS devices should be handled by the
userland /etc/rc.d/geli script.
Whilst you will need to type in your password separately for root and
non-root devices, FreeBSD 12.0 at least makes it so you don't have to type
the password multiple times for non-root devices.
There might be a way to get the boot loader to handle the non-root devices
also - I don't have experience with this.


> Please note, that I'd like to stick as close as possible to what the
> base system offers for this use-case.
>
> MfG CoCo
>


Good luck!

--
From: Benjamin Woods
woodsb02 at gmail.com