Re: ZFS and Encryption at dataset level

> On 10 Mar 2023, at 12:18, ltning-freebsd-hackers@anduin.net wrote:
> 
> On 3/8/23 12:50, Dirk-Willem van Gulik wrote:
>> Just some words of thanks & praise -- compliments to those & all the hard work of getting ZFS encryption at data set into 13.1.
>> Was an absolute breeze to configure & install - and very easy to manage ! And flexible enough to integrate with PKCS11 and HSM's.
> 
> This sounds really interesting! We've done some hackery to achieve something similar, but I'd be really interested in knowing what you've done in this respect, and how! Hope it's something you can share publicly :)

It is a bit hard to share the HSM integration; as that relies on their pkcs#11 tooling, the rather specific key references and the way their callback for N from M works.

But in short; we boot the machine without mounting the encrypted partions. The basic assumption (with some measures like trusted boot, etc) is that this environment is sufficiently safe (i.e. shells and share libraries cannot easily be hijacked) - also relative to the issues with a root empowered entity their access to kernel and the keys stored there once you have loaded them.

We keep a ZFS encryption key on the machine as a file. 

Depending on risk of loss v.s. v.s. pain of not getting up after a reboot v.s. risk of leak v.s. risk of ops-errors/process-errors and the level of 4-eye needed; we then have a few different options.  

In general, this file is then encrypted with a (second) file-key. This file key is stored encrypted against a set of public keys.  At the most strict end - we then have an HSM decrypt this key - with a 2 of M check on chipcard access & pin entry on a desktop device that is integrated with the HSM remotey. At the simplest case - we just use a OpenPGP chipcard in a reader with a pincard on a normal desktop with just GPG and OpenSC and SSH. 

Two of these below (one plain & direct; when you essentially trust the user; one with the file-key separation -- which lets us rotate that file with ease as long as there is enough 4 eye/no backup/separation-of-roles on the encrypted key file).

Below is simplified/edited a bit - we have some sudo-glue in place as `load-keys' is under zfs; we basically made a 'zfs-load-key' to curtail 'zfs' power.

Dw 

#!/bin/sh
#
# Authorized keys in ~/.ssh/authorized_keys on target
#
#   command="cat /.zfs-key; /sbin/zfs load-key -L prompt tank/enc && zfs mount -a" ssh-rsa AAAAB....
#
# Initial generate & encrypt key with:  
#
#   openssl rand -base64 128 | |tr -d '\n' |\
#    gpg -R xx -R xx -R .... -a -e --no-encrypt-to > /.zfs-key
# 
# Assumption: Unlocking user `trusted` as root; no 4 eyes, no role separation, etc
#
set -e

if [ $# = 0 ]; then
	echo Syntax: $0 hostname ...
	exit 1
fi

if ! /usr/local/bin/gpg --card-status 2> /dev/null; then
	echo Seat your personal/npi chipcard first
	exit 2
fi

FIFO=/tmp/fifo.$$
KEYID=${KEYID:-XXXXXXXXXX}

/usr/bin/mkfifo $FIFO

E=0
(
for host in $*
do
	cat $FIFO |\
		/usr/bin/ssh -F /dev/null -i $SSHKEY $SSH_EXTRA_ARGS -T -l root "$host" |\
		/usr/local/bin/gpg --quiet --default-key "$KEYID" >> $FIFO
done
) || E=$?

rm -f $FIFO
exit $E



#!/bin/sh
#
# Authorized keys in ~/.ssh/authorized_keys on target
#
#   command="/usr/local/sbin/zfs-hsm-load-key && zfs mount -a" ssh-rsa AAAAB....
#
# zfs-hsm-load-key essentially does
#	get /.zfs-key-to-keyfile decrypted via the hardware
# 	use it to decrypt /.zfs-key
# 	sudo zfs load-key the output
# 	remount and restart jails as needed.
#
# Initial generate & encrypt key with:  
#
#   openssl rand -base64 128 | |tr -d '\n' |\
#    hsm-file -enc -k - > /.zfs-key-to-keyfile
#
#   openssl rand -base64 128 | |tr -d '\n' |\
#    hsm-file -enc -k - -f /.zfs-key-to-keyfile -k /.zfs-key
# 
set -e

if [ $# != 1 ]; then
	echo Syntax: $0 hostname ...
	exit 1
fi
if ! hsmlnk -p 9731 -lrp; then
	echo Make sure the extender connected to USB
	exit 1
fi

ssh -L 9731:localhost:9731 -F /dev/null -i $SSHKEY $SSH_EXTRA_ARGS -T -l hunclk $host

exit $E