Booting from two drives in stages, was Combining / and /boot in different ZFS pools

[Grzegorz Junka]

Hello All,
Trying this question again but rewording it differently.

Is it possible to boot the system in such a way that the kernel is 
loaded from the /boot folder from one ZFS pool, then once the kernel has 
been loaded it mounts a second ZFS pool and loads the system from it 
(the reason is that the drive with second pool needs a driver that's 
only in the kernel)?

Thanks
GrzegorzJ

On 01/01/2018 18:49, Grzegorz Junka wrote:
> My motherboard doesn't support booting from NVMe and I am trying to 
> install the loader on a USB drive.
>
> I have two ZFS pools:
>
> Two NVMe drives, each with two partitions, 1: freebsd-swap, 2: 
> freebsd-zfs; ZFS pool tank6 mirror set up on nvd0p2 and nvd1p2
> One USB drive with two partitions, 1: freebsd-boot, 2: freebsd-zfs; 
> ZFS pool tank7 using single partition da0p2
>
> The intention is to have the main root '/' on the NVMe drives and only 
> the '/boot' folder on the USB drive (since NVMe is much faster).
>
> Some of the datasets I have set up (listing dataset and mountpoint):
>
> tank6 none
> tank6/ROOT none
> tank6/ROOT/default /
> ...
>
> tank7 none
> tank7/ROOT none
> tank7/ROOT/boot /boot
> ...
>
> After creating all the datasets bsdsysinstall installed all the files 
> in all the datasets as expected. When in bsdinstall shell (memstick) 
> with altroot as /mnt, command 'zfs list' correctly lists all 
> filesystems. Also, when executing ls -l, /mnt/boot contains boot files 
> (including the kernel in /mnt/boot/kernel), and /mnt/ contains the 
> rest of the system.
>
> I have set up the bootfs as follows:
>
> gpart bootcode -b /boot/pmbr nvd0
> gpart bootcode -b /boot/pmbr nvd1
>
> gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 da0
>
> zpool set bootfs=tank6/ROOT/default tank6
> zpool set bootfs=tank7/ROOT/boot tank7
>
> However, when trying to boot from the USB I am getting an error from 
> the system loader:
>
> Can't find /boot/zfsloader
>
> FreeBSD/x86 boot
> Default: tank7/ROOT/boot:/boot/kernel/kernel
>
> My guess is that when booting up, the filesystems of tank6 and tank7 
> are not mounted and combined correctly. I tried to set tank6 as the 
> booting drive in tank7:
>
> zpool set bootfs=tank6/ROOT/default tank7
>
> but this throws an error.
>
> My question is if it's possible to boot the system from '/boot' set up 
> on a different ZFS pool than the ZFS pool containing the root '/' folder?
>
> GrzegorzJ
>