booting bhyve off zfs - a cautionary tale

[Kurt Lidl]

Greetings all -

I'm writing to report a solution to a puzzling problem that I've had for 
a while and just
worked out a solution over the last two days.

I've been working with the boot-bhyve-from-zfs patch that has been
floating around for a while, and had good success with it initially.

My configuration is such:  the machine that run the hypervisor only has
ZFS filesystems.  As such, I just create new zvols on that machine, and
map those into bhyve as the various vtbdX devices.

I have a script that will create the zvols, then carefully partitions 
the zvol,
create a zpool on the zvol, create a few other zfs filesystems on the zpool,
and unpack the distribution .txz files into the resulting zpool.

This worked well for a while and I could create new virtual machines without
issue.  And then, after an upgrade of the hosting machine to a more recent
stable/10 image, all the new images I created would fail to start properly.
The userboot.so support would locate the kernel, and load it, but the
mounting of the root partition would fail:

--- snip, snip ---
ZFS filesystem version: 5
ZFS storage pool version: features support (5000)
Timecounters tick every 10.000 msec
Timecounter "TSC-low" frequency 1149929960 Hz quality 800
Trying to mount root from zfs:nine5/ROOT/default []...
Mounting from zfs:nine5/ROOT/default failed with error 45.

Loader variables:
   vfs.root.mountfrom=zfs:nine5/ROOT/default
--- snip, snip ---

After far too much time groveling through the boot code, I finally had the
realization of what was going on.  Since I was creating the new zpool for
the client machine under stable/10, and I couldn't boot it under stable/9,
there must have been some incompatibility introduced.

Anyway, to make a long debugging story shorter, the newest zpool-features
(enabled by default) in stable/10 prevent the stable/9 kernel from 
successfully
being about to mountroot() the zpool that created under stable/10.

It turns out there is a flag to suppress enabling all the zpool-features 
when
a new zpool is created. So I switched my script to generate the zpool,
specifying "zpool create -d ...", and then enabling the features that 
are OK for stable/9:

zpool set feature at async_destroy=enabled ${zroot}
zpool set feature at empty_bpobj=enabled ${zroot}
zpool set feature at lz4_compress=enabled ${zroot}

After these small changes with regards to the creation of the zpool,
I can now successfully boot the resulting zpool-on-a-zvol without issue.

Hopefully this information will save someone a bunch of time debugging
this problem in the future.

-Kurt