Re: CURRRENT snapshot won't boot due missing ZFS feature

void <void_at_f-m.fm> wrote on
Date: Sat, 16 Sep 2023 12:12:02 UTC :

> On Sat, Sep 16, 2023 at 12:55:19PM +0100, Warner Losh wrote:
> 
> >Yes. The boot loader comes from the host. It must know how to read ZFS. 
> 
> It knows how to read zfs.

I expect Warner was indicating: you have a (efi?) loader that knows
how to deal with the features listed in:

sys/contrib/openzfs/cmd/zpool/compatibility.d/openzfs-2.1-freebsd

being active but not with some new feature(s) listed in:

sys/contrib/openzfs/cmd/zpool/compatibility.d/openzfs-2.2

being active.

The following are the "read-only-compatibile no" features
that are new in openzfs-2.2 compared to openzfs-2.1-freebsd :

blake3
ednor
head_errlog
vdev_zaps_v2

So any of those being active leads to lack of even read-only
activity being compatible. (Although, the loader's subset
of the potential overall activity might allow ignoring some
specific "read-only-compatibile no" status examples.)

For reference:

# diff -u99 /usr/main-src/sys/contrib/openzfs/cmd/zpool/compatibility.d/openzfs-2.1-freebsd /usr/main-src/sys/contrib/openzfs/cmd/zpool/compatibility.d/openzfs-2.2
--- /usr/main-src/sys/contrib/openzfs/cmd/zpool/compatibility.d/openzfs-2.1-freebsd 2021-06-24 20:08:57.206621000 -0700
+++ /usr/main-src/sys/contrib/openzfs/cmd/zpool/compatibility.d/openzfs-2.2 2023-06-10 15:59:25.354999000 -0700
@@ -1,34 +1,40 @@
-# Features supported by OpenZFS 2.1 on FreeBSD
+# Features supported by OpenZFS 2.2 on Linux and FreeBSD
 allocation_classes
 async_destroy
+blake3
+block_cloning
 bookmark_v2
 bookmark_written
 bookmarks
 device_rebuild
 device_removal
 draid
+edonr
 embedded_data
 empty_bpobj
 enabled_txg
 encryption
 extensible_dataset
 filesystem_limits
+head_errlog
 hole_birth
 large_blocks
 large_dnode
 livelist
 log_spacemap
 lz4_compress
 multi_vdev_crash_dump
 obsolete_counts
 project_quota
 redacted_datasets
 redaction_bookmarks
 resilver_defer
 sha512
 skein
 spacemap_histogram
 spacemap_v2
 userobj_accounting
+vdev_zaps_v2
+zilsaxattr
 zpool_checkpoint
 zstd_compress

(Last I checked, /usr/share/zfs/compatibility.d/openzfs-2.2 does
not exist yet. Thus were I had the diff look.)

> On the host in question, there are many guests,
> some with zfs-boot, some not, just file-based.

But with what openzfs features active vs. not active
in each case?

> What the host is not, is zfs-on-root. It boots from ssd (ada0).
> The vdevs are on a sas disk array.
> 
> >So either your bootable partitions must not have com.klarasystems:vdev_zaps_v2
> >in your BEs or you must have a new user boot. I think you can just install
> >the one from 14, but haven't tried it.
> 
> Can you briefly explain how I'd install the one from 14 please?


I do not use bhyve so I do not even know if the
context is using the efi loader from a msdosfs
vs. not. For efi loaders, copying from one msdosfs
with a sufficient vintage to the one with the wrong
vintage (replacing) is sufficient.

For reference (from an aarch64 context):

# find /boot/efi/EFI/ -print
/boot/efi/EFI/
/boot/efi/EFI/FREEBSD
/boot/efi/EFI/FREEBSD/loader.efi
/boot/efi/EFI/BOOT
/boot/efi/EFI/BOOT/bootaa64.efi

There may well be only:

EFI/BOOT/bootaa64.efi

for all I know.

From an amd64 context:

# find /boot/efi/EFI/ -print
/boot/efi/EFI/
/boot/efi/EFI/FREEBSD
/boot/efi/EFI/FREEBSD/loader.efi
/boot/efi/EFI/BOOT
/boot/efi/EFI/BOOT/bootx64.efi

There may well be only:

EFI/BOOT/bootx64.efi

for all I know.

(I set things up to have the EFI capitalization
so that referencing efi/ vs. EFI/ in my context
is unique for the mount point. vs. the msdosfs
directory.)

===
Mark Millard
marklmi at yahoo.com