Re: CURRRENT snapshot won't boot due missing ZFS feature

> On 16. Sep 2023, at 18:43, Mark Millard <marklmi@yahoo.com> wrote:
> 
> void <void_at_f-m.fm> wrote on
> Date: Sat, 16 Sep 2023 12:12:02 UTC :
> 
>> On Sat, Sep 16, 2023 at 12:55:19PM +0100, Warner Losh wrote:
>> 
>>> Yes. The boot loader comes from the host. It must know how to read ZFS. 
>> 
>> It knows how to read zfs.
> 
> I expect Warner was indicating: you have a (efi?) loader that knows
> how to deal with the features listed in:
> 
> sys/contrib/openzfs/cmd/zpool/compatibility.d/openzfs-2.1-freebsd
> 
> being active but not with some new feature(s) listed in:
> 
> sys/contrib/openzfs/cmd/zpool/compatibility.d/openzfs-2.2
> 
> being active.
> 
> The following are the "read-only-compatibile no" features
> that are new in openzfs-2.2 compared to openzfs-2.1-freebsd :
> 
> blake3
> ednor
> head_errlog
> vdev_zaps_v2
> 
> So any of those being active leads to lack of even read-only
> activity being compatible. (Although, the loader's subset
> of the potential overall activity might allow ignoring some
> specific "read-only-compatibile no" status examples.)
> 
> For reference:
> 
> # diff -u99 /usr/main-src/sys/contrib/openzfs/cmd/zpool/compatibility.d/openzfs-2.1-freebsd /usr/main-src/sys/contrib/openzfs/cmd/zpool/compatibility.d/openzfs-2.2
> --- /usr/main-src/sys/contrib/openzfs/cmd/zpool/compatibility.d/openzfs-2.1-freebsd 2021-06-24 20:08:57.206621000 -0700
> +++ /usr/main-src/sys/contrib/openzfs/cmd/zpool/compatibility.d/openzfs-2.2 2023-06-10 15:59:25.354999000 -0700
> @@ -1,34 +1,40 @@
> -# Features supported by OpenZFS 2.1 on FreeBSD
> +# Features supported by OpenZFS 2.2 on Linux and FreeBSD
> allocation_classes
> async_destroy
> +blake3
> +block_cloning
> bookmark_v2
> bookmark_written
> bookmarks
> device_rebuild
> device_removal
> draid
> +edonr
> embedded_data
> empty_bpobj
> enabled_txg
> encryption
> extensible_dataset
> filesystem_limits
> +head_errlog
> hole_birth
> large_blocks
> large_dnode
> livelist
> log_spacemap
> lz4_compress
> multi_vdev_crash_dump
> obsolete_counts
> project_quota
> redacted_datasets
> redaction_bookmarks
> resilver_defer
> sha512
> skein
> spacemap_histogram
> spacemap_v2
> userobj_accounting
> +vdev_zaps_v2
> +zilsaxattr
> zpool_checkpoint
> zstd_compress
> 
> (Last I checked, /usr/share/zfs/compatibility.d/openzfs-2.2 does
> not exist yet. Thus were I had the diff look.)
> 
>> On the host in question, there are many guests,
>> some with zfs-boot, some not, just file-based.
> 
> But with what openzfs features active vs. not active
> in each case?
> 
>> What the host is not, is zfs-on-root. It boots from ssd (ada0).
>> The vdevs are on a sas disk array.
>> 
>>> So either your bootable partitions must not have com.klarasystems:vdev_zaps_v2
>>> in your BEs or you must have a new user boot. I think you can just install
>>> the one from 14, but haven't tried it.
>> 
>> Can you briefly explain how I'd install the one from 14 please?
> 
> 
> I do not use bhyve so I do not even know if the
> context is using the efi loader from a msdosfs
> vs. not. For efi loaders, copying from one msdosfs
> with a sufficient vintage to the one with the wrong
> vintage (replacing) is sufficient.

bhyve in freebsd is traditionally using /boot/userboot.so, I believe.

rgds,
toomas


> 
> For reference (from an aarch64 context):
> 
> # find /boot/efi/EFI/ -print
> /boot/efi/EFI/
> /boot/efi/EFI/FREEBSD
> /boot/efi/EFI/FREEBSD/loader.efi
> /boot/efi/EFI/BOOT
> /boot/efi/EFI/BOOT/bootaa64.efi
> 
> There may well be only:
> 
> EFI/BOOT/bootaa64.efi
> 
> for all I know.
> 
> From an amd64 context:
> 
> # find /boot/efi/EFI/ -print
> /boot/efi/EFI/
> /boot/efi/EFI/FREEBSD
> /boot/efi/EFI/FREEBSD/loader.efi
> /boot/efi/EFI/BOOT
> /boot/efi/EFI/BOOT/bootx64.efi
> 
> There may well be only:
> 
> EFI/BOOT/bootx64.efi
> 
> for all I know.
> 
> (I set things up to have the EFI capitalization
> so that referencing efi/ vs. EFI/ in my context
> is unique for the mount point. vs. the msdosfs
> directory.)
> 
> ===
> Mark Millard
> marklmi at yahoo.com
> 
>