Re: CURRRENT snapshot won't boot due missing ZFS feature

Reply: void : "Re: CURRRENT snapshot won't boot due missing ZFS feature"
In reply to: Warner Losh : "Re: CURRRENT snapshot won't boot due missing ZFS feature"
Go to: [ bottom of page ] [ top of archives ] [ this month ]
From: Michael Gmelin <freebsd_at_grem.de>
Date: Sat, 16 Sep 2023 16:35:01 UTC
As this is the continuation of a thread I started in June,
let me top post again the solution I used back then:

"""
For completeness sake, this is how I boot 14.0 on 13.2 using
  sysutils/vm-bhyve:

  ISO=FreeBSD-14.0-CURRENT-amd64-20230608-653738e895ba-263444-bootonly.iso
  export ISO

  cd /mountpoint/for/pool/vm

  vm iso https://download.freebsd.org/snapshots/ISO-IMAGES/14.0/$ISO
  mkdir .loaders
  tar --strip-components 1 -C .loaders -xf .iso/$ISO boot/userboot.so
  mv .loaders/userboot.so .loaders/userboot14.so

  vm create test14
  sysrc -f test14/test14.conf memory=1G
  sysrc -f test14/test14.conf \
    bhyveload_loader="$(realpath .loaders/userboot14.so)"

OS installation is done the usual way (using tmux instead of cu in this
example):

  pkg install -y tmux
  sysrc -f .config/system.conf console=tmux
  vm install test14 $ISO
  tmux attach -t test14
"""

You can find thew whole thread here:
https://lists.freebsd.org/archives/freebsd-current/2023-June/003835.html


Best
Michael



On Sat, 16 Sep 2023 17:22:20 +0100
Warner Losh <imp@bsdimp.com> wrote:

> On Sat, Sep 16, 2023 at 5:11 PM Toomas Soome <tsoome@me.com> wrote:
> 
> >
> >  
> > > On 16. Sep 2023, at 18:43, Mark Millard <marklmi@yahoo.com> wrote:
> > >
> > > void <void_at_f-m.fm> wrote on
> > > Date: Sat, 16 Sep 2023 12:12:02 UTC :
> > >  
> > >> On Sat, Sep 16, 2023 at 12:55:19PM +0100, Warner Losh wrote:
> > >>  
> > >>> Yes. The boot loader comes from the host. It must know how to
> > >>> read  
> > ZFS.  
> > >>
> > >> It knows how to read zfs.  
> > >
> > > I expect Warner was indicating: you have a (efi?) loader that
> > > knows how to deal with the features listed in:
> > >
> > > sys/contrib/openzfs/cmd/zpool/compatibility.d/openzfs-2.1-freebsd
> > >
> > > being active but not with some new feature(s) listed in:
> > >
> > > sys/contrib/openzfs/cmd/zpool/compatibility.d/openzfs-2.2
> > >
> > > being active.
> > >
> > > The following are the "read-only-compatibile no" features
> > > that are new in openzfs-2.2 compared to openzfs-2.1-freebsd :
> > >
> > > blake3
> > > ednor
> > > head_errlog
> > > vdev_zaps_v2
> > >
> > > So any of those being active leads to lack of even read-only
> > > activity being compatible. (Although, the loader's subset
> > > of the potential overall activity might allow ignoring some
> > > specific "read-only-compatibile no" status examples.)
> > >
> > > For reference:
> > >
> > > # diff -u99  
> > /usr/main-src/sys/contrib/openzfs/cmd/zpool/compatibility.d/openzfs-2.1-freebsd
> > /usr/main-src/sys/contrib/openzfs/cmd/zpool/compatibility.d/openzfs-2.2
> >  
> > > ---  
> > /usr/main-src/sys/contrib/openzfs/cmd/zpool/compatibility.d/openzfs-2.1-freebsd
> > 2021-06-24 20:08:57.206621000 -0700  
> > > +++  
> > /usr/main-src/sys/contrib/openzfs/cmd/zpool/compatibility.d/openzfs-2.2
> > 2023-06-10 15:59:25.354999000 -0700  
> > > @@ -1,34 +1,40 @@
> > > -# Features supported by OpenZFS 2.1 on FreeBSD
> > > +# Features supported by OpenZFS 2.2 on Linux and FreeBSD
> > > allocation_classes
> > > async_destroy
> > > +blake3
> > > +block_cloning
> > > bookmark_v2
> > > bookmark_written
> > > bookmarks
> > > device_rebuild
> > > device_removal
> > > draid
> > > +edonr
> > > embedded_data
> > > empty_bpobj
> > > enabled_txg
> > > encryption
> > > extensible_dataset
> > > filesystem_limits
> > > +head_errlog
> > > hole_birth
> > > large_blocks
> > > large_dnode
> > > livelist
> > > log_spacemap
> > > lz4_compress
> > > multi_vdev_crash_dump
> > > obsolete_counts
> > > project_quota
> > > redacted_datasets
> > > redaction_bookmarks
> > > resilver_defer
> > > sha512
> > > skein
> > > spacemap_histogram
> > > spacemap_v2
> > > userobj_accounting
> > > +vdev_zaps_v2
> > > +zilsaxattr
> > > zpool_checkpoint
> > > zstd_compress
> > >
> > > (Last I checked, /usr/share/zfs/compatibility.d/openzfs-2.2 does
> > > not exist yet. Thus were I had the diff look.)
> > >  
> > >> On the host in question, there are many guests,
> > >> some with zfs-boot, some not, just file-based.  
> > >
> > > But with what openzfs features active vs. not active
> > > in each case?
> > >  
> > >> What the host is not, is zfs-on-root. It boots from ssd (ada0).
> > >> The vdevs are on a sas disk array.
> > >>  
> > >>> So either your bootable partitions must not have  
> > com.klarasystems:vdev_zaps_v2  
> > >>> in your BEs or you must have a new user boot. I think you can
> > >>> just  
> > install  
> > >>> the one from 14, but haven't tried it.  
> > >>
> > >> Can you briefly explain how I'd install the one from 14 please?  
> > >
> > >
> > > I do not use bhyve so I do not even know if the
> > > context is using the efi loader from a msdosfs
> > > vs. not. For efi loaders, copying from one msdosfs
> > > with a sufficient vintage to the one with the wrong
> > > vintage (replacing) is sufficient.  
> >
> > bhyve in freebsd is traditionally using /boot/userboot.so, I
> > believe.  
> 
> 
> Yes. We use the *HOSTS* (running FreeBSD 13) /boot/userboot.so to
> boot the FreeBSD 14
> image. Since we're not using the boot loader from the target image to
> load it for bhyve,
> the loader we're using has to understand the ZFS dataset that it's
> booting off of. FreeBSD
> 13's userboot.so doesn't support all the bells and whistles that the
> ZFS folks have added
> to 14.
> 
> So, either you have to turn off those features (which I got no clue
> how to do in the
> normal installer), or you have to update userboot.so to the FreeBSD 14
> version (which
> I think had a good chance of actually running on FreeBSD 13 since it
> has no 'system'
> references, which are confined to bhyveload).
> 
> Warner
> 
> 
> > >
> > > # find /boot/efi/EFI/ -print
> > > /boot/efi/EFI/
> > > /boot/efi/EFI/FREEBSD
> > > /boot/efi/EFI/FREEBSD/loader.efi
> > > /boot/efi/EFI/BOOT
> > > /boot/efi/EFI/BOOT/bootaa64.efi
> > >
> > > There may well be only:
> > >
> > > EFI/BOOT/bootaa64.efi
> > >
> > > for all I know.
> > >
> > > From an amd64 context:
> > >
> > > # find /boot/efi/EFI/ -print
> > > /boot/efi/EFI/
> > > /boot/efi/EFI/FREEBSD
> > > /boot/efi/EFI/FREEBSD/loader.efi
> > > /boot/efi/EFI/BOOT
> > > /boot/efi/EFI/BOOT/bootx64.efi
> > >
> > > There may well be only:
> > >
> > > EFI/BOOT/bootx64.efi
> > >
> > > for all I know.
> > >
> > > (I set things up to have the EFI capitalization
> > > so that referencing efi/ vs. EFI/ in my context
> > > is unique for the mount point. vs. the msdosfs
> > > directory.)
> > >
> > > ===
> > > Mark Millard
> > > marklmi at yahoo.com
> > >
> > >  
> >
> >
> >  



-- 
Michael Gmelin