From nobody Sat Sep 16 16:11:22 2023 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RnwzZ1NlCz4tcS2 for ; Sat, 16 Sep 2023 16:11:38 +0000 (UTC) (envelope-from tsoome@me.com) Received: from mr85p00im-zteg06023901.me.com (mr85p00im-zteg06023901.me.com [17.58.23.192]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4RnwzY60Whz4kgJ for ; Sat, 16 Sep 2023 16:11:37 +0000 (UTC) (envelope-from tsoome@me.com) Authentication-Results: mx1.freebsd.org; none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=me.com; s=1a1hai; t=1694880696; bh=w5TZeyWFAjqbr1TvYMTijvsoZkBpQbrLTUXkUK1qQEI=; h=Content-Type:Mime-Version:Subject:From:Date:Message-Id:To; b=cSDQakUARcfhF1o5rQD5u+6k951K9dPNtWYiftz3aabinN0I+msMeLU6HoeVRKjvF P2an/cg2MpYzQ2Ihjh/eIjDH8MuoRdb2+75I2aEzcDDPeJYiYoildsjiwbMNpMnpZf qWDUlukwypB6iGOg8GWsMw6u1jReD7EWEshn+p5wRGngOMwU8/d3FJBSUyHsbcHfSI Q202YQQnIl3W3y0pc5QLpYqb7aiiw7Kesi5Gru6b7snjYKZfNpq/mb4xaf3nh+eUG6 LfoEVZowRyKAHc44IBNLFI9Y1Fxry/FtgMWYbSTKab9V8KpTJGvyegAAJUexpQPmks SQ8u/lrYEiBMw== Received: from smtpclient.apple (mr38p00im-dlb-asmtp-mailmevip.me.com [17.57.152.18]) by mr85p00im-zteg06023901.me.com (Postfix) with ESMTPSA id 855306E0276; Sat, 16 Sep 2023 16:11:35 +0000 (UTC) Content-Type: text/plain; charset=us-ascii List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.700.6\)) Subject: Re: CURRRENT snapshot won't boot due missing ZFS feature From: Toomas Soome In-Reply-To: <7EEF3435-064D-4C3C-98E4-2B27A788DB43@yahoo.com> Date: Sat, 16 Sep 2023 19:11:22 +0300 Cc: void , Current FreeBSD Content-Transfer-Encoding: quoted-printable Message-Id: <4D51E8E6-8AF0-4773-A9BA-D53C08B744EA@me.com> References: <7EEF3435-064D-4C3C-98E4-2B27A788DB43.ref@yahoo.com> <7EEF3435-064D-4C3C-98E4-2B27A788DB43@yahoo.com> To: Mark Millard X-Mailer: Apple Mail (2.3731.700.6) X-Proofpoint-GUID: 1RvliQvTGvCA9AZPx4EnlUBqcoowHAs- X-Proofpoint-ORIG-GUID: 1RvliQvTGvCA9AZPx4EnlUBqcoowHAs- X-Proofpoint-Virus-Version: =?UTF-8?Q?vendor=3Dfsecure_engine=3D1.1.170-22c6f66c430a71ce266a39bfe25bc?= =?UTF-8?Q?2903e8d5c8f:6.0.138,18.0.816,17.11.62.513.0000000_definitions?= =?UTF-8?Q?=3D2022-01-18=5F01:2020-02-14=5F02,2022-01-18=5F01,2021-12-02?= =?UTF-8?Q?=5F01_signatures=3D0?= X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 bulkscore=0 mlxscore=0 spamscore=0 clxscore=1011 mlxlogscore=999 malwarescore=0 suspectscore=0 phishscore=0 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2212070000 definitions=main-2309160139 X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:714, ipnet:17.58.16.0/20, country:US] X-Rspamd-Queue-Id: 4RnwzY60Whz4kgJ > On 16. Sep 2023, at 18:43, Mark Millard wrote: >=20 > void wrote on > Date: Sat, 16 Sep 2023 12:12:02 UTC : >=20 >> On Sat, Sep 16, 2023 at 12:55:19PM +0100, Warner Losh wrote: >>=20 >>> Yes. The boot loader comes from the host. It must know how to read = ZFS.=20 >>=20 >> It knows how to read zfs. >=20 > I expect Warner was indicating: you have a (efi?) loader that knows > how to deal with the features listed in: >=20 > sys/contrib/openzfs/cmd/zpool/compatibility.d/openzfs-2.1-freebsd >=20 > being active but not with some new feature(s) listed in: >=20 > sys/contrib/openzfs/cmd/zpool/compatibility.d/openzfs-2.2 >=20 > being active. >=20 > The following are the "read-only-compatibile no" features > that are new in openzfs-2.2 compared to openzfs-2.1-freebsd : >=20 > blake3 > ednor > head_errlog > vdev_zaps_v2 >=20 > So any of those being active leads to lack of even read-only > activity being compatible. (Although, the loader's subset > of the potential overall activity might allow ignoring some > specific "read-only-compatibile no" status examples.) >=20 > For reference: >=20 > # diff -u99 = /usr/main-src/sys/contrib/openzfs/cmd/zpool/compatibility.d/openzfs-2.1-fr= eebsd = /usr/main-src/sys/contrib/openzfs/cmd/zpool/compatibility.d/openzfs-2.2 > --- = /usr/main-src/sys/contrib/openzfs/cmd/zpool/compatibility.d/openzfs-2.1-fr= eebsd 2021-06-24 20:08:57.206621000 -0700 > +++ = /usr/main-src/sys/contrib/openzfs/cmd/zpool/compatibility.d/openzfs-2.2 = 2023-06-10 15:59:25.354999000 -0700 > @@ -1,34 +1,40 @@ > -# Features supported by OpenZFS 2.1 on FreeBSD > +# Features supported by OpenZFS 2.2 on Linux and FreeBSD > allocation_classes > async_destroy > +blake3 > +block_cloning > bookmark_v2 > bookmark_written > bookmarks > device_rebuild > device_removal > draid > +edonr > embedded_data > empty_bpobj > enabled_txg > encryption > extensible_dataset > filesystem_limits > +head_errlog > hole_birth > large_blocks > large_dnode > livelist > log_spacemap > lz4_compress > multi_vdev_crash_dump > obsolete_counts > project_quota > redacted_datasets > redaction_bookmarks > resilver_defer > sha512 > skein > spacemap_histogram > spacemap_v2 > userobj_accounting > +vdev_zaps_v2 > +zilsaxattr > zpool_checkpoint > zstd_compress >=20 > (Last I checked, /usr/share/zfs/compatibility.d/openzfs-2.2 does > not exist yet. Thus were I had the diff look.) >=20 >> On the host in question, there are many guests, >> some with zfs-boot, some not, just file-based. >=20 > But with what openzfs features active vs. not active > in each case? >=20 >> What the host is not, is zfs-on-root. It boots from ssd (ada0). >> The vdevs are on a sas disk array. >>=20 >>> So either your bootable partitions must not have = com.klarasystems:vdev_zaps_v2 >>> in your BEs or you must have a new user boot. I think you can just = install >>> the one from 14, but haven't tried it. >>=20 >> Can you briefly explain how I'd install the one from 14 please? >=20 >=20 > I do not use bhyve so I do not even know if the > context is using the efi loader from a msdosfs > vs. not. For efi loaders, copying from one msdosfs > with a sufficient vintage to the one with the wrong > vintage (replacing) is sufficient. bhyve in freebsd is traditionally using /boot/userboot.so, I believe. rgds, toomas >=20 > For reference (from an aarch64 context): >=20 > # find /boot/efi/EFI/ -print > /boot/efi/EFI/ > /boot/efi/EFI/FREEBSD > /boot/efi/EFI/FREEBSD/loader.efi > /boot/efi/EFI/BOOT > /boot/efi/EFI/BOOT/bootaa64.efi >=20 > There may well be only: >=20 > EFI/BOOT/bootaa64.efi >=20 > for all I know. >=20 > =46rom an amd64 context: >=20 > # find /boot/efi/EFI/ -print > /boot/efi/EFI/ > /boot/efi/EFI/FREEBSD > /boot/efi/EFI/FREEBSD/loader.efi > /boot/efi/EFI/BOOT > /boot/efi/EFI/BOOT/bootx64.efi >=20 > There may well be only: >=20 > EFI/BOOT/bootx64.efi >=20 > for all I know. >=20 > (I set things up to have the EFI capitalization > so that referencing efi/ vs. EFI/ in my context > is unique for the mount point. vs. the msdosfs > directory.) >=20 > =3D=3D=3D > Mark Millard > marklmi at yahoo.com >=20 >=20