Re: Install to ZFS root is using device names hence failing when device tree is changed.

On 9/7/21 10:29 AM, Peter Jeremy wrote:
> On 2021-Sep-06 17:45:31 +0200, Karel Gardas <gardask@gmail.com> wrote:
>> just installed 14-current snapshot from 2.9. on uefi amd64 machine.
>> Installed from USB memstick which was detected as da0 into the ssd
>> hanging on usb3 in external enclosure which was detected as da1.
>>
>> ZFS root pool is then using /dev/da1p3 as swap and /dev/da1p1 as
>> /boot/efi and probably also something as root zpool.
>>
>> Anyway, expected thing happen. When I pulled out USB stick identified as
>> da0 on reboot, the drive on USB3 switch from da1 to da0 and result is
>> unbootable system with complains about various /dev/da1xx drives missing
>> for swap efi boot etc.
> 
> Can you give more details about exactly what the errors and when they
> occur during the boot cycle.  In particular:
> * Low-level boot (anything prior to the FreeBSD kernel) knows nothing
>    about da0 or da1, so any problems there are associated with your
>    BIOS config, not FreeBSD.

Low level boot is all right since kernel is booting. What's broken is 
root/zfs mount and swap enablement.

> * The swap partition will, by default, appear as a hard-wired device
>    name in /etc/fstab - that will definitely need updating.  This will
>    prevent the "swapon" working but won't prevent the boot.

ACK. Good to know.

> * ZFS doesn't care about device names - it looks for ZFS labels on all
>    possible devices.

I think you are wrong here. Let's see zpool status:


karel@freebsd:~ $ zpool status
   pool: zroot
  state: ONLINE
config:

         NAME        STATE     READ WRITE CKSUM
         zroot       ONLINE       0     0     0
           da1p4     ONLINE       0     0     0

errors: No known data errors

Now, let's reboot and see error on serial console when install da0 is 
not attached:



Root mount waiting for: usbus0
Root mount waiting for: usbus0
ugen0.5: <AXAGON AXAGON USB3.0 External HDD> at usbus0
umass0 numa-domain 0 on uhub1
umass0: <AXAGON AXAGON USB3.0 External HDD, class 0/0, rev 3.00/1.00, 
addr 4> on usbus0
umass0:  SCSI over Bulk-Only; quirks = 0x0100
umass0:9:0: Attached to scbus9
da0 at umass-sim0 bus 0 scbus9 target 0 lun 0
da0: <Samsung SSD 860 EVO 250G 0> Fixed Direct Access SPC-4 SCSI device
da0: Serial Number 2017010000C7
da0: 400.000MB/s transfers
da0: 238475MB (488397168 512 byte sectors)
da0: quirks=0x2<NO_6_BYTE>
Dual Console: Serial Primary, Video Secondary
No suitable dump device was found.
Setting hostuuid: 8cdf33eb-6866-42ae-a49d-ae7ee4c0c33c.
Setting hostid: 0xdf6467d8.
no pools available to import
swapon: /dev/da1p3: No such file or directory
Starting file system checks:
Can't open `/dev/da1p1'
/dev/da1p1: UNEXPECTED INCONSISTENCY; RUN fsck_msdosfs MANUALLY.
THE FOLLOWING FILE SYSTEM HAD AN UNEXPECTED INCONSISTENCY:
         msdosfs: /dev/da1p1 (/boot/efi)
Automatic file system check failed; help!
ERROR: ABORTING BOOT (sending SIGTERM to parent)!
2021-09-07T11:23:23.710549+02:00 - init 1 - - /bin/sh on /etc/rc 
terminated abnormally, going to single user mode
Enter full pathname of shell or RETURN for /bin/sh:


So this is problematic /efi parition, if I remove it from the /etc/fstab 
I get this boot:

....

Wow! It boots well, so you were indeed right. Checking zpool status reveals:
karel@freebsd:~ $ zpool status
   pool: zroot
  state: ONLINE
config:

         NAME        STATE     READ WRITE CKSUM
         zroot       ONLINE       0     0     0
           da0p4     ONLINE       0     0     0

errors: No known data errors
karel@freebsd:~ $

so indeed, ZFS on FreeBSD is similar to this on Solaris. On Linux I got 
different experience (e.g. /dev/sdaX hardwired and failing to boot) 
hence I've been in impression that this is also a case of FreeBSD when 
code is shared thorough OpenZFS project.

Great to know and thanks for kicking me into it. So just /efi partition 
mount is the culprit here...

Thanks!
Karel