Random boot zroot/ROOT/default mounting failure with error 5

From: Vardan Melikyan <Vardan-Melikyan_at_proton.me>
Date: Tue, 09 Aug 2022 20:53:20 UTC
Hi,

Freshly installed fbsd 13.1 release on a Thinkpad x270, zfs root on geli. However, the system doesn't always boot successfully, with error:

trying to mount from zfs:zroot/ROOT/default failed with error 5.

It seems to be random, sometimes the system boots fine, oftentimes it doesn't. Listing "valid disk boot devices" shows nothing at all when the error occurs, it is like the nvme driver didn't attach correctly on time, when it does "so I see GEOM_ELI: Device nda0p4.eli created." the system boots fine, otherwise it fails to mount root.

Some debug messages:

iwm0: <Intel(R) Dual Band Wireless AC 8260> mem 0xe1100000-0xe1101fff at device 0.0 on pci2
nvme0: <Generic NVMe Device> mem 0xe1000000-0xe1003fff at device 0.0 on pci3

I'm new to FreeBSD, but the above seems a bit strange, as

iwm0@pci0:3:0:0: class=0x028000 rev=0x3a hdr=0x00 vendor=0x8086 device=0x24f3 subvendor=0x8086 subdevice=0x0010
nvme0@pci0:4:0:0: class=0x010802 rev=0x01 hdr=0x00 vendor=0x1179 device=0x0115 subvendor=0x1179 subdevice=0x0001

so iwm should be on device 0.0 on pci3, and not on pci2, and nvme on device 0.0 on pci4 instead of pci3. Actually when I load/unload ppt driver on pci0:3:0:0 to use bhyve passthrough to a Linux VM the system hangs with nvme/zfs errors, not sure if this is related to the random boot issue I'm having.

Could be a pic enumeration issue?

Any help is greatly appreciated.

Vardan