zroot boot failure after upgrade from newest 13-stable to 14-stable

Hello arm@

Posting this here in case it helps anyone.

tl;dr: source upgrade from

stable/13-n256435-40cec659c167 GENERIC arm64
to
stable/14-n265273-803f088147d3 GENERIC arm64
on an rpi4 zroot using usb3-attached spinning rust, results in a panic, 
(the output pasted below).

The fix was to first boot into kernel.old (not sure if this is relevant, it's what 
was done in order to make sure it wasn't the kernel), where the system came up but 
nothing worked and no ethernet connection. Then rebooted and waited a while at 
the GELI prompt for usb to settle (because of the previous timeout messages). Selected 
single user and waited a short while before ctrl-D for multiuser, and it mostly came up. 
Had to recompile pkg from ports before it would reinstall anything because it 
complained with libssl1.1.1 errors.

It seems that on initial boot, the usb subsystem or the disk doesn't wake up in time.
Subsequent reboots appear to work normally.

#####

umass0:0:0: Attached to scbus0
Solaris: NOTICE: Cannot find the pool label for 'zroot'da0 at umass-sim0 bus 0 scbus0 target 0 lun 0

Mounting from zfs:zroot/ROOT/default failed with error 5; retrying for 3 more seconds
Solaris: NOTICE: Cannot find the pool label for 'zroot'
Solaris: NOTICE: Cannot find the pool label for 'zroot'
Solaris: NOTICE: Cannot find the pool label for 'zroot'

(followed by a panic)

Mounting from zfs:zroot/ROOT/default failed with error 5; retrying for 3 more seconds
Solaris: NOTICE: Cannot find the pool label for 'zroot'
Solaris: NOTICE: Cannot find the pool label for 'zroot'

GEOM_ELI: Cannot create device da0p3.eli.
Solaris: NOTICE: Cannot find the pool label for 'zroot'
GEOM_ELI: Device gpt/zfs0.eli created.
GEOM_ELI: Encryption: AES-XTS 256
GEOM_ELI:     Crypto: software
   x0: 0x0000000000000000
   x1: 0x0000000000000300  x0: 0x0000000000000000

   x1: 0x0000000000000200  x2: 0x0000000000000001

   x2: 0x0000000000000001  x3: 0xffff0000419ab480

   x3: 0xffff000041993480
   x4: 0xffff0000dfa4bb20
   x4: 0xffff0000dfa55b20  x5: 0xffff0000dfa4b780

   x5: 0xffff0000dfa55780  x6: 0x0000000000000000

   x6: 0x0000000000000000
   x7: 0x0000000000000100  x7: 0x0000000000000008

   x8: 0x0000000000000080
   x8: 0x0000000000000180
   x9: 0x0000000000000001  x9: 0x0000000000000001

  x10: 0x0000000000000000 x10: 0x0000000000000000

  x11: 0xffffa000023c52c0 x11: 0xffffa000023c52c0

  x12: 0xffffa000023a71f8
  x12: 0x0000000000000080
  x13: 0x0000000000002af8 x13: 0x0000000000002af8

  x14: 0x0000000000002857
  x14: 0x0000000000002857 x15: 0x0000000000002af8

  x15: 0x0000000000002af8 x16: 0xffff000001454758

  x17: 0xffff000000823f40 x16: 0xffff000001454758

  x18: 0xffff0000dfa4b840 x17: 0xffff000000823f40

  x19: 0x0000000000000000 x18: 0xffff0000dfa55840
  x19: 0x0000000000000000

  x20: 0x0000000000000000
  x20: 0x0000000000000000 x21: 0xffff000001454968

  x22: 0x0000000000000001
  x21: 0xffff000001454968
  x23: 0xffffa000023c52b8 x22: 0x0000000000000001
  x24: 0xffffa000023c52e0

  x25: 0xffffa0000235d200
  x23: 0xffffa000023c52b8 x26: 0x0000000000000001

  x24: 0xffffa000023c52e0
  x27: 0xffff000000f980a8 x25: 0xffffa0000235d400

  x28: 0xffff000000f9809c
  x29: 0xffff0000dfa4b840
  x26: 0x0000000000000001
   sp: 0xffff0000dfa4b840
  x27: 0xffff000000f980a8  lr: 0xffff00000143965c

  elr: 0xffff000000823f7c x28: 0xffff000000f9809c

spsr: 0x0000000020000045
  x29: 0xffff0000dfa55840 far: 0x0000000000000318

   sp: 0xffff0000dfa55840
  esr: 0x0000000096000004
   lr: 0xffff00000143965cpanic: data abort in critical section or under mutex
cpuid = 3
time = 14
KDB: stack backtrace:
#0 0xffff00000050d95c at kdb_backtrace+0x58
#1 0xffff0000004b9478 at vpanic+0x19c
#2 0xffff0000004b92d8 at panic+0x44
#3 0xffff000000897180 at data_abort+0x2b0
#4 0xffff000000874014 at handle_el1h_sync+0x14
Uptime: 14s
Resetting system ... 

U-Boot 2021.07 (May 12 2022 - 07:00:33 +0000)

--