ZFS: i/o error - all block copies unavailable

From: John Doherty <bsdlists_at_jld3.net>
Date: Fri, 06 Jan 2023 19:08:20 UTC
Posting this to freebsd-fs because the problem appears to be related to 
ZFS.

I have a system running FreeBSD 12.3-RELEASE to which I have only remote 
access (about 3,300 miles from where I am).

This morning I ran "freebsd-update fetch" and "freebsd-update install" 
on it. This was to get it up to date with 12.3 in preparation for a 
further update to 13.1-RELEASE.

I rebooted the machine after "freebsd-update install" and it doesn't 
boot all the way. I connected to its IPMI and by recording the screen 
while trying to boot it, I was able to capture error messages that went 
by too fast to read otherwise:

ZFS: i/o error - all block copies unavailable
ZFS: can't read MOS of pool zp0

zp0 is not the pool from which the machine boots. It boots from a pool 
named zroot, which is a pair of mirrored SATA SSDs. zp0 is a pool of 24 
SAS disks in the chassis.

If I let the machine try to boot normally, it gets to a point at which I 
can ping both of its network interfaces (it has one 1GbE and one 10GbE) 
but sshd is apparently not running, or at any rate, attempts to connect 
via ssh just get me "Connection refused."

Via IPMI, I can get to a beastie menu. Trying to boot the machine to 
single-user mode doesn't really get me anywhere and I never get a login 
prompt. Not sure if that's because the machine is not booting or some 
IPMI oddity. The IPMI text console display stops updating at

"Loading kernel..."
x8+0xla5dbc

and that never changes.

I'm pretty well stumped about how to proceed from here. Any advice or 
insights much appreciated.