Re: FreeBSD 13.2-STABLE can not boot from damaged mirror AND pool stuck in "resilver" state even without new devices.

From: Miroslav Lachman <000.fbsd_at_quip.cz>
Date: Sun, 07 Jan 2024 15:38:04 UTC 
On 07/01/2024 15:49, Lev Serebryakov wrote:
> On 05.01.2024 18:28, Lev Serebryakov wrote:
> 
>>     After that my server fails to boot, gtpzfsboot from second disk 
>> (ada1) reports several "zio_read error: 5" and
>>
>> ZFS: i/o error - all block copies unavailable
>> ZFS: can't read MOS of pool zroot
>>
>>     after that.
>   I've re-created pool from scratch
> 
>   zpool create znewroot ada0p3 && zfs send zroot | zfs receive znewroot 
> && zpool destroy zroot && zpool attach znewroot ada0p3 ada1p3
> 
>   but gptzfsboot still can not boot from it with same diagnostics :-(

How large are your disks in a question?

I was bitten by this not a long time ago when migrating my 2TB pool by 
zfs send to larger disks (4TB), then I see the error:

ZFS: i/o error - all block copies unavailable
ZFS: can't read MOS of pool zroot

As far as I search the internet it is caused by the boot code (later 
stage which is in a file in /boot directory) was moved too far from the 
beginning of the disk and some old BIOS cannot allow the system to 
continue booting. I am not a boot expert so my words can be wrong but I 
hope you get the point. It can be result of the system update, or zfs 
send | zfs recv. In my case the pool was unbootable in HP Miniserver Ge 
8 but boots perfectly fine in an old Supermicro with X9SCA-F board. The 
problem is not in a pool, nor disks, nor FreeBSD but in a BIOS.
I solved it by creating new mirrored pool of the size about 40GB at the 
beginning of the disks (40GB GPT partition for freebsd-zfs) where I 
installed the FreeBSD system and next freebsd-zfs partition covering the 
rest of the 4TB disks for data storage. Everything works fine.
You can also have just a small /boot partition for the boot and later 
overlayed by main ZFS pool, but it seems to me as bad for maintaining.

example of my partitions layout
# gpart show -p
=>        40  7814037088    ada0  GPT  (3.6T)
           40        1024  ada0p1  freebsd-boot  (512K)
         1064       40960  ada0p2  efi  (20M)
        42024    83886080  ada0p3  freebsd-zfs  (40G)
     83928104    20971520  ada0p4  freebsd-swap  (10G)
    104899624  7707033600  ada0p5  freebsd-zfs  (3.6T)
   7811933224     2103904          - free -  (1.0G)

It can also be avoided if your machine supports EFI boot, but my HP 
Microserver Gen 8 does not support it.

Kind regards
Miroslav Lachman