GTP ZFS boot failed after upgrading to 9.2-STABLE (can't read MOS)

[Ion-Mihai Tetcu]

Hi,


After upgrading to 
9.2-STABLE #8 r259396: Sun Dec 15 01:20:44 EET 2013 GENERIC  amd64
the system in question didn't come up:
 ZFS: i/o error - all block copies unavailable
 ZFS: can't read MOS of pool zroot
 gptszfsboot: failed to mount default pool zroot

GPTZFSBoot setup like in the wiki, except I didn't bothered with gnop.


At the time of the upgrade the boot disk was ada0.
Luckly I can boot without  problems from any of the other 2 disks.

Pool history:
1. started as a mirror over partitions on a 500GB HDD and a 1TB HDD
(still present in the system - ada1 bellow, can boot from it)
2. the 500GB was replaced by ada0, the system was able to boot from the
new disk; the pool extended via zfs online -e
3. ada1 (1TB) was replaced by ada2, the pool not extended yet
--- At this point I could boot from any of the disks.
4. zfs scrub the mirror without any error
5. upgrade
 (svn up, buildworld, buildkernel, installkernel, mergemaster -p,
 installworld, mergemaster -iU, delete-old-libs, update ports .ko
 modules, reboot)
6. The error above.
7. Boot from ada2, gpart bootcode ... ada0, still the same error.

I have snapshots of the pool from before the upgrade, so I could try to
rollback and see if it makes any difference. (The machine is in
production so I can't do it on the spot).

 # zpool list -v
NAME               SIZE  ALLOC   FREE    CAP  DEDUP  HEALTH  ALTROOT
zroot              896G   608G   288G    67%  1.00x  ONLINE  -
  mirror           896G   608G   288G      901G
    gpt/z_ES3_2T      -      -      -         -
    gpt/z_wd2T        -      -      -         -


 # gpart show -l -p
=>        34  3907029101    ada0  GPT  (1.8T)
          34           6          - free -  (3.0k)
          40         216  ada0p1  boot_wd2T  (108k)
         256    67108864  ada0p2  swap_wd2T  (32G)
    67109120  3774873600  ada0p3  z_wd2T  (1.8T)
  3841982720    65046415          - free -  (31G)

=>        34  1953525101    ada1  GPT  (931G)
          34           6          - free -  (3.0k)
          40         216  ada1p1  boot1  (108k)
         256    67108864  ada1p2  swap1  (32G)
    67109120  1885339648  ada1p3  disk1  (899G)
  1952448768     1076367          - free -  (525M)

=>        34  3907029101    ada2  GPT  (1.8T)
          34         216  ada2p1  boot_ES3_2T  (108k)
         250    67108864  ada2p2  swap_ES3_2T  (32G)
    67109114  3837788160  ada2p3  z_ES3_2T  (1.8T)
  3904897274     2131861          - free -  (1.0G)


 # zfs list -t snapshot | wc -l
   16025

 /dev/ada0:
Device Model:     WDC WD2000F9YZ-09N20L0
Serial Number:    WD-WCC1P0590651
LU WWN Device Id: 5 0014ee 25e1b7330
Firmware Version: 01.01A01
User Capacity:    2,000,398,934,016 bytes [2.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
 /dev/ada1:
Model Family:     Western Digital RE4
Device Model:     WDC WD1003FBYX-01Y7B1
Serial Number:    WD-WCAW35154447
LU WWN Device Id: 5 0014ee 2b2a4da9a
Firmware Version: 01.01V02
User Capacity:    1,000,204,886,016 bytes [1.00 TB]
 /dev/ada2:
Model Family:     Seagate Constellation ES.3
Device Model:     ST2000NM0033-9ZM175
Serial Number:    Z1X0W9SP
LU WWN Device Id: 5 000c50 064bc4572
Firmware Version: SN03
User Capacity:    2,000,398,934,016 bytes [2.00 TB]


Similar to this, it seems:
 From: Łukasz Wąsikowski <lukasz at wasikowski.net>
 To: freebsd-fs at freebsd.org, freebsd-stable at freebsd.org
 Subject: ZFS: can't read MOS of pool
 Date: Mon, 22 Jul 2013 18:18:49 +0200


Any help is greatly appreciated.

-- 
Ion-Mihai Tetcu <itetcu at FreeBSD.org>