strange ZFS v28 states after disk upgrades/rebuilds

[Dmitry Morozovsky]

On Sun, 21 Aug 2011, Dmitry Morozovsky wrote:

> Dear colleagues,
> 
> I'm not sure how did I make this, but try to explain:
> 
> my home file server was fresh 8-stable/amd64, booted from CF, and ZFS-root with 
> 5x1.5T raidz + ssd as cache. raidz was built on raw disks ad4..ad12. ZFS v28
> 
> I'm starting to upgrade disks to Hitachi 3T, now with GPT on them. First change 
> (ad12) worked seamlessly. Next two were not, some hangs, some reboots, some 
> unables to import pool on boot (latter each time disappeard after booting 
> single into CF /bootdisk, mount -u -w /, zpool import, reboot)
> 
> after last reboot from single user I've got (BTW, resilvering seems to survive 
> reboot, but can't report proper resilvering speed)

[snip]

> 
> then disk activity stops, and zpool is locked:
> 
> -- 8< --
> root at hamster:~# zpool status -v
> load: 0.00  cmd: zpool 6300 [spa_namespace_lock] 6.02r 0.00u 0.00s 0% 2032k
> -- 8< --

after hard reset, boot proceeds normally, but now I have similarly strange 
config:

marck at hamster:~> zpool status
  pool: hm
 state: DEGRADED
 scan: resilvered 1.80G in 2h26m with 0 errors on Sun Aug 21 15:39:15 2011
config:

        NAME                                              STATE     READ WRITE CKSUM
        hm                                                DEGRADED     0     0     0
          raidz1-0                                        DEGRADED     0     0     0
            ad4                                           ONLINE       0     0     0
            ad6                                           ONLINE       0     0     0
            replacing-2                                   DEGRADED     0     0     0
              13001111841528871597                        UNAVAIL      0     0     0  was /dev/ad8
              gptid/3962b8a3-cb6d-11e0-a2b4-0007e90d0cbb  ONLINE       0     0     0
            replacing-3                                   DEGRADED     0     0     0
              4143382663317400064                         UNAVAIL      0     0     0  was /dev/ad10
              6273508279307911610                         UNAVAIL      0     0     0  was /dev/ad10
              13164605370838846626                        UNAVAIL      0     0     0  was /dev/ad10
              gptid/fabf95d4-cb4a-11e0-bdbd-0007e90d0cbb  ONLINE       0     0     0
            gptid/9faf12fa-ca5b-11e0-b59d-0007e90d0cbb    ONLINE       0     0     0
        cache
          ad14h                                           ONLINE       0     0     0

errors: No known data errors

Relevant part of zpool history:

2011-08-16.22:13:58 zpool set autoexpand=on hm
2011-08-17.01:20:14 zpool scrub hm
2011-08-17.11:20:53 zpool clear hm
2011-08-19.16:07:16 zpool replace hm ad12 /dev/gpt/hm4
2011-08-20.00:35:51 zpool replace hm ad4 gpt/hm0
2011-08-20.03:38:47 zpool clear hm
2011-08-20.17:29:31 zpool import hm
2011-08-20.17:35:12 zpool offline hm ad10
2011-08-20.17:35:23 zpool online hm ad10
2011-08-20.17:41:15 zpool scrub -s hm
2011-08-20.17:42:40 zpool offline hm 6273508279307911610
2011-08-20.17:46:40 zpool scrub -s hm
2011-08-20.17:46:59 zpool replace hm ad10 /dev/gpt/hm3
2011-08-20.20:25:51 zpool offline hm ad10
2011-08-20.20:26:44 zpool online hm ad10
2011-08-20.20:26:57 zpool online hm 6273508279307911610
2011-08-20.20:27:42 zpool online hm gpt/hm3
2011-08-20.20:28:02 zpool offline hm gpt/hm3
2011-08-20.20:28:25 zpool online hm gpt/hm3
2011-08-20.20:30:27 zpool scrub -s hm
2011-08-20.20:34:19 zpool export hm
2011-08-20.20:34:51 zpool import hm
2011-08-20.20:40:02 zpool replace hm /dev/ad10 /dev/gpt/hm3
2011-08-20.23:47:14 zpool import hm
2011-08-21.02:02:03 zpool replace hm /dev/ad8 /dev/gpt/hm2
2011-08-21.13:12:38 zpool import hm


-- 
Sincerely,
D.Marck                                     [DM5020, MCK-RIPE, DM3-RIPN]
[ FreeBSD committer:                                 marck at FreeBSD.org ]
------------------------------------------------------------------------
*** Dmitry Morozovsky --- D.Marck --- Wild Woozle --- marck at rinet.ru ***
------------------------------------------------------------------------