amd64/148504: ZFS' zpool does not allow replacing drives to be offlined or removed

[Stuart Morgan]

>Number:         148504
>Category:       amd64
>Synopsis:       ZFS' zpool does not allow replacing drives to be offlined or removed
>Confidential:   no
>Severity:       non-critical
>Priority:       medium
>Responsible:    freebsd-amd64
>State:          open
>Quarter:        
>Keywords:       
>Date-Required:
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Sun Jul 11 23:00:18 UTC 2010
>Closed-Date:
>Last-Modified:
>Originator:     Stuart Morgan
>Release:        8.1-PRERELEASE
>Organization:
>Environment:
FreeBSD vault.internal 8.1-PRERELEASE FreeBSD 8.1-PRERELEASE #4: Thu Jul  1 09:47:23 UTC 2010     root at vault.internal:/usr/obj/usr/src/sys/vault  amd64
>Description:
Consider the ZFSv14 array as shown below:

$ zpool status
  pool: vault
 state: DEGRADED
 scrub: none requested
config:

        NAME                       STATE     READ WRITE CKSUM
        vault                      DEGRADED     0     0     0
          mirror                   ONLINE       0     0     0
            da0.eli                ONLINE       0     0     0
            da8.eli                ONLINE       0     0     0
            ad4.eli                ONLINE       0     0     0
          mirror                   DEGRADED     0     0     0
            da1.eli                ONLINE       0     0     0
            replacing              UNAVAIL      0 32.0K     0  insufficient replicas
              176228571956779269   UNAVAIL      0 34.6K     0  was /dev/ad6.eli
              1650754158582652028  UNAVAIL      0 34.6K     0  was /dev/da99.eli
            da9.eli                ONLINE       0     0     0

errors: No known data errors


It is not possible to remove 176228571956779269 and 1650754158582652028.

[root at vault ~]# zpool offline vault 176228571956779269
zcannot offline 176228571956779269: no valid replicas
[root at vault ~]# zpool offline vault 1650754158582652028
cannot offline 1650754158582652028: no valid replicas

Other combinations of detach, replace etc do not work either. The problem seems to be that the zpool binary will not remove a replacing drive when the replacement drive is removed or cannot complete the replacement operation.

Output of zdb:

vault
    version=14
    name='vault'
    state=0
    txg=174072
    pool_guid=14228437791114878229
    hostid=4269824854
    hostname='vault.internal'
    vdev_tree
        type='root'
        id=0
        guid=14228437791114878229
        children[0]
                type='mirror'
                id=0
                guid=15203813302661836458
                metaslab_array=23
                metaslab_shift=34
                ashift=12
                asize=2000394125312
                is_log=0
                children[0]
                        type='disk'
                        id=0
                        guid=7238937831460639650
                        path='/dev/da0.eli'
                        whole_disk=0
                        DTL=51
                children[1]
                        type='disk'
                        id=1
                        guid=14125934351009663627
                        path='/dev/da8.eli'
                        whole_disk=0
                        DTL=53
                children[2]
                        type='disk'
                        id=2
                        guid=8553277371515810839
                        path='/dev/ad4.eli'
                        whole_disk=0
                        DTL=169
        children[1]
                type='mirror'
                id=1
                guid=14672070090293526603
                metaslab_array=118
                metaslab_shift=34
                ashift=12
                asize=2000394125312
                is_log=0
                children[0]
                        type='disk'
                        id=0
                        guid=4020825290461516758
                        path='/dev/da1.eli'
                        whole_disk=0
                        DTL=123
                children[1]
                        type='replacing'
                        id=1
                        guid=496235429895718168
                        whole_disk=0
                        children[0]
                                type='disk'
                                id=0
                                guid=176228571956779269
                                path='/dev/ad6.eli'
                                whole_disk=0
                                not_present=1
                                DTL=54
                        children[1]
                                type='disk'
                                id=1
                                guid=1650754158582652028
                                path='/dev/da99.eli'
                                whole_disk=0
                                not_present=1
                                DTL=168
                children[2]
                        type='disk'
                        id=2
                        guid=5694121814811722857
                        path='/dev/da9.eli'
                        whole_disk=0
                        DTL=120

>How-To-Repeat:
1) Create a pool
2) Create 2 mirrored vdevs
3) Replace one of the devices with another
4) Interrupt the replacement process so that both devices are now unavailable
5) Run zpool status and attempt to remove them.
>Fix:
N/A

>Release-Note:
>Audit-Trail:
>Unformatted: