ZFS zpool replace issues (if vfs.zfs.debug is not set)

[Nikolay Denev]

Hello,

I have 4 drives (ad4,ad5,ad6 and ad7) raidz1 pool with GPT partitions like this:

nas# gpart show ad6
=>        34  1953525101  ad6  GPT  (932G)
          34         128    1  freebsd-boot  (64K)
         162         350       - free -  (175K)
         512      400000    2  freebsd-swap  (195M)
      400512  1953124623    3  freebsd-zfs  (931G)

The freebsd-swap partition used to be UFS partition for /boot before
gptzfsboot was able to boot from raidz pools.
Then I've decided to get rid of these 195MB partitions by offlining
the devices in my pool one by one, deleting the swap and zfs partitons,
creating one new zfs partition that uses all the available space, 
doing zpool replace and wait to resilver.

So what I did was :
   
   zpool offline zfs ad4p3 (so i can modify the disk with gpart)
   gpart delete -i2 ad4
   gpart delete -i3 ad4
   gpart add -b 512 -s 1953524623 -t freebsd-zfs ad4

and then when I've tried :
   
   zpool replace zfs ad4p3 ad4p2 

I got "permission denied" error.
Strangely when I've enabled the sysctl "vfs.zfs.debug" and tried again
the zpool replace succeeded. And this happened on all of the drives I've replaced so far.
 
The other probably unrelated issue is that when I've started replacing one of the vdevs in the pool
another vdev suddenly changed it's name from adXp3 to gptid/64149f82-44f0-11de-ae31-001ff3fc24c1
This seems really strange especially because I was not touching this particular vdev...



--
Regards,
Nikolay Denev