zfs remove vdev functionality

[Daniel Kalchev]

I was thinking on how to implement vdev removal on ZFS and came to this 
idea:

If we can have an per-vdev flag that prohibits new allocations on the 
vdev, but permits reads and frees - then we could mark so  the vdev we 
intend to remove form the zpool and issue an scrub-like command that 
will rewrite all blocks allocated form that particular vdev. Since ZFS 
is COW, this will effectively move all blocks off that "no write" vdev 
and we can now detach it.

All this could be implemented with the new ZFS feature flags, so no 
version bumps etc are necessary.

Is there something I didn't think of?

Daniel