FreeBSD 12.4-R zpool remove dedup devices

From: David Christensen <dpchrist_at_holgerdanske.com>
Date: Sun, 03 Dec 2023 00:00:36 UTC
freebsd-questions:

I have a FreeBSD machine:

2023-12-02 15:48:22 toor@f2 ~
# freebsd-version -kru ; uname -a
12.4-RELEASE-p6
12.4-RELEASE-p6
12.4-RELEASE-p8
FreeBSD f2.tracy.holgerdanske.com 12.4-RELEASE-p6 FreeBSD 
12.4-RELEASE-p6 GENERIC  amd64


It has a pool built from three HDD's (data) and two SSD's (dedup):

2023-12-02 14:49:59 toor@f2 ~
# zpool status p2obs
   pool: p2obs
  state: ONLINE
   scan: scrub repaired 0 in 0 days 01:21:31 with 0 errors on Sat Oct 28 
22:47:59 2023
config:

	NAME                 STATE     READ WRITE CKSUM
	p2obs                ONLINE       0     0     0
	  raidz1-0           ONLINE       0     0     0
	    gpt/p2a.eli      ONLINE       0     0     0
	    gpt/p2d.eli      ONLINE       0     0     0
	    gpt/p2c.eli      ONLINE       0     0     0
	dedup	
	  mirror-1           ONLINE       0     0     0
	    gpt/dedup2a.eli  ONLINE       0     0     0
	    gpt/dedup2b.eli  ONLINE       0     0     0

errors: No known data errors


I am attempting to tear down the pool piecemeal and am unable to remove 
the SSD's:

2023-12-02 14:50:59 toor@f2 ~
# zpool remove p2obs gpt/dedup2a.eli gpt/dedup2b.eli
cannot remove gpt/dedup2a.eli: operation not supported on this type of pool
cannot remove gpt/dedup2b.eli: operation not supported on this type of pool

2023-12-02 14:52:53 toor@f2 ~
# zpool remove p2obs mirror gpt/dedup2a.eli gpt/dedup2b.eli
cannot remove mirror: no such device in pool
cannot remove gpt/dedup2a.eli: operation not supported on this type of pool
cannot remove gpt/dedup2b.eli: operation not supported on this type of pool

2023-12-02 14:53:03 toor@f2 ~
# zpool remove p2obs mirror-1 gpt/dedup2a.eli gpt/dedup2b.eli
cannot remove mirror-1: invalid config; all top-level vdevs must have 
the same sector size and not be raidz.
cannot remove gpt/dedup2a.eli: operation not supported on this type of pool
cannot remove gpt/dedup2b.eli: operation not supported on this type of pool

2023-12-02 14:53:09 toor@f2 ~
# zpool remove p2obs dedup gpt/dedup2a.eli gpt/dedup2b.eli
cannot remove dedup: no such device in pool
cannot remove gpt/dedup2a.eli: operation not supported on this type of pool
cannot remove gpt/dedup2b.eli: operation not supported on this type of pool

2023-12-02 14:53:26 toor@f2 ~
# zpool remove p2obs dedup mirror gpt/dedup2a.eli gpt/dedup2b.eli
cannot remove dedup: no such device in pool
cannot remove mirror: no such device in pool
cannot remove gpt/dedup2a.eli: operation not supported on this type of pool
cannot remove gpt/dedup2b.eli: operation not supported on this type of pool

2023-12-02 14:53:35 toor@f2 ~
# zpool remove p2obs dedup mirror-1 gpt/dedup2a.eli gpt/dedup2b.eli
cannot remove dedup: no such device in pool
cannot remove mirror-1: invalid config; all top-level vdevs must have 
the same sector size and not be raidz.
cannot remove gpt/dedup2a.eli: operation not supported on this type of pool
cannot remove gpt/dedup2b.eli: operation not supported on this type of pool


STFW I am starting to think that FreeBSD 12.4-R uses ZFS-On-Linux, which 
only recently added dedup devices, currently does not support removing 
dedup devices, my only option is to destroy the pool (?).


Comments or suggestions?


TIA,

David