Unable to replace drive in raidz1

From: Chris Ross <cross+freebsd_at_distal.com>
Date: Fri, 06 Sep 2024 14:24:10 UTC
Hello.  I have searched the interwebs a bit and seen tell of this and others
like it, but I haven’t found a solution.

I have a pool with three 3-disk raidz1 vols.  I want to replace the disks
in the first vdev with larger disks.  I’ve done this before, but may’ve
done something wrong here.

I belive I used “zpool remove tank da3”, but command history doesn’t
have that.  I’ve used many commands since I started.  I might’ve
“zpool offline”d the device.  I’m sorry I don’t remember the original
command.

Then replaced the disk and rebooted.  This of course renumbered the
disks.  :-(. But, finding the new/replacement disk (da10), I try to
“zpool replace tank da3 da10”.
This always produces:

cannot replace da3 with da10: already in replacing/spare config; wait for completion or use 'zpool detach’

Now, I can’t use “zpool detach” I’ve learned because that doesn’t work
on zraid.  And I can’t tell what it _thinks_ is happening.  I even
Did a scrub of the pool and let that finish, but am still seeing the
same.

I have now:

—8<—8<—8<---
>>>>>> zpool status -v tank
  pool: tank
 state: DEGRADED
status: One or more devices are faulted in response to persistent errors.
	Sufficient replicas exist for the pool to continue functioning in a
	degraded state.
action: Replace the faulted device, or use 'zpool clear' to mark the device
	repaired.
  scan: scrub repaired 0B in 17:14:03 with 0 errors on Fri Sep  6 09:08:34 2024
config:

	NAME                      STATE     READ WRITE CKSUM
	tank                      DEGRADED     0     0     0
	  raidz1-0                DEGRADED     0     0     0
	    da3                   FAULTED      0     0     0  external device fault
	    da1                   ONLINE       0     0     0
	    da2                   ONLINE       0     0     0
	  raidz1-1                ONLINE       0     0     0
	    diskid/DISK-K1GMBN9D  ONLINE       0     0     0
	    diskid/DISK-K1GMEDMD  ONLINE       0     0     0
	    diskid/DISK-K1GMAX1D  ONLINE       0     0     0
	  raidz1-2                ONLINE       0     0     0
	    diskid/DISK-3WJDHJ2J  ONLINE       0     0     0
	    diskid/DISK-3WK3G1KJ  ONLINE       0     0     0
	    diskid/DISK-3WJ7ZMMJ  ONLINE       0     0     0

errors: No known data errors
—8<—8<—8<—

I’ll note that before the switcharoo, the second and third vdev’s listed
“da4 da5 da6” and “da7 da8 da9”.  The moving around of names caused the
above, which again I’ve seen before, and am fine with.

(Oh, you can see I ran a “zpool offline -f” on it most recently.  But
that was in the list of things I’ve tried that haven’t helped.)

Please let me know if someone knows how I’ve gotten to this state, and
what I need to do to correct it.  What is the "in replacing/spare config”
meaning?

                      - Chris