From nobody Fri Sep 06 15:32:35 2024 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4X0gGS6hM0z5VKrb for ; Fri, 06 Sep 2024 15:32:48 +0000 (UTC) (envelope-from asomers@gmail.com) Received: from mail-ot1-f46.google.com (mail-ot1-f46.google.com [209.85.210.46]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "WR4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4X0gGS1tX2z4lBH for ; Fri, 6 Sep 2024 15:32:48 +0000 (UTC) (envelope-from asomers@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=fail reason="SPF not aligned (relaxed), No valid DKIM" header.from=freebsd.org (policy=none); spf=pass (mx1.freebsd.org: domain of asomers@gmail.com designates 209.85.210.46 as permitted sender) smtp.mailfrom=asomers@gmail.com Received: by mail-ot1-f46.google.com with SMTP id 46e09a7af769-70f79f75da6so1375389a34.0 for ; Fri, 06 Sep 2024 08:32:48 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1725636767; x=1726241567; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=EZUYAKKbBG5qhmeCFbVqNb4w91uYZLEo0RLpXUTTd8o=; b=fR+o3BwetXasvL/1xB3bjb99eVdETEjIdFk6Nz3GtcCgNKcbkorMeV5qiM/V1qj3sm 7GhSXPzX6bAaT1BNuRGKmk++rd0hoqKqcF83YOXcIEGbYt2EeRu7KQ92VtmPuO124PKQ dUxNu1rfsGeKEVMPCAsSyaOrG6drvoIkU4FZL4aF37baa5mbsklpssYqOtMuvhO5sv3j CgMyaLvbnK6r/Dgn6nAVbPzNRKBJ+xrH4ln41A5be3EXAmPZirM8osQVK1+CqvFZ1/lC 16oYJNsWbQ9wYulm1ZY8XSti28D5HEH2+1Q/JebKmN4jZEspBF2K5uziqBi7aDlqZAR6 qF0Q== X-Gm-Message-State: AOJu0YwAWxSn2rMvxEyj9dZEkuxdSPpPHso+n/g0vWzNy2DLsqtGb16V swgLoCHWTF4Lp0KCX0hD3amk9b8FqlsLYgwnsdGcA68y1zz62Wgiy3ybPSkEzwVzE2mbqFpLsAY Z2tDWn0NPWXbXZUgEAW2/CUMFoc6gIw== X-Google-Smtp-Source: AGHT+IHAsiG19ocRXM67BUc9nCwL4yW7Z0hYHTZ7d8opB4QNwQO2xsw9WU7JqcIVEvyTdHW9jtyRCZBk0Geh+jRBLzU= X-Received: by 2002:a05:6808:1450:b0:3db:1661:4d64 with SMTP id 5614622812f47-3e029cfdf0bmr3697143b6e.13.1725636767083; Fri, 06 Sep 2024 08:32:47 -0700 (PDT) List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@FreeBSD.org MIME-Version: 1.0 References: <5ED5CB56-2E2A-4D83-8CDA-6D6A0719ED19@distal.com> In-Reply-To: From: Alan Somers Date: Fri, 6 Sep 2024 09:32:35 -0600 Message-ID: Subject: Re: Unable to replace drive in raidz1 To: Chris Ross Cc: FreeBSD Filesystems Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Spamd-Bar: -- X-Spamd-Result: default: False [-2.46 / 15.00]; NEURAL_HAM_LONG(-0.99)[-0.991]; NEURAL_HAM_SHORT(-0.86)[-0.860]; NEURAL_HAM_MEDIUM(-0.61)[-0.612]; FORGED_SENDER(0.30)[asomers@freebsd.org,asomers@gmail.com]; R_SPF_ALLOW(-0.20)[+ip4:209.85.128.0/17]; RWL_MAILSPIKE_GOOD(-0.10)[209.85.210.46:from]; MIME_GOOD(-0.10)[text/plain]; DMARC_POLICY_SOFTFAIL(0.10)[freebsd.org : SPF not aligned (relaxed), No valid DKIM,none]; RCVD_TLS_LAST(0.00)[]; TO_DN_ALL(0.00)[]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; MIME_TRACE(0.00)[0:+]; ARC_NA(0.00)[]; MISSING_XM_UA(0.00)[]; FREEFALL_USER(0.00)[asomers]; FREEMAIL_ENVFROM(0.00)[gmail.com]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; TO_MATCH_ENVRCPT_SOME(0.00)[]; FROM_NEQ_ENVFROM(0.00)[asomers@freebsd.org,asomers@gmail.com]; RCVD_COUNT_ONE(0.00)[1]; MLMMJ_DEST(0.00)[freebsd-fs@freebsd.org]; TAGGED_RCPT(0.00)[freebsd]; R_DKIM_NA(0.00)[]; ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US]; RCVD_IN_DNSWL_NONE(0.00)[209.85.210.46:from] X-Rspamd-Queue-Id: 4X0gGS1tX2z4lBH On Fri, Sep 6, 2024 at 8:30=E2=80=AFAM Chris Ross wrote: > > Oh, sorry. Failed to indicate versions. I was running 13.2 on amd64, an= d > while facing this problem have updated to 14.1. The current state is > 14.1, and I=E2=80=99m still seeing the same, but I started in 13.2 and sa= w this. > > > On Sep 6, 2024, at 10:24, Chris Ross wrote: > > > > Hello. I have searched the interwebs a bit and seen tell of this and o= thers > > like it, but I haven=E2=80=99t found a solution. > > > > I have a pool with three 3-disk raidz1 vols. I want to replace the dis= ks > > in the first vdev with larger disks. I=E2=80=99ve done this before, bu= t may=E2=80=99ve > > done something wrong here. > > > > I belive I used =E2=80=9Czpool remove tank da3=E2=80=9D, but command hi= story doesn=E2=80=99t > > have that. I=E2=80=99ve used many commands since I started. I might= =E2=80=99ve > > =E2=80=9Czpool offline=E2=80=9Dd the device. I=E2=80=99m sorry I don= =E2=80=99t remember the original > > command. > > > > Then replaced the disk and rebooted. This of course renumbered the > > disks. :-(. But, finding the new/replacement disk (da10), I try to > > =E2=80=9Czpool replace tank da3 da10=E2=80=9D. > > This always produces: > > > > cannot replace da3 with da10: already in replacing/spare config; wait f= or completion or use 'zpool detach=E2=80=99 > > > > Now, I can=E2=80=99t use =E2=80=9Czpool detach=E2=80=9D I=E2=80=99ve le= arned because that doesn=E2=80=99t work > > on zraid. And I can=E2=80=99t tell what it _thinks_ is happening. I e= ven > > Did a scrub of the pool and let that finish, but am still seeing the > > same. > > > > I have now: > > > > =E2=80=948<=E2=80=948<=E2=80=948<--- > >>>>>>> zpool status -v tank > > pool: tank > > state: DEGRADED > > status: One or more devices are faulted in response to persistent error= s. > > Sufficient replicas exist for the pool to continue functioning in= a > > degraded state. > > action: Replace the faulted device, or use 'zpool clear' to mark the de= vice > > repaired. > > scan: scrub repaired 0B in 17:14:03 with 0 errors on Fri Sep 6 09:08:= 34 2024 > > config: > > > > NAME STATE READ WRITE CKSUM > > tank DEGRADED 0 0 0 > > raidz1-0 DEGRADED 0 0 0 > > da3 FAULTED 0 0 0 external de= vice fault > > da1 ONLINE 0 0 0 > > da2 ONLINE 0 0 0 > > raidz1-1 ONLINE 0 0 0 > > diskid/DISK-K1GMBN9D ONLINE 0 0 0 > > diskid/DISK-K1GMEDMD ONLINE 0 0 0 > > diskid/DISK-K1GMAX1D ONLINE 0 0 0 > > raidz1-2 ONLINE 0 0 0 > > diskid/DISK-3WJDHJ2J ONLINE 0 0 0 > > diskid/DISK-3WK3G1KJ ONLINE 0 0 0 > > diskid/DISK-3WJ7ZMMJ ONLINE 0 0 0 > > > > errors: No known data errors > > =E2=80=948<=E2=80=948<=E2=80=948<=E2=80=94 > > > > I=E2=80=99ll note that before the switcharoo, the second and third vdev= =E2=80=99s listed > > =E2=80=9Cda4 da5 da6=E2=80=9D and =E2=80=9Cda7 da8 da9=E2=80=9D. The m= oving around of names caused the > > above, which again I=E2=80=99ve seen before, and am fine with. > > > > (Oh, you can see I ran a =E2=80=9Czpool offline -f=E2=80=9D on it most = recently. But > > that was in the list of things I=E2=80=99ve tried that haven=E2=80=99t = helped.) > > > > Please let me know if someone knows how I=E2=80=99ve gotten to this sta= te, and > > what I need to do to correct it. What is the "in replacing/spare confi= g=E2=80=9D > > meaning? > > > > - Chris "zpool replace" is indeed the correct command. There's no need to run "zpool offline" first, and "zpool remove" is wrong. Since "zpool replace" is still failing, are you sure that da10 is still the correct device name after all disks got renumbered? If you're sure, then you might run "zdb -l /dev/da10" to see what ZFS thinks is on that disk. -Alan