From nobody Fri Sep 06 17:59:26 2024 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4X0kWw0LNRz5VN9Y for ; Fri, 06 Sep 2024 17:59:40 +0000 (UTC) (envelope-from asomers@gmail.com) Received: from mail-vs1-f42.google.com (mail-vs1-f42.google.com [209.85.217.42]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "WR4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4X0kWv1FRDz4HB3 for ; Fri, 6 Sep 2024 17:59:39 +0000 (UTC) (envelope-from asomers@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=fail reason="SPF not aligned (relaxed), No valid DKIM" header.from=freebsd.org (policy=none); spf=pass (mx1.freebsd.org: domain of asomers@gmail.com designates 209.85.217.42 as permitted sender) smtp.mailfrom=asomers@gmail.com Received: by mail-vs1-f42.google.com with SMTP id ada2fe7eead31-49bc12c0041so676117137.0 for ; Fri, 06 Sep 2024 10:59:39 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1725645578; x=1726250378; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Fz1K8g7jg6+HCV9yZhd+AyuBrOmuFwH+yOqBBdJnfeI=; b=T3Y02fSG1ObzADaXe/wjS5uDSTGk/XtLqMRGUe22jhCIcx0ZvS2W5/ATFJadHnRd8J DXIrxdt1/gCzYn1Fm+zJt3CMRPChLlnbc1Wqpd6QHpH6ued0K2z1zcPwbrGl/hK+0squ VBbv3AY+pWI/h+fc37yrOSlgEyZi0mQqCFRcGLsFQ4dCuGv4p68g8fr97zmxUOPc73v3 at6NlY6G1JkAArZY6U91hnXS45FMg0NPSiDJDPf0/cLdHXDts9Pn3eJWlnVQQ+iaJYB/ R0t7+B5f00wmnRgWnhGcYH+K+4ebsJwrzTU7Ln+VYIxTAE5OGIJnLP5fJb3DYcbqu8O0 auFw== X-Forwarded-Encrypted: i=1; AJvYcCU95VTD+Ey5KHHxaA+foGfASPoI4aIeNN2GgSEJ/XTbwMwPyRXKsnrQgKaDEk02Ryypelau+oDg5pKR@freebsd.org X-Gm-Message-State: AOJu0YyAKMs/TFtEqbMfHu47AKJgVzHKuxGLt3iP0sdD/qUyoyM6j2wK nYIOa2CyTZCIQBkDALY9f+TL7f2tBkCEIr9arZgvzEpw8wLwRL+2kRKUm0nQWXqpCWMRo/o0vur eyIMk1r9zKuvAbkUyTg45vyikuUA= X-Google-Smtp-Source: AGHT+IGflI4t6Esp/PX6XcR70jXxkugLNwwc227OEZPOCR23B2m5FLyReq9zEosk5WISQyEgSvN12DCll1hddTff81o= X-Received: by 2002:a05:6102:292a:b0:498:cdb0:1d03 with SMTP id ada2fe7eead31-49bde263682mr4144787137.23.1725645578077; Fri, 06 Sep 2024 10:59:38 -0700 (PDT) List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@FreeBSD.org MIME-Version: 1.0 References: <5ED5CB56-2E2A-4D83-8CDA-6D6A0719ED19@distal.com> <6A20ABDA-9BEA-4526-94C1-5768AA564C13@distal.com> In-Reply-To: <6A20ABDA-9BEA-4526-94C1-5768AA564C13@distal.com> From: Alan Somers Date: Fri, 6 Sep 2024 11:59:26 -0600 Message-ID: Subject: Re: Unable to replace drive in raidz1 To: Chris Ross Cc: mike tancsa , FreeBSD Filesystems Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Spamd-Bar: -- X-Spamd-Result: default: False [-2.37 / 15.00]; NEURAL_HAM_LONG(-1.00)[-0.995]; NEURAL_HAM_MEDIUM(-0.75)[-0.746]; NEURAL_HAM_SHORT(-0.73)[-0.731]; FORGED_SENDER(0.30)[asomers@freebsd.org,asomers@gmail.com]; R_SPF_ALLOW(-0.20)[+ip4:209.85.128.0/17]; MIME_GOOD(-0.10)[text/plain]; DMARC_POLICY_SOFTFAIL(0.10)[freebsd.org : SPF not aligned (relaxed), No valid DKIM,none]; RCPT_COUNT_THREE(0.00)[3]; FROM_HAS_DN(0.00)[]; TO_DN_ALL(0.00)[]; RCVD_TLS_LAST(0.00)[]; ARC_NA(0.00)[]; MISSING_XM_UA(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_COUNT_ONE(0.00)[1]; ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US]; FREEMAIL_ENVFROM(0.00)[gmail.com]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; TO_MATCH_ENVRCPT_SOME(0.00)[]; FROM_NEQ_ENVFROM(0.00)[asomers@freebsd.org,asomers@gmail.com]; FREEFALL_USER(0.00)[asomers]; MLMMJ_DEST(0.00)[freebsd-fs@freebsd.org]; TAGGED_RCPT(0.00)[freebsd]; R_DKIM_NA(0.00)[]; RWL_MAILSPIKE_POSSIBLE(0.00)[209.85.217.42:from]; RCVD_IN_DNSWL_NONE(0.00)[209.85.217.42:from] X-Rspamd-Queue-Id: 4X0kWv1FRDz4HB3 On Fri, Sep 6, 2024 at 11:50=E2=80=AFAM Chris Ross wrote: > > > > > On Sep 6, 2024, at 13:02, Alan Somers wrote: > > > > This looks like you got into a split-brain situation where the disks > > have inconsistent labels. Most disks think that da10 is not a member > > of the pool, but da10 thinks that it is. Perhaps you added it as a > > spare, then physically removed it, and then did a "zpool remove" to > > remove the spare from the configuration? > > I did configure it as a spare, and remove it as a spare, but I > haven=E2=80=99t moved any disks physically since the once when I > switched it in. And this problem started before I ever tried > adding da10 into the pool as a spare. > > > If you're very very very > > sure that there is no data on da10 that you care about, you can do > > "zpool labelclear -f /dev/da10=E2=80=9D > > > I am sure, and I didn=E2=80=99t even need the -f. But, no change. > > % sudo zpool labelclear /dev/da10 > Password: > > % sudo zdb -l /dev/da10 > failed to unpack label 0 > failed to unpack label 1 > failed to unpack label 2 > failed to unpack label 3 > > % sudo zpool replace tank da3 da10 > cannot replace da3 with da10: already in replacing/spare config; wait for= completion or use 'zpool detach' > > > :-( > > - Chris If there is no label on da10, and "zpool status" doesn't show any spares, then I don't know what the problem is. It's possible that /sbin/zpool is printing an incorrect error message; it's fairly notorious for that. You could try to debug it. Other wild guesses include: * maybe da3 is the disk with the out-of-date label. You could try physically removing it before doing "zpool replace" * Since exported pools can't have active spares, you could try exporting the pool and then reimporting it. -Alan