From nobody Fri Sep 06 19:16:39 2024 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4X0mDs1wkJz5VXTp for ; Fri, 06 Sep 2024 19:16:45 +0000 (UTC) (envelope-from morganw@gmail.com) Received: from mail-yb1-xb34.google.com (mail-yb1-xb34.google.com [IPv6:2607:f8b0:4864:20::b34]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "WR4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4X0mDr2nmXz4ms7 for ; Fri, 6 Sep 2024 19:16:44 +0000 (UTC) (envelope-from morganw@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20230601 header.b=nI94q35h; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of morganw@gmail.com designates 2607:f8b0:4864:20::b34 as permitted sender) smtp.mailfrom=morganw@gmail.com Received: by mail-yb1-xb34.google.com with SMTP id 3f1490d57ef6-e1d4368ad91so291233276.0 for ; Fri, 06 Sep 2024 12:16:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1725650203; x=1726255003; darn=freebsd.org; h=content-transfer-encoding:mime-version:message-id:references :in-reply-to:user-agent:subject:cc:to:from:date:from:to:cc:subject :date:message-id:reply-to; bh=YQRPyN5Lr6f5iBwMT31HjdHtgVfmm0u6a+0rFTb7n54=; b=nI94q35hxKsRs3AiLT4kWQbWt/yQl+9VyPQ9Od9vzsAyDLbialMKs83RpLrKtgLejI LpjM5+oGOjizIGNwFS+X8vEwU6LJqdPRCJhlK4OvmZu6pgsimDvwVN77kI6HNNAf1n+Q p9DdabaVaxfqu9PbCFbLWWV8ac2fYHMUQRzZ1DBGfLFx4SlwDxJq5BEySxPY7HYPdeVh oSbGf3nOAdO6ViWsjNpuiizbxQM4rK98GQShkw4v3opyj/jgEtfX3ci2wgOVvCAKKHXI dT6CYSn87031Dyq2XOa85qNQkCrtfHdPPA3nIbx52ERBbjGcm3ibS5bSXHnxUH8REkrG NpzA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1725650203; x=1726255003; h=content-transfer-encoding:mime-version:message-id:references :in-reply-to:user-agent:subject:cc:to:from:date:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=YQRPyN5Lr6f5iBwMT31HjdHtgVfmm0u6a+0rFTb7n54=; b=Fp8xOT19EIvmJFxLTiwGSgEJ4H9muy6vS6V1WIPy+3UUpDXQ/nt6PnokD0+TVPYEDe y9v1oYXbimNUcptH2UlNxZQbDSIBz2Gempt/lAlDRCwy62znauWnMJ+gpyNiCiZcprqw 5MfVBXK6U9i6ny5+E9yLr9q6YU0Savzjs9J01e6PSvXMB09J6DTq+uDtciima+JBeIvT xlITXqe10Wy/g1DsZAj58NCl29gNyyKsbKpqGdYnPPqH2nf3Njy6ybSA8GjgwhH/wF1n DswNSCU6KMsk+OwBlNoKK1iDqZEXgv5ERyq1CfMdS1CdrQOEf7p1Dty4rGLIePX+iJ96 yDSg== X-Gm-Message-State: AOJu0YyqdfeCMlCtsGVb/SZF2umQJd+buH9M36aPFV80UHuuzc6sx/Lm lzajMJM5jpFKMDK63JszNKPo2jZraatKO/Rde8KSkL1TrcWgn7Ln X-Google-Smtp-Source: AGHT+IFT03FCDG+mHKAWoxKc3eVS5aykd/F+XQMuqKt7BNrShJYZv3DcETwWZ4/q5rGlipSflm1w0g== X-Received: by 2002:a05:690c:30c:b0:6d6:b852:ed2e with SMTP id 00721157ae682-6db25f4a488mr94546727b3.2.1725650203451; Fri, 06 Sep 2024 12:16:43 -0700 (PDT) Received: from ?IPv6:::1? ([2600:381:642f:e2a:d8f2:e3e8:b579:a70b]) by smtp.gmail.com with ESMTPSA id 00721157ae682-6db56337dd3sm1056077b3.42.2024.09.06.12.16.41 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Fri, 06 Sep 2024 12:16:43 -0700 (PDT) Date: Fri, 06 Sep 2024 14:16:39 -0500 From: Wes Morgan To: Chris Ross CC: freebsd-fs@freebsd.org Subject: Re: Unable to replace drive in raidz1 User-Agent: K-9 Mail for Android In-Reply-To: References: <5ED5CB56-2E2A-4D83-8CDA-6D6A0719ED19@distal.com> <6A20ABDA-9BEA-4526-94C1-5768AA564C13@distal.com> <0CF1E2D7-6C82-4A8B-82C3-A5BF1ED939CF@distal.com> <29003A7C-745D-4A06-8558-AE64310813EA@distal.com> <42346193-AD06-4D26-B0C6-4392953D21A3@gmail.com> Message-ID: List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@FreeBSD.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spamd-Bar: --- X-Spamd-Result: default: False [-4.00 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20230601]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; MIME_GOOD(-0.10)[text/plain]; RCVD_TLS_LAST(0.00)[]; FROM_HAS_DN(0.00)[]; ARC_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; RCPT_COUNT_TWO(0.00)[2]; FREEMAIL_FROM(0.00)[gmail.com]; TO_DN_SOME(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; FROM_EQ_ENVFROM(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; MID_RHS_MATCH_FROM(0.00)[]; MLMMJ_DEST(0.00)[freebsd-fs@freebsd.org]; TAGGED_RCPT(0.00)[freebsd]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::b34:from] X-Rspamd-Queue-Id: 4X0mDr2nmXz4ms7 On September 6, 2024 1:54:32 PM CDT, Chris Ross wrote: > > >> On Sep 6, 2024, at 14:39, Wes Morgan wrote: >>=20 >>=20 >> You should make the changes to your /boot/loader=2Econf as suggested ea= rlier by Freddie Cash and reboot=2E This will eliminate all the confusion w= ith diskid=2E Then run "zpool clear", which, if da3 is still online and not= completely dead, the pool should come out of the faulted state=2E Check zp= ool status to look for this alleged replacement in progress=2E If it is tru= ly trying to replace a device, it should show up in zpool status with the a= ctual device, or the guid if it can't find the device=2E > >I saw and appreiciated that response, but didn=E2=80=99t respond on that = thread because I don=E2=80=99t _want_ to turn all of those things off=2E A= t least, I don=E2=80=99t want to refer to everything by the auto-numbered d= a# that I think that will cause=2E And, Freddie, your comment about GPT pa= rtition labels I think doesn=E2=80=99t apply because I don=E2=80=99t have G= PT on my disks=2E Just all one big ZFS device=2E This is why I=E2=80=99m = looking at glabel=E2=80=99s generic labeling now=2E You probably don't want that=2E You will have to use the glabel dev, which= will not be the same size as your other devices=2E IIRC you have no contro= l over what device node the system finds first for the pool=2E Even if you = use GPT labels, the daXpY device will still exist=2E=20 >The former da3 is off-line, out of the chassis=2E I replaced a disk in a= full chassis, having them both online at the same time is not possible=2E = That drive in ZFS=E2=80=99s mind is only faulted because I tried =E2=80=9C= zpool offline -f=E2=80=9D on it to see if that helped=2E It sounds like you have replaced the wrong device=2E Check the "zpool hist= ory" to see what you did=2E=20 In your earlier message, three devices were shown in each raidz, when what= you should be seeing is that one raidz has an offline device identified by= guid and maybe "was /dev/da3" that is being replaced, along with the repla= cement device=2E I don't see any of that=2E=20 >> If you have initiated a replace, and the replacing disk has now been "l= ost" or unlabeled, you are in a bind=2E I ran into this problem many years = ago, and I thought it was fixed, but the bug was called something like "can= 't replace a replacing vdev"=2E I ultimately solved my problem by manually = editing a fake vdev to have the same guid as the missing device, restarting= the replace and then canceling it before zfs realized it was fake=2E But, = I am almost certain that zpool cancel can do this now, with the guid=2E > >I didn=E2=80=99t initiate a replace until after the disks were physically= changed=2E Although in this conversation realize that things likely got c= onfused by the replacement in the kernel=E2=80=99s mind of da3 with what us= ed to be da4=2E :-/ This is why your zpool history will be helpful=2E What did you actually tr= y to replace, and what did you mean to replace=2E=20 >> If da10 has a label that says it is in the pool, it is probably the "re= placing" vdev and should be picked up=E2=80=A6 > >Da10, now also /dev/label/drive03, seems to think it=E2=80=99s in the poo= l somewhere, according to zdb -l=2E >But I=E2=80=99m not sure if this helps=2E And, following your other mess= age saying I shouldn=E2=80=99t put labels >on disks that are to be used in their entirety as ZFS devices, I=E2=80=99= ve deleted that label and >zlabelclear=E2=80=99d this device now=2E (since the zfs label still had = the /dev/label/ path in it)