From nobody Mon Jun 20 23:32:13 2022 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id A3231863877 for ; Mon, 20 Jun 2022 23:33:23 +0000 (UTC) (envelope-from bsdlists@jld3.net) Received: from mail.jld3.net (mail.jld3.net [45.55.236.93]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4LRmDL5T8fz4jS6; Mon, 20 Jun 2022 23:33:22 +0000 (UTC) (envelope-from bsdlists@jld3.net) Received: from localhost (localhost [127.0.0.1]) by mail.jld3.net (Postfix) with ESMTP id 7FAA94160D; Mon, 20 Jun 2022 17:33:22 -0600 (MDT) X-Virus-Scanned: amavisd-new at jld3.net Received: from mail.jld3.net ([127.0.0.1]) by localhost (mail.jld3.net [127.0.0.1]) (amavisd-new, port 10024) with LMTP id 2-YXOIfPIY4R; Mon, 20 Jun 2022 17:33:22 -0600 (MDT) Received: from [172.21.35.205] (c-24-9-144-115.hsd1.co.comcast.net [24.9.144.115]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) (Authenticated sender: jld) by mail.jld3.net (Postfix) with ESMTPSA id B0F2340790; Mon, 20 Jun 2022 17:33:21 -0600 (MDT) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.jld3.net B0F2340790 DKIM-Filter: OpenDKIM Filter v2.11.0 mail.jld3.net B0F2340790 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=jld3.net; s=8d052f02dde2; t=1655768001; bh=cG33NQ/asgxeIXyAHRsgAseJqpibYZ8Qu+oJElVuzwA=; h=From:To:Subject:Date:Message-ID:MIME-Version:From; b=OMyuJVn3zIjKnWbEUnQZhUZy0PAMire6z4JtRB74vOzX0Dh390Co0quoKn2xxTdum x0HA6o73EHQlttFTZ6otMdneTk3llluUJONCyxnHoIJ5VBL7onAglvIZPUCu8ipv0p 2iDq9UkY3GIzMCALVOre1GaESBYuUK95G0y/CtBA= From: "John Doherty" To: "Alan Somers" Cc: freebsd-fs Subject: Re: "spare-X" device remains after resilvering Date: Mon, 20 Jun 2022 17:32:13 -0600 X-Mailer: MailMate (1.13.2r5673) Message-ID: <7B7E61B8-0508-4FFD-B697-FBFB881E2B09@jld3.net> In-Reply-To: References: <34A91D31-1883-40AE-82F3-57B783532ED7@jld3.net> <768F3745-D7FF-48C8-BA28-ABEB49BAFAA8@jld3.net> List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; format=flowed X-Rspamd-Queue-Id: 4LRmDL5T8fz4jS6 X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=jld3.net header.s=8d052f02dde2 header.b=OMyuJVn3; dmarc=pass (policy=quarantine) header.from=jld3.net; spf=pass (mx1.freebsd.org: domain of bsdlists@jld3.net designates 45.55.236.93 as permitted sender) smtp.mailfrom=bsdlists@jld3.net X-Spamd-Result: default: False [-4.00 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; ARC_NA(0.00)[]; R_DKIM_ALLOW(-0.20)[jld3.net:s=8d052f02dde2]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:45.55.236.93]; MIME_GOOD(-0.10)[text/plain]; NEURAL_HAM_LONG(-1.00)[-1.000]; RCVD_COUNT_THREE(0.00)[4]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[jld3.net:+]; RCPT_COUNT_TWO(0.00)[2]; DMARC_POLICY_ALLOW(-0.50)[jld3.net,quarantine]; NEURAL_HAM_SHORT(-1.00)[-0.999]; MLMMJ_DEST(0.00)[freebsd-fs]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:14061, ipnet:45.55.192.0/18, country:US]; MID_RHS_MATCH_FROM(0.00)[]; RECEIVED_SPAMHAUS_PBL(0.00)[24.9.144.115:received] X-ThisMailContainsUnwantedMimeParts: N On Mon 2022-06-20 05:08 PM MDT -0600, wrote: >> I don't think I can detach anything because this is all raidz2 and >> detach only works with components of mirrors. ... > Ahh, but you can detach in this case, because the "spare-9" vdev is > itself a type of mirror. Try that command. I think it will do what > you want, with no extra resilvering required. Oh OK, my misunderstanding again. Not resilvering again would be great. For reference or if anyone else is following along, the current state (abbreviated) is this: pool: zp1 state: DEGRADED ... config: NAME STATE READ WRITE CKSUM zp1 DEGRADED 0 0 0 raidz2-0 ONLINE 0 0 0 ... raidz2-1 ONLINE 0 0 0 ... raidz2-2 ONLINE 0 0 0 ... raidz2-3 DEGRADED 0 0 0 ... spare-9 DEGRADED 0 0 0 6960108738988598438 OFFLINE 0 0 0 was /dev/gpt/disk39 gpt/disk41 ONLINE 0 0 0 And you're right, "zpool detach zp1 6960108738988598438" worked fine so I now have this: pool: zp1 state: DEGRADED ... NAME STATE READ WRITE CKSUM zp1 DEGRADED 0 0 0 raidz2-0 ONLINE 0 0 0 ... raidz2-1 ONLINE 0 0 0 ... raidz2-2 ONLINE 0 0 0 ... raidz2-3 DEGRADED 0 0 0 gpt/disk30 ONLINE 0 0 0 3343132967577870793 OFFLINE 0 0 0 was /dev/gpt/disk31 gpt/disk32 ONLINE 0 0 0 gpt/disk33 ONLINE 0 0 0 gpt/disk34 ONLINE 0 0 0 gpt/disk35 ONLINE 0 0 0 gpt/disk36 ONLINE 0 0 0 gpt/disk37 ONLINE 0 0 0 gpt/disk38 ONLINE 0 0 0 gpt/disk41 ONLINE 0 0 0 spares gpt/disk42 AVAIL gpt/disk43 AVAIL gpt/disk44 AVAIL I thought I had tried that but obviously not. Now there is no more remnant of what was gpt/disk39 and gpt/disk41 is just a normal, permanent member of the raidz2-3 vdev and no longer in the configured spares list. Perfect. The pool is still degraded because there is still another offline device that needs to be replaced. To fix that, I can do this: # zpool replace zp1 3343132967577870793 gpt/disk42 Wait for the resilver to finish, and then do: # zpool detach zp1 3343132967577870793 And everything should be fine with all the vdevs and the pool itself in "online" status again (and gpt/disk42 no longer an available spare). This is great. You've been a big help, can't thank you enough.