From nobody Fri Sep 06 19:34:36 2024 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4X0mdm6wVwz5VZd3 for ; Fri, 06 Sep 2024 19:34:52 +0000 (UTC) (envelope-from cross+freebsd@relay.distal.com) Received: from relay.wiredblade.com (relay.wiredblade.com [168.235.95.80]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4X0mdm3ZfLz4rdG for ; Fri, 6 Sep 2024 19:34:52 +0000 (UTC) (envelope-from cross+freebsd@relay.distal.com) Authentication-Results: mx1.freebsd.org; none dkim-signature: v=1; a=rsa-sha256; d=relay.distal.com; s=mail; c=relaxed/relaxed; q=dns/txt; h=From:Subject:Date:Message-ID:To:CC:MIME-Version:Content-Type:In-Reply-To:References; bh=aQdvkE/DXmOzK1hMVi0fgM5rPPI2Opkqb/rWZV/hS7w=; b=sGPsB1mWfsTqkfZPlvtocU2umbPNExm8fzrX9HjN1pycQLQS5s7poYLJrEcYMtK6qx4mSvWKny4MaeoQn7xHyfCBeN8hh3VCJlMCYXwJ598HpZ9/U0EFQtjFNOFpdpJUT3ems3QXTYIrsURBCe6KOfFFh/5Ejb8vpj7ezsWwuXgjelp9ZlSme5JAL/M7VMeTwYT1M7uwGkxUWY0ZEAp7oahdfkjciIWHdrrWVjWbmok+xvRt99n2q21oB4 aaf4jdsfgSfx4RDQPwJ5kwHn9LrgQQxmebmKvFBdTVC1fXMfd9FNNos1wI8Z9OKy8zITDT2+yUNN8k1zwl4K5SusPgRQ== Received: from mail.distal.com (pool-108-51-233-124.washdc.fios.verizon.net [108.51.233.124]) by relay.wiredblade.com with ESMTPSA (version=TLSv1.2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256) ; Fri, 6 Sep 2024 19:34:50 +0000 Received: from smtpclient.apple ( [2001:420:c0c4:1001::9f]) by tristain.distal.com (OpenSMTPD) with ESMTPSA id 3fc5ded1 (TLSv1.2:ECDHE-RSA-AES256-GCM-SHA384:256:NO); Fri, 6 Sep 2024 15:34:48 -0400 (EDT) From: Chris Ross Message-Id: Content-Type: multipart/alternative; boundary="Apple-Mail=_26064149-B678-4693-B259-B3840725655C" List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@FreeBSD.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3776.700.51\)) Subject: Re: Unable to replace drive in raidz1 Date: Fri, 6 Sep 2024 15:34:36 -0400 In-Reply-To: Cc: freebsd-fs@freebsd.org To: Wes Morgan References: <5ED5CB56-2E2A-4D83-8CDA-6D6A0719ED19@distal.com> <6A20ABDA-9BEA-4526-94C1-5768AA564C13@distal.com> <0CF1E2D7-6C82-4A8B-82C3-A5BF1ED939CF@distal.com> <29003A7C-745D-4A06-8558-AE64310813EA@distal.com> <42346193-AD06-4D26-B0C6-4392953D21A3@gmail.com> X-Mailer: Apple Mail (2.3776.700.51) X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; TAGGED_FROM(0.00)[freebsd]; ASN(0.00)[asn:3842, ipnet:168.235.92.0/22, country:US] X-Rspamd-Queue-Id: 4X0mdm3ZfLz4rdG --Apple-Mail=_26064149-B678-4693-B259-B3840725655C Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 > On Sep 6, 2024, at 15:16, Wes Morgan wrote: >=20 > You probably don't want that. You will have to use the glabel dev, = which will not be the same size as your other devices. IIRC you have no = control over what device node the system finds first for the pool. Even = if you use GPT labels, the daXpY device will still exist.=20 Right. But if I don=E2=80=99t _use_ those device names, it won=E2=80=99t = matter. If I use /dev/label/foo, or /dev/gpt/foo, I=E2=80=99ll just = always use those. I just did that with the ufs disk I have since it = moved names, now it=E2=80=99s "/dev/ufs/drive12=E2=80=9D in /etc/fstab = et al. I want to have some sort of label. I=E2=80=99d rather not have to add a = partitioning scheme to the disk if I know I=E2=80=99m just going to use = the whole disk just to get a label, but I suppose if I have to I can. = Though I=E2=80=99d have to do it one disk at a time. :-) >=20 >> The former da3 is off-line, out of the chassis. I replaced a disk in = a full chassis, having them both online at the same time is not = possible. That drive in ZFS=E2=80=99s mind is only faulted because I = tried =E2=80=9Czpool offline -f=E2=80=9D on it to see if that helped. >=20 > It sounds like you have replaced the wrong device. Check the "zpool = history" to see what you did.=20 >=20 > In your earlier message, three devices were shown in each raidz, when = what you should be seeing is that one raidz has an offline device = identified by guid and maybe "was /dev/da3" that is being replaced, = along with the replacement device. I don't see any of that.=20 History attached. There is no replacement device (sub-vdev) until after = the =E2=80=9Czpool replace=E2=80=9D starts, which it won=E2=80=99t. >> I didn=E2=80=99t initiate a replace until after the disks were = physically changed. Although in this conversation realize that things = likely got confused by the replacement in the kernel=E2=80=99s mind of = da3 with what used to be da4. :-/ >=20 > This is why your zpool history will be helpful. What did you actually = try to replace, and what did you mean to replace.=20 All of my history since the last previous boot in May. 2024-09-05.09:40:14 zpool offline tank da3 2024-09-05.14:26:44 zpool import -c /etc/zfs/zpool.cache -a -N 2024-09-05.14:32:45 zpool import -c /etc/zfs/zpool.cache -a -N 2024-09-05.14:52:18 zpool offline tank da3 2024-09-05.14:53:51 zpool offline tank da3 2024-09-05.14:59:43 zpool offline -f tank da3 2024-09-05.15:02:53 zpool clear tank 2024-09-05.15:07:41 zpool online tank da3 2024-09-05.15:10:00 zpool add tank spare da10 2024-09-05.15:10:20 zpool offline -f tank da3 2024-09-05.15:35:23 zpool remove tank da10 2024-09-05.15:54:35 zpool scrub tank 2024-09-05.16:01:12 zpool set autoreplace=3Don tank 2024-09-05.16:01:24 zpool set autoexpand=3Don tank 2024-09-05.16:02:16 zpool add -o ashift=3D9 tank spare da10 2024-09-06.10:10:20 zpool remove tank da10 So, I offline=E2=80=99d the disk-to-be-replaced at 09:40 yesterday, then = I shut the system down, removed that physical device replacing it with a = larger disk, and rebooted. I suspect the =E2=80=9Coffline=E2=80=9Ds = after that are me experimenting when it was telling me it couldn=E2=80=99t= start the replace action I was asking for. The scrub I started yesterday just because the replace says sometihng = about an operation in progress, so I did that. It completed with no = issues, but nothing changed w.r.t. my current problem. I=E2=80=99m pretty sure the problem here is that the old da3 went away, = and a new da3 came online as a member of raidz1-1. The new disk I added = came online as da10, for some reason. I had to resolve the issue of the = UFS disk which used to be da10 now being da9, but that was easy enough. = Just unexpected. - Chris= --Apple-Mail=_26064149-B678-4693-B259-B3840725655C Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8

On Sep 6, 2024, at 15:16, Wes Morgan = <morganw@gmail.com> wrote:

You = probably don't want that. You will have to use the glabel dev, which = will not be the same size as your other devices. IIRC you have no = control over what device node the system finds first for the pool. Even = if you use GPT labels, the daXpY device will still exist. 

Right.  But if I don=E2=80=99= t _use_ those device names, it won=E2=80=99t matter.  If I use = /dev/label/foo, or /dev/gpt/foo, I=E2=80=99ll just always use those. =  I just did that with the ufs disk I have since it moved names, now = it=E2=80=99s "/dev/ufs/drive12=E2=80=9D in /etc/fstab et = al.

I want to have some sort of label. =  I=E2=80=99d rather not have to add a partitioning scheme to the = disk if I know I=E2=80=99m just going to use the whole disk just to get = a label, but I suppose if I have to I can.  Though I=E2=80=99d have = to do it one disk at a time.  :-)


The former da3 is off-line, out of the chassis. =  I replaced a disk in a full chassis, having them both online at = the same time is not possible.  That drive in ZFS=E2=80=99s mind is = only faulted because I tried =E2=80=9Czpool offline -f=E2=80=9D on it to = see if that helped.

It = sounds like you have replaced the wrong device. Check the "zpool = history" to see what you did. 

In your earlier message, = three devices were shown in each raidz, when what you should be seeing = is that one raidz has an offline device identified by guid and maybe = "was /dev/da3" that is being replaced, along with the replacement = device. I don't see any of that. 

History attached.  There = is no replacement device (sub-vdev) until after the =E2=80=9Czpool = replace=E2=80=9D starts, which it won=E2=80=99t.

I didn=E2=80=99t initiate a replace until after = the disks were physically changed.  Although in this conversation = realize that things likely got confused by the replacement in the = kernel=E2=80=99s mind of da3 with what used to be da4. =  :-/

This is = why your zpool history will be helpful. What did you actually try to = replace, and what did you mean to replace. 

All of my history since the last previous boot in = May.

2024-09-05.09:40:14 zpool offline = tank da3
2024-09-05.14:26:44 zpool import -c = /etc/zfs/zpool.cache -a -N
2024-09-05.14:32:45 zpool import -c = /etc/zfs/zpool.cache -a -N
2024-09-05.14:52:18 zpool offline = tank da3
2024-09-05.14:53:51 zpool offline tank = da3
2024-09-05.14:59:43 zpool offline -f tank = da3
2024-09-05.15:02:53 zpool clear = tank
2024-09-05.15:07:41 zpool online tank = da3
2024-09-05.15:10:00 zpool add tank spare = da10
2024-09-05.15:10:20 zpool offline -f tank = da3
2024-09-05.15:35:23 zpool remove tank = da10
2024-09-05.15:54:35 zpool scrub = tank
2024-09-05.16:01:12 zpool set autoreplace=3Don = tank
2024-09-05.16:01:24 zpool set autoexpand=3Don = tank
2024-09-05.16:02:16 zpool add -o ashift=3D9 tank spare = da10
2024-09-06.10:10:20 zpool remove tank = da10

So, I offline=E2=80=99d the = disk-to-be-replaced at 09:40 yesterday, then I shut the system down, = removed that physical device replacing it with a larger disk, and = rebooted.  I suspect the =E2=80=9Coffline=E2=80=9Ds after that are = me experimenting when it was telling me it couldn=E2=80=99t start the = replace action I was asking for.

The scrub I = started yesterday just because the replace says sometihng about an = operation in progress, so I did that.  It completed with no issues, = but nothing changed w.r.t. my current = problem.

I=E2=80=99m pretty sure the problem = here is that the old da3 went away, and a new da3 came online as a = member of raidz1-1.  The new disk I added came online as da10, for = some reason.  I had to resolve the issue of the UFS disk which used = to be da10 now being da9, but that was easy enough.  Just = unexpected.

      - = Chris
= --Apple-Mail=_26064149-B678-4693-B259-B3840725655C--