From nobody Fri Sep 06 22:02:31 2024 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4X0qwP2LqFz5W5B5 for ; Fri, 06 Sep 2024 22:02:45 +0000 (UTC) (envelope-from asomers@gmail.com) Received: from mail-oi1-f177.google.com (mail-oi1-f177.google.com [209.85.167.177]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "WR4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4X0qwN49XSz4GLt for ; Fri, 6 Sep 2024 22:02:44 +0000 (UTC) (envelope-from asomers@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=fail reason="SPF not aligned (relaxed), No valid DKIM" header.from=freebsd.org (policy=none); spf=pass (mx1.freebsd.org: domain of asomers@gmail.com designates 209.85.167.177 as permitted sender) smtp.mailfrom=asomers@gmail.com Received: by mail-oi1-f177.google.com with SMTP id 5614622812f47-3e0059dc06bso1630141b6e.0 for ; Fri, 06 Sep 2024 15:02:44 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1725660163; x=1726264963; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=vaWvswPE7Hfpi9K8aN9u3LR82RVyWXHrDN1FgBF0x+c=; b=shxWYJC1zM29FGIx4F+TyPa4jex6DtZ99Ov1RXGJjuhbnFJzierGKdtbX1eecDwkFl kLHGNXMmZGCfFUyOqJBi82zpFnYjJYvwS/4BvSVg8cOX9464Wj/uT6FKnDvAZfcGgvCg DchfLFfSoh8/4XVrtPVoV99FDLMhN5ofkPY1N8AbySK9PW4gQKMRlJ5pswv6QHm46pwN YS9cdhHDYML0zJUgSLWvNNdmnXO9E1NbheKlXOAF/BVe0ei5mM6ZlY9TenYgTenwi+Wd 4Pe1BXThLeNxjkv1W6TkNbwSyUFjpYPNTGytklXDsC9EgwPM55URtt4MjYLgjBwbaSs/ NtpQ== X-Forwarded-Encrypted: i=1; AJvYcCU5tNtrICsJcIBg0e2a6OcY6tj1Yf+YBCQ4/wfy8EdkKTwVdJ6GoZEBQ18uYnZC+gsISbCWXY3RwLRo@freebsd.org X-Gm-Message-State: AOJu0YyG9Xtcm6tvHV8UdMQ/d0Ln9msXJQOvnK0D8UeQgUGlXtRoRnHi Dknf8foLkaulldmt4h2iiMDf4PtkEpjyUKNRAtbHbPZ+/1m8zNMwCxxXBAa10pvSLYf4ppd3B83 E22YEx05Lf+hn6jevycTJ5VZKvNU= X-Google-Smtp-Source: AGHT+IGKV2zVeSjhAC3VeL7dBzQUA7P1h6B/xEzlZ4T+xfvu/QWrg4eoeQ/n+QBDE7mhZu449tiuE/4/5YexYgzfP2s= X-Received: by 2002:a05:6808:6544:b0:3db:331:9d41 with SMTP id 5614622812f47-3e029ef8ac9mr4598310b6e.23.1725660162888; Fri, 06 Sep 2024 15:02:42 -0700 (PDT) List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@FreeBSD.org MIME-Version: 1.0 References: <5ED5CB56-2E2A-4D83-8CDA-6D6A0719ED19@distal.com> <6A20ABDA-9BEA-4526-94C1-5768AA564C13@distal.com> <0CF1E2D7-6C82-4A8B-82C3-A5BF1ED939CF@distal.com> <29003A7C-745D-4A06-8558-AE64310813EA@distal.com> <42346193-AD06-4D26-B0C6-4392953D21A3@gmail.com> <50B791D8-F0CC-431E-93B8-834D57AB3C14@gmail.com> In-Reply-To: From: Alan Somers Date: Fri, 6 Sep 2024 16:02:31 -0600 Message-ID: Subject: Re: Unable to replace drive in raidz1 To: Chris Ross Cc: Wes Morgan , FreeBSD Filesystems Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Spamd-Bar: -- X-Spamd-Result: default: False [-2.90 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-0.999]; FORGED_SENDER(0.30)[asomers@freebsd.org,asomers@gmail.com]; R_SPF_ALLOW(-0.20)[+ip4:209.85.128.0/17]; MIME_GOOD(-0.10)[text/plain]; DMARC_POLICY_SOFTFAIL(0.10)[freebsd.org : SPF not aligned (relaxed), No valid DKIM,none]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; FREEMAIL_CC(0.00)[gmail.com,freebsd.org]; RCVD_TLS_LAST(0.00)[]; ARC_NA(0.00)[]; TO_DN_ALL(0.00)[]; MIME_TRACE(0.00)[0:+]; MISSING_XM_UA(0.00)[]; FREEFALL_USER(0.00)[asomers]; FREEMAIL_ENVFROM(0.00)[gmail.com]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; TO_MATCH_ENVRCPT_SOME(0.00)[]; FROM_NEQ_ENVFROM(0.00)[asomers@freebsd.org,asomers@gmail.com]; RCVD_COUNT_ONE(0.00)[1]; MLMMJ_DEST(0.00)[freebsd-fs@freebsd.org]; TAGGED_RCPT(0.00)[freebsd]; R_DKIM_NA(0.00)[]; ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US]; RWL_MAILSPIKE_POSSIBLE(0.00)[209.85.167.177:from]; RCVD_IN_DNSWL_NONE(0.00)[209.85.167.177:from] X-Rspamd-Queue-Id: 4X0qwN49XSz4GLt On Fri, Sep 6, 2024 at 3:49=E2=80=AFPM Chris Ross wrote: > > > > > On Sep 6, 2024, at 17:22, Wes Morgan wrote: > > > > The labels are helpful for fstab, but zfs doesn't need fstab. In the ea= rly days of zfs on freebsd the unpartitioned device was recommended; maybe = that's not accurate any longer, but I still follow it for a pool that conta= ins vdevs with multiple devices (raidz). > > > > If you use, e.g., da0 in a pool, you cannot later replace it with a lab= eled device of the same size; it won't have enough sectors. > > The problem is shown here. da3 was in a pool. Then, when the system reb= ooted, da3 was the kernels name for a different device in a different pool.= Had I known then how to interact with the guid (status -g), I likely woul= d=E2=80=99ve been fine. > > >> So, I offline=E2=80=99d the disk-to-be-replaced at 09:40 yesterday, th= en I shut the system down, removed that physical device replacing it with a= larger disk, and rebooted. I suspect the =E2=80=9Coffline=E2=80=9Ds after= that are me experimenting when it was telling me it couldn=E2=80=99t start= the replace action I was asking for. > > > > This is probably where you made your mistake. Rebooting shifted another= device into da3. When you tried to offline it, you were probably either ta= rgeting a device in a different raidz or one that wasn't in the pool. The o= utput of those original offline commands would have been informative. You c= ould also check dmesg and map the serial numbers to device assignments to f= igure out what device moved to da3. > > I offline=E2=80=99d =E2=80=9Cda3=E2=80=9D before I rebooted. After reboo= ting, I tried the obvious and correct (i thought) =E2=80=9Czpool replace da= 3 da10=E2=80=9D only to get the error I=E2=80=99ve been getting since. Aga= in, had I known how to use the guid for the device that used to be da3 but = now isn=E2=80=99t, that might=E2=80=99ve worked. I can=E2=80=99t know now. > > Then, while trying to fix the problem, I likely made it worse by trying t= o interact with da3, which in the pools brain was a missing disk in raidz1-= 0, but the kernel also knew /dev/da3 to be a working disk (that happened to= be one of the drives in raidz1-1). I feel that zfs did something wrong so= mewhere if it _ever_ tried to talk to /dev/da3 when I said =E2=80=9Cda3=E2= =80=9D after I rebooted and it found that device to be part of raidz1-1, bu= t. > > > > Sounds about right. In another message it seemed like the pool had star= ted an autoreplace. So I assume you have zfsd enabled? That is what issues = the replace command. Strange that it is not anywhere in the pool history. T= here should be syslog entries for any actions it took. > > I don=E2=80=99t think so. That message about some =E2=80=9Calready in re= placing/spare config=E2=80=9D came up before anything else. At which point= , I=E2=80=99d never had a spare in this pool, and there was no replace show= n in zpool status. > > > In your case, it appears that you had two missing devices - the origina= l "da3" that was physically removed, and the new da3 that you forced offlin= e. You added da10 as a spare, when what you needed to do was a replace. Spa= re devices do not auto-replace without zfsd running and autoreplace set to = on. > > I did offline =E2=80=9Cda3=E2=80=9D a couple of times, again thinking I w= as working with what zpool showed as =E2=80=9Cda3=E2=80=9D. If it did anyt= hing with /dev/da3 there, then I think that may be a bug. Or, at least, so= mething that should be made more clear. It _didn=E2=80=99t_ offline the di= skid/DISK-K1GMBN9D from raidz1-1, which is what the kernel has at da3. So. > > > This should all be reported in zpool status. In your original message, = there is no sign of a replacement in progress or a spare device, assuming y= ou didn't omit something. If the pool is only showing that a single device = is missing, and that device is to be replaced by da10, zero out the first a= nd last sectors (I think a zfs label is 128k?) to wipe out any labels and u= se the replace command, not spare, e.g. "zpool replace tank da3 da10", or u= se the missing guid as suggested elsewhere. This should work based on the i= nformation provided. > > I=E2=80=99ve never seen a replacement going on, and I have had the new di= sk =E2=80=9Cda10=E2=80=9D as a spare a couple of times while testing. But = it wasn=E2=80=99t left there after I determined that that also didn=E2=80= =99t let me get it replaced into the raidz. > > And, that attempt to replace is what I=E2=80=99ve tried many times, with = multiple id=E2=80=99s. I have cleared the label on da10 multiple times. T= hat replace doesn=E2=80=99t work, giving this error message in all cases. > > - Chris > > > % glabel status > Name Status Components > diskid/DISK-BTWL503503TW480QGN N/A ada0 > gpt/l2arc N/A ada0p1 > gptid/9d00849e-0b82-11ec-a143-84b2612f2c38 N/A ada0p1 > diskid/DISK-K1GMBN9D N/A da3 > diskid/DISK-3WJDHJ2J N/A da6 > diskid/DISK-3WK3G1KJ N/A da7 > diskid/DISK-3WJ7ZMMJ N/A da8 > diskid/DISK-K1GMEDMD N/A da4 > diskid/DISK-K1GMAX1D N/A da5 > ufs/drive12 N/A da9 > diskid/DISK-ZGG0A2PA N/A da10 > > % zpool status tank > pool: tank > state: DEGRADED > status: One or more devices are faulted in response to persistent errors. > Sufficient replicas exist for the pool to continue functioning in= a > degraded state. > action: Replace the faulted device, or use 'zpool clear' to mark the devi= ce > repaired. > scan: scrub repaired 0B in 17:14:03 with 0 errors on Fri Sep 6 09:08:3= 4 2024 > config: > > NAME STATE READ WRITE CKSUM > tank DEGRADED 0 0 0 > raidz1-0 DEGRADED 0 0 0 > da3 FAULTED 0 0 0 external de= vice fault > da1 ONLINE 0 0 0 > da2 ONLINE 0 0 0 > raidz1-1 ONLINE 0 0 0 > diskid/DISK-K1GMBN9D ONLINE 0 0 0 > diskid/DISK-K1GMEDMD ONLINE 0 0 0 > diskid/DISK-K1GMAX1D ONLINE 0 0 0 > raidz1-2 ONLINE 0 0 0 > diskid/DISK-3WJDHJ2J ONLINE 0 0 0 > diskid/DISK-3WK3G1KJ ONLINE 0 0 0 > diskid/DISK-3WJ7ZMMJ ONLINE 0 0 0 > > errors: No known data errors > > % sudo zpool replace tank da3 da10 > Password: > cannot replace da3 with da10: already in replacing/spare config; wait for= completion or use 'zpool detach' > > % zpool status -g tank > pool: tank > state: DEGRADED > status: One or more devices are faulted in response to persistent errors. > Sufficient replicas exist for the pool to continue functioning in= a > degraded state. > action: Replace the faulted device, or use 'zpool clear' to mark the devi= ce > repaired. > scan: scrub repaired 0B in 17:14:03 with 0 errors on Fri Sep 6 09:08:3= 4 2024 > config: > > NAME STATE READ WRITE CKSUM > tank DEGRADED 0 0 0 > 16506780107187041124 DEGRADED 0 0 0 > 9127016430593660128 FAULTED 0 0 0 external de= vice fault > 4094297345166589692 ONLINE 0 0 0 > 17850258180603290288 ONLINE 0 0 0 > 5104119975785735782 ONLINE 0 0 0 > 6752552549817423876 ONLINE 0 0 0 > 9072227575611698625 ONLINE 0 0 0 > 13778609510621402511 ONLINE 0 0 0 > 11410204456339324959 ONLINE 0 0 0 > 1083322824660576293 ONLINE 0 0 0 > 12505496659970146740 ONLINE 0 0 0 > 11847701970749615606 ONLINE 0 0 0 > > errors: No known data errors > > % sudo zpool replace tank 9127016430593660128 da10 > cannot replace 9127016430593660128 with da10: already in replacing/spare = config; wait for completion or use 'zpool detach' > > % sudo zpool replace tank 9127016430593660128 diskid/DISK-ZGG0A2PA > cannot replace 9127016430593660128 with diskid/DISK-ZGG0A2PA: already in = replacing/spare config; wait for completion or use 'zpool detach' Another user reports the same error message. In their case, it's an inappropriate error message from /sbin/zpool. Can you try a "zpool status -v" and "diskinfo -f /dev/da10"? That will show you if you have the same problem. If your pool has a 512B block size but the new disk is 4kn, then you cannot use it as a replacement. https://github.com/openzfs/zfs/issues/14730