From nobody Sun Jan 07 21:15:14 2024 X-Original-To: freebsd-stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4T7VMz1Lzqz560hg for ; Sun, 7 Jan 2024 21:15:27 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mail-ej1-x632.google.com (mail-ej1-x632.google.com [IPv6:2a00:1450:4864:20::632]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4T7VMy4nckz45P7 for ; Sun, 7 Jan 2024 21:15:26 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-ej1-x632.google.com with SMTP id a640c23a62f3a-a28d61ba65eso120249566b.3 for ; Sun, 07 Jan 2024 13:15:26 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bsdimp-com.20230601.gappssmtp.com; s=20230601; t=1704662125; x=1705266925; darn=freebsd.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=vgHAAVGVnuMYvEQ3QnRnNsHbKlCKig/3HCPsAT5KGZU=; b=aKrEo8Hmc8oilm3o+C8F+KiCRmgtYyt9gzVQVXW470qBIJMbLKpjBLq/BagkBobQlD GFqC6wZKENMIbV53brGN2AIo7pzFMBqnnsDNlnu3dfElAGF7+cO5UqsgreOzh+WuM9sd UFxhFdzo57kL1xzgq7UcQwTXZoBnad6MsT+SkeEkOpCKTptZRxvLnxVKd4+dVPoBCzem QWgofQ0tOqC3e9SxO6gs5QXc38pet+IKcxVRiw2c9hQhNqwW8jmPR6rvY5tGLjPsxfjG kfDDpIGVPHTwdG2Ty8dZOX02JH8PUvroJkuXaYE5o7thDAGx8cED6tU97uKwzei5/Hzy 2qwg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1704662125; x=1705266925; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=vgHAAVGVnuMYvEQ3QnRnNsHbKlCKig/3HCPsAT5KGZU=; b=ZoL7qafSW0R4P6/HccJSkIAvAmcX6cptzXG+y9P2PACL8KBb6VTvsrGMjU7Sra9xhH oz5WndyfSPXpfaNmLwajbkyRgLVeCgO1JlWS8RnaPLiW/Kzc4N2wWCLS1YDYR/+EKTk/ RaMHDvFzdABTJ8/fZBTipEEdqQfFS+BGJVL18QByP7PuFYORmYp2DJlgE9XTNA4hApST uZgjLrAyWtTAKNeKoWr1rxYztRdOBubjaQPCYfh+bd188VMyWdd9Teu5FfuBpHCy6+27 GoTxmFzqjsemVDqNP0STS/gkeJ0/17UTVdUUN7r7TqVnCmILL/8q48SQ8hwhWr1X+H0U UmWA== X-Gm-Message-State: AOJu0YypEfoZRWcyHQMyyGwbFKa11VDte6Waa3PBtcUTW6dMv8QMORen lApuAxBsadjKi7lEgG5JjE2ubbM5JpZdg2Bd59r2TMnlXlYPIyBfTDZSciE6pXU= X-Google-Smtp-Source: AGHT+IHFhioVO+WSOS73stbnwyNMxN77R5XrtFVLfzanDmMUJu9h6D/pAJZBbTJ6adefF93zun91Vs5Y+5oZZYYXOX4= X-Received: by 2002:a17:907:a05:b0:a28:dfe4:1d0b with SMTP id bb5-20020a1709070a0500b00a28dfe41d0bmr860061ejc.31.1704662125138; Sun, 07 Jan 2024 13:15:25 -0800 (PST) List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org MIME-Version: 1.0 References: <065f4f5c-f38b-45f4-b7e7-5248f871f7e6@FreeBSD.org> <2f91eeb7-430b-49e2-817b-5acd0f445fe9@FreeBSD.org> In-Reply-To: <2f91eeb7-430b-49e2-817b-5acd0f445fe9@FreeBSD.org> From: Warner Losh Date: Sun, 7 Jan 2024 14:15:14 -0700 Message-ID: Subject: Re: FreeBSD 13.2-STABLE can not boot from damaged mirror AND pool stuck in "resilver" state even without new devices. To: lev@freebsd.org Cc: freebsd-fs , freebsd-stable Content-Type: multipart/alternative; boundary="0000000000006e66a9060e619437" X-Rspamd-Queue-Id: 4T7VMy4nckz45P7 X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US] --0000000000006e66a9060e619437 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Sun, Jan 7, 2024 at 1:57=E2=80=AFPM Lev Serebryakov wr= ote: > On 07.01.2024 21:49, Lev Serebryakov wrote: > > > On 07.01.2024 19:34, Warner Losh wrote: > > > >> I must have missed it. What were the diagnostics? > > Oh, and two "nvlist inconsistency" before that vvvv > > > zio_read error: 5 > > zio_read error: 5 > > zio_read error: 5 > 5 is EIO which the loader uses internally for any error that the disk reports. I've not read through all the code involved here, but I think that means there might be read errors for real. Though the nvlist inconsistency might be an issue. So, if this is a mirror, then ada0 blank and ada1 with good data, in theory you should be fine. However, perhaps ZFS is finding that there's an error from ada1 for real. Does all of ada1 read with a simple dd? Not sure about the losing devices you described later on. > ZFS: i/o error - all block copies unavailable > > ZFS: can't read MOS of pool zroot > > > > > > To be honest, I thinks there is something else. Because sequence of > events were (sorry, too long, but I think, tht every detail matters here)= : > Yea. There's something that's failing, which zio_read is woefully under reporting for our diagnostic efforts. And/or something is getting confused by the blank disk and/or the partially resilvered disk. > (1) Update to 13.2 from 12.4. With installation of new gptzfsboot with > gpart on both disks. It could place new /boot far away, but see (2) > > (2) Reboot, which completed, but showed that ada0 has problems > > (3) Replacement of ada0 by DC technicians, new disk is 512/4096, old > disk is 512/512, pool has ashift=3D9 > > (4) Server refuses to boot from ada1 (ada0 is empty) with diagnostics > (see above) > > (5) Linux rescue system, passing 2 devices to qemu with FreeBSD (becaus= e > Linux shows that ZFS is on whole disk, not on partition!). > > (6) Re-creation of GPT on ada0, start of resilver (with sub-optimal > ashift!). > > (7) Interruption of resilver with reboot, because it is painfully slow > under qemu. > > (8) Wipe of ada0 (at this point resilver status of pool becomes crazy) > to put live FreeBSD image to boot somehow. > > (9) Many tries to cancel resilver and boot from single-disk "historical= " > pool on ada1, no success. I've attributed it to the strange state of pool= : > one component, no mirrior, but "resilvering". > > (10) Boot from small UFS partition (which replaces swap partition). > > (11) Pool on ada1 (old, live, 512/512 disk) is still "Reslivering" > without any additional components (with zero speed, of course). > > (12) Prepare partitions on ada0 again, creating new pool with ashift=3D= 12, > send|receive. > > (13) Removing partition table on ada1 (with old pool, ashift=3D9, still > resilvering after many-many reboots with only one device in it). > > And pleas note: this pool on ada1 (old, live disk) was NOT upgraded > after 12-STABLE. It was old, 12-STABLE "level" pool with all new features > disabled. > Yea, this isn't *THAT*OtHER* problem :). Warner > -- > // Lev Serebryakov > > --0000000000006e66a9060e619437 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable


=
On Sun, Jan 7, 2024 at 1:57=E2=80=AFP= M Lev Serebryakov <lev@freebsd.org> wrote:
On = 07.01.2024 21:49, Lev Serebryakov wrote:

> On 07.01.2024 19:34, Warner Losh wrote:
>
>> I must have missed it. What were the diagnostics?

=C2=A0 Oh, and two "nvlist inconsistency" before that vvvv

> zio_read error: 5
> zio_read error: 5
> zio_read error: 5



So= , if this is a mirror, then ada0 blank and ada1 with good data, in theory
you should be fine. However, perhaps ZFS is finding that there'= ;s an error from
ada1 for real. Does all of ada1 read with a simp= le dd?

Not sure about the losing devices you descr= ibed later on.

> ZFS: i/o error - all block copies unavailable
> ZFS: can't read MOS of pool zroot
>
>
>=C2=A0 =C2=A0To be honest, I thinks there is something else. Because se= quence of events were (sorry, too long, but I think, tht every detail matte= rs here):

Yea. There's something th= at's failing, which zio_read is woefully under reporting for our diagno= stic efforts. And/or something is
getting confused by the blank d= isk and/or the partially resilvered disk.


> (1) Update to 13.2 from 12.4. With installation of new gptzfsboot with= gpart on both disks. It could place new /boot far away, but see (2)
> (2) Reboot, which completed, but showed that ada0 has problems
> (3) Replacement of ada0 by DC technicians, new disk is 512/4096, old d= isk is 512/512, pool has ashift=3D9
> (4) Server refuses to boot from ada1 (ada0 is empty) with diagnostics = (see above)
> (5) Linux rescue system, passing 2 devices to qemu with FreeBSD (becau= se Linux shows that ZFS is on whole disk, not on partition!).
> (6) Re-creation of GPT on ada0, start of resilver (with sub-optimal as= hift!).
> (7) Interruption of resilver with reboot, because it is painfully slow= under qemu.
> (8) Wipe of ada0 (at this point resilver status of pool becomes crazy)= to put live FreeBSD image to boot somehow.
> (9) Many tries to cancel resilver and boot from single-disk "hist= orical" pool on ada1, no success. I've attributed it to the strang= e state of pool: one component, no mirrior, but "resilvering". > (10) Boot from small UFS partition (which replaces swap partition). > (11) Pool on ada1 (old, live, 512/512 disk) is still "Reslivering= " without any additional components (with zero speed, of course).
> (12) Prepare partitions on ada0 again, creating new pool with ashift= =3D12, send|receive.
> (13) Removing partition table on ada1 (with old pool, ashift=3D9, stil= l resilvering after many-many reboots with only one device in it).

=C2=A0 And pleas note: this pool on ada1 (old, live disk) was NOT upgraded = after 12-STABLE. It was old, 12-STABLE "level" pool with all new = features disabled.

Yea, this isn't = *THAT*OtHER* problem :).=C2=A0

Warner
= =C2=A0
--
// Lev Serebryakov

--0000000000006e66a9060e619437--