From nobody Mon Jan 08 00:15:47 2024 X-Original-To: freebsd-stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4T7ZND4PXjz56N8d; Mon, 8 Jan 2024 00:15:56 +0000 (UTC) (envelope-from ler@lerctr.org) Received: from thebighonker.lerctr.org (thebighonker.lerctr.org [192.147.25.65]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "*.lerctr.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4T7ZND27c7z4cHJ; Mon, 8 Jan 2024 00:15:56 +0000 (UTC) (envelope-from ler@lerctr.org) Authentication-Results: mx1.freebsd.org; none DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lerctr.org; s=ler2019; h=Content-Transfer-Encoding:Content-Type:Message-ID:References: In-Reply-To:Subject:Cc:To:From:Date:MIME-Version:Sender:Reply-To:Content-ID: Content-Description; bh=g7z0hYleGgP4Nu4An7VBd3Wr1PnrWOWntywRHGB3OFo=; b=myI5H mSmpdQIWt4reJPJ5eVYJU661LQ7RGjz910HiU1sgHBVVoVAe1OSLjSD8jJwwGi0/CyxEI9EM9wzaA PuWU++4eGNq8fUWx8UyKtcg0NDRMk2mTZT8VMhbkIfas/Z7i5TEofgW6sD3Nr+N++QrBZj8JFH080 k5vFhIT6Nik4+CYiN8o7iBb13Py0n3mCeQLwvdyGPxt05NmmByJItLjg1TYDTTtgvzUq2KHB7uAc4 p8OcRWMA820WYeru14a1ozmmpSADfIRqRw7svWmbaqnGhg8bN3yu7VA72LsOa5N4gH7NlkX6S9cSx 3zXuAPlqGfwEUw9ztW72nigJagGFA==; Received-SPF: pass (thebighonker.lerctr.org: domain of lerctr.org designates 192.147.25.65 as permitted sender) client-ip=192.147.25.65; envelope-from=ler@lerctr.org; helo=webmail.lerctr.org; Received: from thebighonker.lerctr.org ([192.147.25.65]:39661 helo=webmail.lerctr.org) by thebighonker.lerctr.org with esmtpsa (TLS1.3) tls TLS_AES_256_GCM_SHA384 (Exim 4.97.1 (FreeBSD)) (envelope-from ) id 1rMdIx-00000000BsB-1MKe; Sun, 07 Jan 2024 18:15:47 -0600 Received: from 99-190-128-217.lightspeed.austtx.sbcglobal.net ([99.190.128.217]) by webmail.lerctr.org with HTTP (HTTP/1.1 POST); Sun, 07 Jan 2024 18:15:47 -0600 List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org MIME-Version: 1.0 Date: Sun, 07 Jan 2024 18:15:47 -0600 From: Larry Rosenman To: lev@freebsd.org Cc: Warner Losh , freebsd-fs , freebsd-stable Subject: Re: FreeBSD 13.2-STABLE can not boot from damaged mirror AND pool stuck in "resilver" state even without new devices. In-Reply-To: <962b242d-546f-46ce-9eb2-9bd2a10f4608@FreeBSD.org> References: <065f4f5c-f38b-45f4-b7e7-5248f871f7e6@FreeBSD.org> <2f91eeb7-430b-49e2-817b-5acd0f445fe9@FreeBSD.org> <962b242d-546f-46ce-9eb2-9bd2a10f4608@FreeBSD.org> Message-ID: <30315c170f7146a5e1a05e4a2eff3d1b@lerctr.org> X-Sender: ler@lerctr.org Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: 4T7ZND27c7z4cHJ X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:18474, ipnet:192.147.25.0/24, country:US] On 01/07/2024 3:19 pm, Lev Serebryakov wrote: > On 07.01.2024 22:15, Warner Losh wrote: > >> So, if this is a mirror, then ada0 blank and ada1 with good data, in >> theory >> you should be fine. However, perhaps ZFS is finding that there's an >> error from >> ada1 for real. Does all of ada1 read with a simple dd? > Yep, it is read with dd, I've checked it > >> Not sure about the losing devices you described later on. >> >> > ZFS: i/o error - all block copies unavailable >> > ZFS: can't read MOS of pool zroot >> > >> > >> >   To be honest, I thinks there is something else. Because >> sequence of events were (sorry, too long, but I think, tht every >> detail matters here): >> >> >> Yea. There's something that's failing, which zio_read is woefully >> under reporting for our diagnostic efforts. And/or something is >> getting confused by the blank disk and/or the partially resilvered >> disk. > > My theory, that something is confused when one disk is 512/4096 and > other is 512/512. > > I want to check it on VM, but can not find VM that both (1) allows > CMS boot and (2) allows to configure logical and physical sector of > virtual HDD. > > bhyve could configure sector sizes, but doesn't support BIOS, and > VBox and qemu-system can not emulate sector sizes (or I can not google > proper configuration). When I first saw this, I wonder what ashift is set to on the pool? -- Larry Rosenman http://www.lerctr.org/~ler Phone: +1 214-642-9640 E-Mail: ler@lerctr.org US Mail: 5708 Sabbia Dr, Round Rock, TX 78665-2106