From nobody Mon Jan 08 11:15:21 2024 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4T7s174xCtz55PPc; Mon, 8 Jan 2024 11:15:23 +0000 (UTC) (envelope-from lev@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [96.47.72.83]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4T7s174RCkz50r4; Mon, 8 Jan 2024 11:15:23 +0000 (UTC) (envelope-from lev@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1704712523; h=from:from:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=K8zAIJCc5HIP1p9sowRu93OYK8OXP/FYEhTQxRIsymQ=; b=HBJ3LVu9sJsbLe4y7n66xSsP3LfuPJxkXsy7stS281ceJrZcJjbBCW1mRmxcFiSNh8Z8t7 RbVZ8p/GRjrCsbuqmM6z+7FJXCo6KmL6ePL/exf8rrnRhDCp30fJLfl32vDhLLdPoxGyZ7 qKF+PM7tiADRvYWT5G1BwQ6ZsadOn7LgdwOd+EnlmImLFfYAl6Sx1XpbIlxwtdlCuBA9P7 QSfWkKe5KP4VjLy6lR6PIbm0A9cuj5cyhnhwZhCbLo6PmACD4vYBRHKmHR9dfXJgOKGT9+ 6ItP14Yg7jl3Eb3hyQasDqQAEfEL1qFsHi1YQqnEqtcZPnl6gzxgTKA6RPR+sg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1704712523; h=from:from:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=K8zAIJCc5HIP1p9sowRu93OYK8OXP/FYEhTQxRIsymQ=; b=KzoVsmr2f+QKrAItmhcfVsZqpny6DL6CyaKjPyTSjfiOCTZVCPFX0Ydr0cq/6eZfn9M/rY aMPyyobTrg9S6IPPmJSCny3Xxvv0HedRvzclOJI5wj3KxEXTnv7FuG8YGTjBnYPfq9BEo+ yDVPbYpKzZw/k2GCNXfv4muMc7S/d4RebCIDXMxjVNa0zV0PRyWIIhmveEgFKkFZhIBtEk ESCpoTFcDwCWvvW9g9tDdBnfbxwjugquTNujIdCkMmaKoa26Emrf0DuGBnQcAZmoIpbvfW 0iVavhaZFYhbEaXFX/JzZN7xfbyZVrEoEVogBpcS7Rh3xdrwG15q+MpVtMOKPw== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1704712523; a=rsa-sha256; cv=none; b=LY7uSc4tSswc0vfvcXImkihRJHAnyZ8tjOBg/cJUsoK8Nj1IM0PzROMcaSq/okD3plgSVj YnR/96qXHSuhvWnjLE4R+l4vEvYX/bdP2Z0yOB0lpcnUXbXkETQeqSNTFtOALVE5rFsq94 RJcJfbplQveTbK5HsR27USKUMOXWNuIlPP6HWqm4rymRk+wMvIbNEo0SMhI9JSFMNju6G6 IMEEro4qd7jfMxydwWrcqAjRnpEuVe1QDXawVR25CRgWhxuFVvO0VIb7vBTbqE4nKchpxm mzHRqrfFOwnDUkOtKarX/V1F5e+OMuTN60DSM1UbLzexijNq8PayWCJI8ci0QA== Received: from onlyone.not-for.work (onlyone.not-for.work [148.251.9.81]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) (Authenticated sender: lev/mail) by smtp.freebsd.org (Postfix) with ESMTPSA id 4T7s172pgPz16gj; Mon, 8 Jan 2024 11:15:23 +0000 (UTC) (envelope-from lev@FreeBSD.org) Received: from [192.168.136.24] (83-84-181-95.cable.dynamic.v4.ziggo.nl [83.84.181.95]) (Authenticated sender: lev@serebryakov.spb.ru) by onlyone.not-for.work (Postfix) with ESMTPSA id 23D013C2; Mon, 8 Jan 2024 14:15:21 +0300 (MSK) Message-ID: Date: Mon, 8 Jan 2024 12:15:21 +0100 List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Reply-To: lev@FreeBSD.org Subject: Re: FreeBSD 13.2-STABLE can not boot from damaged mirror AND pool stuck in "resilver" state even without new devices. To: Larry Rosenman Cc: freebsd-fs , freebsd-stable References: <065f4f5c-f38b-45f4-b7e7-5248f871f7e6@FreeBSD.org> <2f91eeb7-430b-49e2-817b-5acd0f445fe9@FreeBSD.org> <962b242d-546f-46ce-9eb2-9bd2a10f4608@FreeBSD.org> <30315c170f7146a5e1a05e4a2eff3d1b@lerctr.org> Content-Language: en-US From: Lev Serebryakov Organization: FreeBSD In-Reply-To: <30315c170f7146a5e1a05e4a2eff3d1b@lerctr.org> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit On 08.01.2024 1:15, Larry Rosenman wrote: >>> So, if this is a mirror, then ada0 blank and ada1 with good data, in theory >>> you should be fine. However, perhaps ZFS is finding that there's an error from >>> ada1 for real. Does all of ada1 read with a simple dd? >>   Yep, it is read with dd, I've checked it >> >>> Not sure about the losing devices you described later on. >>> >>>      > ZFS: i/o error - all block copies unavailable >>>      > ZFS: can't read MOS of pool zroot >>>      > >>>      > >>>      >   To be honest, I thinks there is something else. Because sequence of events were (sorry, too long, but I think, tht every detail matters here): >>> >>> >>> Yea. There's something that's failing, which zio_read is woefully under reporting for our diagnostic efforts. And/or something is >>> getting confused by the blank disk and/or the partially resilvered disk. >> >>   My theory, that something is confused when one disk is 512/4096 and other is 512/512. >> >>   I want to check it on VM, but can not find VM that both (1) allows CMS boot and (2) allows to configure logical and physical sector of virtual HDD. >> >>   bhyve could configure sector sizes, but doesn't support BIOS, and VBox and qemu-system can not emulate sector sizes (or I can not google proper configuration). > > When I first saw this, I wonder what ashift is set to on the pool? old pool was with ashift=9, but new one is with ashift=12. -- // Lev Serebryakov