From nobody Fri Jan 06 21:06:50 2023 X-Original-To: fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4NpbW15Kbmz2pTwP for ; Fri, 6 Jan 2023 21:06:53 +0000 (UTC) (envelope-from bsdlists@jld3.net) Received: from mail.jld3.net (mail.jld3.net [45.55.236.93]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4NpbW11p5yz3tqc for ; Fri, 6 Jan 2023 21:06:53 +0000 (UTC) (envelope-from bsdlists@jld3.net) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=jld3.net header.s=8d052f02dde2 header.b=Ezbm5FWk; spf=pass (mx1.freebsd.org: domain of bsdlists@jld3.net designates 45.55.236.93 as permitted sender) smtp.mailfrom=bsdlists@jld3.net; dmarc=pass (policy=quarantine) header.from=jld3.net Received: from localhost (localhost [127.0.0.1]) by mail.jld3.net (Postfix) with ESMTP id 8866F41654 for ; Fri, 6 Jan 2023 14:06:52 -0700 (MST) X-Virus-Scanned: amavisd-new at jld3.net Received: from mail.jld3.net ([127.0.0.1]) by localhost (mail.jld3.net [127.0.0.1]) (amavisd-new, port 10024) with LMTP id huDSKb2aHoRi for ; Fri, 6 Jan 2023 14:06:51 -0700 (MST) Received: from [172.21.35.229] (c-24-9-144-115.hsd1.co.comcast.net [24.9.144.115]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) (Authenticated sender: jld) by mail.jld3.net (Postfix) with ESMTPSA id 8DD9F4061A for ; Fri, 6 Jan 2023 14:06:51 -0700 (MST) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.jld3.net 8DD9F4061A DKIM-Filter: OpenDKIM Filter v2.11.0 mail.jld3.net 8DD9F4061A DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=jld3.net; s=8d052f02dde2; t=1673039211; bh=eA/FpytURlSNRlulfhoWXlSN1waRefvqliD6lbz9HWc=; h=From:To:Subject:Date:Message-ID:MIME-Version:From; b=Ezbm5FWkY7D6xGaCevyf2AiCXyv83tv1wbfs3PrqgfwsGX28MLk2FhrArBRT0SXxy L8JXGm9XZUdId8LRICE05oIZvWztWzuxOaUOkLqLvZL/L894WzKA2u0pmau4PD6zVM 8st/yCq304DdANugtRZfaXP2ao3KW9h0Zw1hn0i0= From: "John Doherty" To: freebsd-fs Subject: Re: ZFS: i/o error - all block copies unavailable Date: Fri, 06 Jan 2023 14:06:50 -0700 X-Mailer: MailMate (1.13.2r5673) Message-ID: <3C4D1747-53D4-4A5A-9153-5777D2D895CE@jld3.net> In-Reply-To: <867AADBE-60F8-4B54-BBE8-875D381DA296@jld3.net> References: <9DDC68E1-4321-45D9-831E-A73F70C66191@jld3.net> <48b9848c-2e47-0d4b-f8b0-6a8ef7a7c982@sentex.net> <867AADBE-60F8-4B54-BBE8-875D381DA296@jld3.net> List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset="UTF-8"; format=flowed Content-Transfer-Encoding: 8bit X-Spamd-Result: default: False [-4.00 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-0.999]; DMARC_POLICY_ALLOW(-0.50)[jld3.net,quarantine]; R_SPF_ALLOW(-0.20)[+ip4:45.55.236.93]; R_DKIM_ALLOW(-0.20)[jld3.net:s=8d052f02dde2]; MIME_GOOD(-0.10)[text/plain]; MLMMJ_DEST(0.00)[fs@freebsd.org]; RCVD_VIA_SMTP_AUTH(0.00)[]; MIME_TRACE(0.00)[0:+]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; DKIM_TRACE(0.00)[jld3.net:+]; RCPT_COUNT_ONE(0.00)[1]; RCVD_COUNT_THREE(0.00)[4]; FROM_HAS_DN(0.00)[]; ARC_NA(0.00)[]; TO_DN_ALL(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; ASN(0.00)[asn:14061, ipnet:45.55.192.0/18, country:US]; PREVIOUSLY_DELIVERED(0.00)[fs@freebsd.org]; MID_RHS_MATCH_FROM(0.00)[] X-Rspamd-Queue-Id: 4NpbW11p5yz3tqc X-Spamd-Bar: --- X-ThisMailContainsUnwantedMimeParts: N BIOS settings look fine. Boot option #1 is a Micron SSD as it should be. It's not clear which of the two installed devices like that it is. There's one on SATA port 0 and an identical one on SATA port 1. Also, occurred to me after my previous message that the machine can't be trying to boot from zp0, there's no OS installed there. If I let it try to boot normally, it gets far enough that I can ping its network interfaces, so it must be trying to boot from the zroot pool. Now wondering whether there's a problem with one of the two SSDs configured as the zroot pool. So far, don't see how to choose whether it boots from SATA P0 or SATA P1 but there's got to be a way, I think. On Fri 2023-01-06 12:17 PM MST -0700, wrote: > Thanks for your response, that's a good idea. The HDDs are connected > to a Supermicro AOC-S3008L-L8e, which has a Broadcom 3008 SAS > controller supported by mpt(4). I have to take a break now but will > try getting into the BIOS next. > > On Fri 2023-01-06 12:13 PM MST -0700, wrote: > > >> On 1/6/2023 2:08 PM, John Doherty wrote: >> > >>> Posting this to freebsd-fs because the problem appears to be related >>> to ZFS. >>> >>> I have a system running FreeBSD 12.3-RELEASE to which I have only >>> remote access (about 3,300 miles from where I am). >>> >>> This morning I ran "freebsd-update fetch" and "freebsd-update >>> install" on it. This was to get it up to date with 12.3 in >>> preparation for a further update to 13.1-RELEASE. >>> >>> I rebooted the machine after "freebsd-update install" and it doesn't >>> boot all the way. I connected to its IPMI and by recording the >>> screen while trying to boot it, I was able to capture error messages >>> that went by too fast to read otherwise: >>> >>> ZFS: i/o error - all block copies unavailable >>> ZFS: can't read MOS of pool zp0 >>> > >> >> Is the disk controller's driver being loaded by the kernel ? What is >> the driver ?  Is it possible that the disk order has been messed up >> and its trying to boot not from the boot pool, but from disks in zp0 >> and not trying zroot ? If you have ipmi, try and go into the BIOS and >> make sure the disk its trying to boot from first is indeed part of >> zroot