From nobody Fri Jan 06 19:08:20 2023 X-Original-To: fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4NpXtP1lxbz2pDVv for ; Fri, 6 Jan 2023 19:08:29 +0000 (UTC) (envelope-from bsdlists@jld3.net) Received: from mail.jld3.net (mail.jld3.net [45.55.236.93]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4NpXtN3N9fz3Pl3 for ; Fri, 6 Jan 2023 19:08:28 +0000 (UTC) (envelope-from bsdlists@jld3.net) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=jld3.net header.s=8d052f02dde2 header.b=PcQ6S59d; spf=pass (mx1.freebsd.org: domain of bsdlists@jld3.net designates 45.55.236.93 as permitted sender) smtp.mailfrom=bsdlists@jld3.net; dmarc=pass (policy=quarantine) header.from=jld3.net Received: from localhost (localhost [127.0.0.1]) by mail.jld3.net (Postfix) with ESMTP id 4248B41654 for ; Fri, 6 Jan 2023 12:08:22 -0700 (MST) X-Virus-Scanned: amavisd-new at jld3.net Received: from mail.jld3.net ([127.0.0.1]) by localhost (mail.jld3.net [127.0.0.1]) (amavisd-new, port 10024) with LMTP id p7xzjg94tbbq for ; Fri, 6 Jan 2023 12:08:21 -0700 (MST) Received: from [172.21.35.229] (c-24-9-144-115.hsd1.co.comcast.net [24.9.144.115]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) (Authenticated sender: jld) by mail.jld3.net (Postfix) with ESMTPSA id 646A140615 for ; Fri, 6 Jan 2023 12:08:21 -0700 (MST) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.jld3.net 646A140615 DKIM-Filter: OpenDKIM Filter v2.11.0 mail.jld3.net 646A140615 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=jld3.net; s=8d052f02dde2; t=1673032101; bh=OjUXQj43iAgf3QXb9TLQmVAP5kZPD4FkZbdN47L3kI8=; h=From:To:Subject:Date:Message-ID:MIME-Version:From; b=PcQ6S59dRWp58v/EIVP3ndUjj+hpZJwLCElh5nhDZgKFtxLCFhd3EMD6/MBgU4Rvj YbotTmcK65q2PJY5ttc5vZp4jsK9THxtxQxqPhWCjq4Njgj/3pP1hfjawZCQxBLu1r L/8CQM2/ckZ5QP1tCV3yK72p6W48cfUCR71wIrAw= From: "John Doherty" To: freebsd-fs Subject: ZFS: i/o error - all block copies unavailable Date: Fri, 06 Jan 2023 12:08:20 -0700 X-Mailer: MailMate (1.13.2r5673) Message-ID: <9DDC68E1-4321-45D9-831E-A73F70C66191@jld3.net> List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; format=flowed X-Spamd-Result: default: False [-4.00 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; DMARC_POLICY_ALLOW(-0.50)[jld3.net,quarantine]; R_SPF_ALLOW(-0.20)[+ip4:45.55.236.93]; R_DKIM_ALLOW(-0.20)[jld3.net:s=8d052f02dde2]; MIME_GOOD(-0.10)[text/plain]; MLMMJ_DEST(0.00)[fs@freebsd.org]; RCVD_VIA_SMTP_AUTH(0.00)[]; MIME_TRACE(0.00)[0:+]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; DKIM_TRACE(0.00)[jld3.net:+]; RCPT_COUNT_ONE(0.00)[1]; RCVD_COUNT_THREE(0.00)[4]; FROM_HAS_DN(0.00)[]; ARC_NA(0.00)[]; TO_DN_ALL(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; ASN(0.00)[asn:14061, ipnet:45.55.192.0/18, country:US]; PREVIOUSLY_DELIVERED(0.00)[fs@freebsd.org]; MID_RHS_MATCH_FROM(0.00)[] X-Rspamd-Queue-Id: 4NpXtN3N9fz3Pl3 X-Spamd-Bar: --- X-ThisMailContainsUnwantedMimeParts: N Posting this to freebsd-fs because the problem appears to be related to ZFS. I have a system running FreeBSD 12.3-RELEASE to which I have only remote access (about 3,300 miles from where I am). This morning I ran "freebsd-update fetch" and "freebsd-update install" on it. This was to get it up to date with 12.3 in preparation for a further update to 13.1-RELEASE. I rebooted the machine after "freebsd-update install" and it doesn't boot all the way. I connected to its IPMI and by recording the screen while trying to boot it, I was able to capture error messages that went by too fast to read otherwise: ZFS: i/o error - all block copies unavailable ZFS: can't read MOS of pool zp0 zp0 is not the pool from which the machine boots. It boots from a pool named zroot, which is a pair of mirrored SATA SSDs. zp0 is a pool of 24 SAS disks in the chassis. If I let the machine try to boot normally, it gets to a point at which I can ping both of its network interfaces (it has one 1GbE and one 10GbE) but sshd is apparently not running, or at any rate, attempts to connect via ssh just get me "Connection refused." Via IPMI, I can get to a beastie menu. Trying to boot the machine to single-user mode doesn't really get me anywhere and I never get a login prompt. Not sure if that's because the machine is not booting or some IPMI oddity. The IPMI text console display stops updating at "Loading kernel..." x8+0xla5dbc and that never changes. I'm pretty well stumped about how to proceed from here. Any advice or insights much appreciated.