From nobody Thu Nov 07 18:57:39 2024 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4XkrtJ012Fz5cPGP; Thu, 07 Nov 2024 18:57:44 +0000 (UTC) (envelope-from void@f-m.fm) Received: from fhigh-a6-smtp.messagingengine.com (fhigh-a6-smtp.messagingengine.com [103.168.172.157]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4XkrtG4FsJz4ctZ; Thu, 7 Nov 2024 18:57:42 +0000 (UTC) (envelope-from void@f-m.fm) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=f-m.fm header.s=fm3 header.b=CQfwYDW7; dkim=pass header.d=messagingengine.com header.s=fm3 header.b=ANv+h9fT; spf=pass (mx1.freebsd.org: domain of void@f-m.fm designates 103.168.172.157 as permitted sender) smtp.mailfrom=void@f-m.fm; dmarc=pass (policy=none) header.from=f-m.fm Received: from phl-compute-04.internal (phl-compute-04.phl.internal [10.202.2.44]) by mailfhigh.phl.internal (Postfix) with ESMTP id 277D711401DA; Thu, 7 Nov 2024 13:57:42 -0500 (EST) Received: from phl-mailfrontend-02 ([10.202.2.163]) by phl-compute-04.internal (MEProxy); Thu, 07 Nov 2024 13:57:42 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=f-m.fm; h=cc:cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:subject :subject:to:to; s=fm3; t=1731005862; x=1731092262; bh=1gHSNj+3WB 2dXPGccRPCtUi8aWohcvBDtW6p+EKxWmU=; b=CQfwYDW7mi7sduHd7dFo/aL7eK 5G32sFdRh8dYTK7dPTqObnnkb2E6L6lIKClQ0zDX2xPPnp0+MiGMmichc2tjfwEe 61IwwJsmQ/luNcdW6gfJknf3q4n7Nar0XaD3DU2pWJkqe8Cnyf05pgPk7PpRV34P X1tmFXN5ilb7i60inwok/iDAJENtfdKvtkzCriOToxa7E+VBm2oHRJroF/Az8P8g jwTquDUZ8LWkKtKiJvGwHV2jXYoPGJzz93MVicODu1y1hez7Yus0w1GvRErx3UUm ReJ0zdpWXhW4kS8V4o5V5UQZn0xcz17fYHEdUe14Bb7TO74EzCdHRkc5FdQA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:subject:subject:to :to:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm3; t= 1731005862; x=1731092262; bh=1gHSNj+3WB2dXPGccRPCtUi8aWohcvBDtW6 p+EKxWmU=; b=ANv+h9fTtDP5/Zw+NkYywieoqxZHyKxezYHHSuXYh2MCC+6RRrF sGan234fnfJlyDXO09IT5746oHfQTuZ+6BSKiOVZmWKgqr2BtEebJ0k1XBHJN2/c 94+JQQfKMFCkDp7OfUAVcOgUDJv+pu6LY8/PUflKJ+AcTNzo0Ls7WPly9YU9XO5x kZSAY+D64FtvHuobZbUU5MULp1Vu/Qim6zq75t/AixThH3N5pjRVj/km3ty0t/9a SV87H2pDaf+8LTSz2x+ioom8c43meC894peHTZiG/puLmDd/AysUuweUxzpof/hO pdLz1zP5Xf7aG6Lcp3WBXBCeHJ+dbDkOt5Q== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeefuddrtdeggdduudekucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdggtfgfnhhsuhgsshgtrhhisggvpdfu rfetoffkrfgpnffqhgenuceurghilhhouhhtmecufedttdenucenucfjughrpeffhffvve fukfhfgggtuggjsehttdertddttddvnecuhfhrohhmpehvohhiugcuoehvohhiugesfhdq mhdrfhhmqeenucggtffrrghtthgvrhhnpeduvdeugeduudefteehheffvdegteelkeekvd evffdukeeugeeuteffteefkeelhfenucevlhhushhtvghrufhiiigvpedtnecurfgrrhgr mhepmhgrihhlfhhrohhmpehvohhiugesfhdqmhdrfhhmpdhnsggprhgtphhtthhopedvpd hmohguvgepshhmthhpohhuthdprhgtphhtthhopehfrhgvvggsshguqdhfshesfhhrvggv sghsugdrohhrghdprhgtphhtthhopehshihsihhnshhtrghllhesfhhrvggvsghsugdroh hrgh X-ME-Proxy: Feedback-ID: i2541463c:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Thu, 7 Nov 2024 13:57:41 -0500 (EST) Date: Thu, 7 Nov 2024 18:57:39 +0000 From: void To: freebsd-fs@freebsd.org Cc: sysinstall@freebsd.org Subject: Re: problem with zfs raidz3 install on an 8-disk system, can't find loader Message-ID: Mail-Followup-To: freebsd-fs@freebsd.org, sysinstall@freebsd.org References: <62b73206-0f14-423a-978e-7d553a9c263c@app.fastmail.com> List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@FreeBSD.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii; format=flowed Content-Disposition: inline In-Reply-To: <62b73206-0f14-423a-978e-7d553a9c263c@app.fastmail.com> X-Spamd-Result: default: False [-3.57 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.97)[-0.975]; MID_RHS_NOT_FQDN(0.50)[]; DMARC_POLICY_ALLOW(-0.50)[f-m.fm,none]; R_DKIM_ALLOW(-0.20)[f-m.fm:s=fm3,messagingengine.com:s=fm3]; R_SPF_ALLOW(-0.20)[+ip4:103.168.172.128/27]; RCVD_IN_DNSWL_LOW(-0.10)[103.168.172.157:from]; MIME_GOOD(-0.10)[text/plain]; RCPT_COUNT_TWO(0.00)[2]; RCVD_TLS_LAST(0.00)[]; FREEMAIL_FROM(0.00)[f-m.fm]; MIME_TRACE(0.00)[0:+]; RCVD_COUNT_THREE(0.00)[3]; ARC_NA(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; FROM_HAS_DN(0.00)[]; FREEMAIL_ENVFROM(0.00)[f-m.fm]; TO_DN_NONE(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; DKIM_TRACE(0.00)[f-m.fm:+,messagingengine.com:+]; MLMMJ_DEST(0.00)[freebsd-fs@freebsd.org,sysinstall@freebsd.org]; MISSING_XM_UA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; ASN(0.00)[asn:209242, ipnet:103.168.172.0/24, country:US]; DWL_DNSWL_NONE(0.00)[messagingengine.com:dkim] X-Rspamd-Queue-Id: 4XkrtG4FsJz4ctZ X-Spamd-Bar: --- On Thu, Nov 07, 2024 at 12:58:32PM +0000, Dave Cottlehuber wrote: >From inside the installer, it would be worth sharing with the list: > >- dmesg >- sysctl machdep.bootmethod >- relevant disk controller bits of `pciconf -lv` > >and comparing what you see when booting from h/w (not memstick.img) >to see if it comes up in UEFI or BIOS mode. > >While dealing with these issues I would start out first with a >single drive and see if FreeBSD will boot to "plain zfs" first. > >I'm not familiar with raidz3, but that's where I'd start off. alas, not possible now as it's for production. There's another, similar, machine I can test with though. I'll try that and report back. This other, similar machine has had freebsd since 12-current. it may have been installed as ufs and subsequently zfs added for data. Need to check. The particular machine (the one with the issue reported initially) was running -current but it was using the hpa smartarray. Nothing special was needed for the install back then. The smartarray presented 2x devices as it was configured for 2* 4 hds in raid mirror. The zfs was then installed as stripe, resulting in about (and this is from memory) 7Tb space. I wantd HBA (read: JBOD) mode for this system so I had to destroy the hardware array and put the card in HBA mode. When booted, the installer saw all the individual disks and apparently made a raidz3 as configured via the installer. *Currently* the installer presents all the disks, no special driver needed. What makes me think this is a bug is that everything can be added to the zpool but when the system boots, it can't boot. To get around the installer issue, I installed onto one of the disks as ufs (da0). This proceeded normally. Now with only 7 disks for zfs I made a raidz2: % zpool status pool: data state: ONLINE config: NAME STATE READ WRITE CKSUM data ONLINE 0 0 0 raidz2-0 ONLINE 0 0 0 da1 ONLINE 0 0 0 da2 ONLINE 0 0 0 da3 ONLINE 0 0 0 da4 ONLINE 0 0 0 da5 ONLINE 0 0 0 da6 ONLINE 0 0 0 da7 ONLINE 0 0 0 This appears to function normally. The issue *appears* to be that the installer doesn't install the guts of what is required to boot the array, despite configuring and apparently installing everything else. This is similar to a long-standing issue affecting installing zfs on arm64. --