From nobody Mon Nov 27 01:10:47 2023 X-Original-To: freebsd-stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4SdnbH4l7Qz52Svj for ; Mon, 27 Nov 2023 01:11:07 +0000 (UTC) (envelope-from areilly@bigpond.net.au) Received: from clamta10.bpe.bigpond.com (clamta10.bpe.bigpond.com [203.42.22.26]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4SdnbD64S4z4Hm5 for ; Mon, 27 Nov 2023 01:11:04 +0000 (UTC) (envelope-from areilly@bigpond.net.au) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=bigpond.net.au header.s=202303 header.b="lY+37Aj "; spf=pass (mx1.freebsd.org: domain of areilly@bigpond.net.au designates 203.42.22.26 as permitted sender) smtp.mailfrom=areilly@bigpond.net.au; dmarc=pass (policy=reject) header.from=bigpond.net.au DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=bigpond.net.au; s=202303; h=To:Date:Message-Id:Subject:Mime-Version: Content-Type:From; bh=u3cGnVLKY8Zpzszur8LkugUO1rNpFr6GtutemYLYTYU=; b=lY+37Aj hVP9tdWoZRrXmUeUNqxyiXp359ymkLvqz9F32OQ3Iqt3k2x+W0r5DhvRkmBwZp8buKxEWPhnCv5MZ YW0Z4WEZHd5ZdoxludeGWBZxIbB4oE4JXZzooVJKwLRrg0lBhP6HfoeL4Jlrg21E5kBObG1V9f3ph 0GamfCUjysorohUnsgy6HoIkw2iDUxFYNP12XBTeNtF1f94z7ev/EijZeqKN7rK5EuQz8Lm+nbf9D nn1REt0U9jx7lbzPSaHIYJRLg0Eoic/Jdz2RtVRuqJeQ/6e7hk2WYyHutVA15ZDvB44EhVyMFIjg5 q+3pDFmARhXSKo2HaXlFNeTeQzg==; Received: from claprdcmr03 by claprdomr10 with esmtp (envelope-from ) id 1r7Q9L-000EaF-1j for freebsd-stable@freebsd.org; Mon, 27 Nov 2023 12:10:59 +1100 Received: from [121.223.155.16] (helo=smtpclient.apple) by claprdcmr03 with esmtpa (envelope-from ) id 1r7Q9L-0004FR-0m for freebsd-stable@freebsd.org; Mon, 27 Nov 2023 12:10:59 +1100 From: Andrew Reilly Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3774.200.91.1.1\)) Subject: An amusing comedy of errors: my FreeBSD-14 upgrade Message-Id: <8C86654F-C686-4878-8764-A5C4E90B4264@bigpond.net.au> Date: Mon, 27 Nov 2023 12:10:47 +1100 To: freebsd-stable@freebsd.org X-Mailer: Apple Mail (2.3774.200.91.1.1) X-tce-id: areilly@bigpond.net.au X-tce-ares-id: e{0afa20f9-23d3-4b05-bdf2-49f3e3631a68}1 X-tce-spam-action: no action X-tce-spam-score: 0.0 X-Cm-Analysis: v=2.4 cv=IO/ESCjG c=1 sm=1 tr=0 ts=6563eca3 a=g7JjhvAvvPZ9ycLFhywHJA==:117 a=g7JjhvAvvPZ9ycLFhywHJA==:17 a=kj9zAlcOel0A:10 a=BNY50KLci1gA:10 a=pvAka7dGAAAA:8 a=YvgnshHwSUfJoFZQJ_AA:9 a=CjuIK1q_8ugA:10 a=2JfvvIpjdqwvDtMeNY0X:22 X-Cm-Envelope: MS4xfARo6o7onwm+WjHBVQ94l9eDuiPkblepbJDcNSumHqzpSiGiAFWEmsywzKftq+qlvU/S7rRgwtsejdDlXQw/tgNYeAz7hsh8T6kB4Q3LLmJ2Sx9Dp/aM kiQayWsbUsnFxsFzqusJ/7YMMst0nMzUZLb0ZLuGPJr4ahqb48DgifUWsmv2wK4XGAczwB8XuN1KEA== X-Spamd-Result: default: False [-3.22 / 15.00]; NEURAL_HAM_SHORT(-1.00)[-1.000]; NEURAL_HAM_LONG(-0.95)[-0.946]; NEURAL_HAM_MEDIUM(-0.68)[-0.677]; DMARC_POLICY_ALLOW(-0.50)[bigpond.net.au,reject]; MV_CASE(0.50)[]; R_DKIM_ALLOW(-0.20)[bigpond.net.au:s=202303]; R_SPF_ALLOW(-0.20)[+ip4:203.42.22.0/25]; MIME_GOOD(-0.10)[text/plain]; RCVD_IN_DNSWL_LOW(-0.10)[203.42.22.26:from]; RCVD_COUNT_TWO(0.00)[2]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:1221, ipnet:203.40.0.0/13, country:AU]; FREEMAIL_ENVFROM(0.00)[bigpond.net.au]; MLMMJ_DEST(0.00)[freebsd-stable@freebsd.org]; FROM_EQ_ENVFROM(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; DKIM_TRACE(0.00)[bigpond.net.au:+]; FROM_HAS_DN(0.00)[]; TO_DN_NONE(0.00)[]; FREEMAIL_FROM(0.00)[bigpond.net.au]; RCPT_COUNT_ONE(0.00)[1]; RCVD_TLS_LAST(0.00)[] X-Rspamd-Queue-Id: 4SdnbD64S4z4Hm5 X-Spamd-Bar: --- [TL/DR: -14 removed pam_opie.so , so leaving those = lines in pam.d/system etc prevented su/sudo;=20 zpool upgrade root drive without updating gptzfsboot boot loader = prevented rebooting.] Thought that I'd write this down, in case it helps anyone. The upgrade = from stable-13 to stable-14 that I made to my file server over the = weekend was one of the bumpier upgrades in my thirty year history with = FreeBSD (yes, I was there at the transition from the patchkit). I'm = happy to relate that all of the pain related here was self-inflicted, = and FreeBSD itself shone through with its delightful robustness and = straightforward nature. All ends well. My file server, for now, is a miniITX AMD Zen 1700 system with four 8T = spinning rust drives in a single RaidZ Zpool, about 250G of NVME M.2 = flash holding root, usr, var and swap (boot to ZFS) and a USB connected = backup system also running ZFS. I track -stable and update ports (with = portmaster), user-space and kernel weekly. The start was very simple, now that FreeBSD is on git: I just git = switch'ed to the stable/14 branch, which went without a hitch. Then I ran my usual weekly rebuild script, which got to the end without = fuss, but with the usual complaint from etcupdate that there were some = unresolved issues. I should have been paying more attention to that: I = was not using etcupdate correctly, and had not been since switching over = from mergemaster a year or so ago. Needless to say, misconfiguration = was the start of the troubles, and they kicked in immediately: I = couldn't reboot after the upgrade. I couldn't reboot or sudo or su = because my /etc/pam.d configuration still referred to pam_opie.so, = because I had not noticed that being removed. I _could_ still ssh into = the system because my ssh config had disabled pam. Didn't help though, = because I was still stuck as me, and couldn't edit the config files, = because of sudo (I've since rebuilt sudo to not use pam either!) Easy enough to fix, right? Power down and reboot into single-user mode = and go from there. Unfortunately I had ripped the graphics card out of = the system some long time ago as an attempt to keep a bit of heat out of = the box and had apparently lost it in a couple of intervening house = moves. Perhaps I'd donated it to the electronics recycling mob along = with a box of old cables and power supplies. Too late to go and get one = Saturday afternoon, I found a store some distance away that would sell = me one on Sunday morning. With new graphics card in hand, I powered = down, took the lid off the server, carefully lifted out the hard-drive = cage and installed the GPU. Plugged in monitor and keyboard and powered = up. Single user mode did the job: edited the pam.d files and was just about = to reboot when I checked zpool status to see why the boot messages had = said something about my main array operating in "degraded" mode. One = drive was apparently not found/attached. On closer inspection I = discovered that I'd dislocated the power supply plug when I took the = cage out. I'd fix that when I did the next power-down. But in the mean = time, zpool status had also taunted me with new features that I could = enable. So I did, on the root drive. And power-cycled. And stared = dumbly at the boot screen telling me that it couldn't find any bootable = drives, because the one that was there had an incompatible zpool = version. Aargh! The boot loader had not been updated! Couldn't even = get to single user mode to fix it. I downloaded the 14-release bootonly image from the FreeBSD web site and = found a suitable thumb drive to put it on. Power cycled the box again = and told the boot menu to boot from the thumb drive. There followed a = great deal of gpart footling while I tried to remember just how I had = the drives arranged, but in the end I found the magic incantation (gpart = bootcode -p /boot/gptzfsboot -i 1 nda0) to install the new version of = the boot loader. Rebooted again, this time to the main system, rather = than the thumb drive, and that worked. ZFS resilvered the previously = missing drive quicker than I could notice, and subsequent scrubs found = nothing in need of fixing. Everything is now hunky-dory. Thanks to the always-wonderful FreeBSD team for continuing to produce a = system that can be understood at sufficient detail to fairly easily dig = oneself out of what might otherwise be catastrophic misadventures!