Re: Failed to execute custom kernels which where build on a RPi 4 operated by 13.1-RELEASE

> Am 05.07.2022 um 10:51 schrieb John Kennedy <warlock@phouka.net>:
> 
>  Let me take a slightly different track here...
> 
>  I believe you've said that 14.0-CURRENT (which works without hanging) and
> 13.1-STABLE have fixed the RTC problem but you'd specifically like to be
> running 13.1-RELEASE.  I think you've implied that you've booted up on
> 13.1-RELEASE (but did have RTC problem, presumably because the changed
> haven't been MFCed all the way into the releng/13.1 branch.  The delay
> is the whole reason I'm running stable/13 myself, although like I said
> for amd64 reasons and just keeping the arm64 systems par.

I like RELEASE, because with that freebsd-update does work (remember arm64 is 1st tier since v13). Therefore, keeping the system updated is less hassle with RELEASE, compared to STABLE or CURRENT. That said, I kept my BeagleBone Blacks for years on CURRENT without big issues, by using an approach, which I lined out in a BLog post of mine:

https://obsigna.com/articles/1530995431.html

Actually, I switched my RPi 4 installation from 13.1-RELEASE to 14.0-CURRENT by this way, and if that matters, I could easily switch it back to 13.1 or something else. 

>   While Mark has made me a bit paranoid with what could go wrong at the
> pre-FreeBSD stages, if you can reliably recompile 14, those wouldn't
> seem to be an issue, if everything else is constant.
> 
>  Personally, I wouldn't run 14 in a production environment.  It's been
> very good to me in a bhyve environment, but it's not called -STABLE for
> a reason.  The reason I like releng/13.1 over stable/13 is that I have a
> Pavlovian urge to keep things patched.  So I can totally appreciate you
> wanting to run something stable and secure that doesn't get too many
> updates.  I'd just try to aim you at stable/13 vs main, at least if you
> can't backport the RTC fix into releng/13.1.

Again, I like RELEASE because I like the comfort of freebsd-update, nothing else. If I can't have this, there are reasons to choose CURRENT over STABLE. For example, Netflix does operate their streaming serves with CURRENT, and they explained why - hopefully I found the correct link here:

https://papers.freebsd.org/2021/eurobsdcon/gallatin-netflix-freebsd-400gbps/

For me the reason is, if something becomes broken, I switch back to the previous snapshot (using said approach), I report the issue and usually wait one week until it becomes fixed by the next snapshot.

>  But that being said, still a custom kernel of sorts, yes?

With 14.0-CURRENT the main reason for building a custom kernel is to increase the performance by building the NODEBUG one. Since I also want to do some experiments with the SDIO stuff, I choose the GENERIC-MMCCAM-NODEBUG variant for now.

https://wiki.freebsd.org/SDIO#Get_hands_dirty

Depending on the milage, I might switch back to GENERIC-NODEBUG in the future. 

>  So is your 14.x setup the same as your -RELEASE setup?

Yes, my clone utility (it is in the ports and a newer version is already in my GiHub repository: https://github.com/cyclaero/clone) allows me to pick exactly the system components from the SD card snapshot while leaving all my setup untouched.

>  When I bootstrapped myself initially, I burned a SDCARD and booted up (passed,
> so check).  Then I did the bsdinstall onto my USB disk, stomped on the
> EFI/msdos contents with the known-good SDCARD contents.  Yanked out
> SDCARD, booted up, ok, USB-only known-good there.

I build embedded mostly autonomous systems with the BBB and I will switch in the foreseeable future to RPi 4. These need a reliable RTC. 16 to 32 GB is more than sufficient, and when the customers need something updated or the SD card becomes broken, then I send them a new SD card by mail. Important data should anyway be frequently transferred via VPN to either my remote cloud servers or directly to the servers of the customer. In this scenario the SD card is a consumable. The BBB's are set up to run from the internal 4 GB flash, in case the SD card becomes broken. For RPi systems, I will need to ship spare SD cards witch may operate the system. If something goes wrong, the customer may replace the SD card and the system would be up and running again in no time.

>  Happily, I could pkg-install things (like git) to recompile the
> kernel+world.  Reboot with that, "custom" kernel check.  Recompiling all
> the packages locally was a nice stress-test.

I utilize a dual mode approach for updating ports and packages. That works very well for years now:

https://obsigna.com/articles/1519780374.html

>  I don't think "downgrading" from 14 to 13 is generally recommended (in
> my case, ZFS options would generally get me, but I believe you said
> you're UFS).

I hate ZFS. Overly sophisticated for no real benefit.

>  Basically, you've got situations where uboot can hand off to modern
> FreeBSD kernels on your hardware.  I think you're seeing a lot of lines
> of text because we're being paranoid about blind spots where you can't
> see some missing error message that would pinpoint the problem.

I don't want to mess around with u-boot. If the stock one does not work, then something is broken and needs to be reported. That said, I don't believe, that the present issue is an u-boot one.

>  All else being equal, if you can boot a stock 13.1-RELEASE kernel
> image, you should be able to recompile that kernel in place and expect
> it to boot.  So we're looking for the "not equal", something that we're
> assuming that is wrong.

That would be the second step. The first step would be that somebody else confirms my finding that building and running a custom kernel on a stock FreeBSD 13.1-RELEASE on RPi 4 does not work out. And actually that was my initial question.

- In case somebody raises her/his hand telling, that this worked flawlessly on their system,
  then I would have a more in deep look, what might have gone wrong here.

- In case the issue would be confirmed, then I would submit a bug report, and the discussion
  may continue in a more productive way on bugs.freebsd.org.