BIOS /boot/loader is full, time to make some hard choices

Greetings,

I just wasted a day chasing a problem with /boot/loader being too large.

Due to a number of limitations in our infrastructure, we have a hard limit
of 640k hard limit of total memory usage (though we can use upper memory
for things like cache). 640k includes all the 'text' of the loader, plus
btx, plus data, bss and stack. The BIOS calls we make are limited to 1MB
due to calling them in 16-bit mode and needing to use SEGMENT:OFFSET
addressing for various addresses (which on the IBM-PC means 640k).

So, we're stuck with a limit of about 510,000 bytes for /boot/loader. Up
through at last FreeBSD/12 (2020) /boot/loader was about 330kB. Now we're
over 500kB due to a lot of extra things that have happened since then:
frame buffer, OpenZFS being larger and pulling in more crypto and
decompression software, and new filesystems. Between them all, we're
pushing the hard limits.

Now, we could in theory allow loading in high memory, have loadable modules
in the loader, etc. I don't like this idea because we're going to kill BIOS
in a few years and all these things add extra hair to a poorly tested, less
well supported path.

We need to subset what's in the BIOS loader. We need to make more things
optional, and we need to set a policy to know how to choose between this
and that when (not if) we run out of space again. I plan on making it
easier to compile out certain file systems globally, as well as certain
large features (possibly also some crypto / decompression algorithms).

The policy I'm proposing is to include as many filesystems, crypto and
compression in the boot loader that fit, in that order. And then other
features as space allows. This means that /boot/loader.efi will have all
these things (since there's no limits there), but for /boot/loader (BIOS)
we'll have a subset. If we still want to have the graphics support in the
BIOS loader for the installer, we'll have to leave out filesystem support,
crypto support, etc. For the cd image and/or the memstick image this is
fine: we only need to boot off of UFS/CD9660 (since we don't build ZFS
images yet), so we can have the graphics support, but since we install so
many different filesystems, with different features for crypto and/or
decompression, we may need to skip graphics for the default /boot/loader we
install for BIOS booting.

So, since we still have a little time, we have time to talk about the
future we want, but *I* also have only a limited amount of time to do
tooling here. Doing build stuff is in scope. Reworking things, say, so we
can have loadable modules likely isn't.

Warner