Re: /stable/12 future

On 8 Dec 2021, at 21:58, Dimitry Andric <dim@FreeBSD.org> wrote:
> 
> On 8 Dec 2021, at 20:47, Jan Beich <jbeich@FreeBSD.org> wrote:
>> 
>> 12.3-RELEASE still uses Clang/libc++ 10 from 1 year ago. Do you plan
>> to update in future as /stable/12 is supported until 2024-06-30?
>> 
>> libc++ lags behind libstdc++ on C++20 features and sticking to an old
>> version puts the entire branch on a deathbed. Given drm-kmod on
>> /stable/12 is stuck with Linux 4.16 era (discontinued after 2018-06-26)
>> /stable/12 is already mostly dead from desktop POV.
>> 
>> See also
>> https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=215193
>> https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=260139`
> 
> I'd rather merge llvm-project 13.0.0 to stable/12, but I didn't want to
> do this just before 12.3 wrapped, and it's also a bit of a hassle to
> figure out all the cherry-pickings... Since stable/12 lags behind on
> many other build infrastructure improvements, some commits don't apply
> cleanly.

So I'm working on this now, and making some progress. The first steps
have been to merge everything up to llvm-project 11.0.1.

While almost everything in "make universe" works, I have hit one pretty
annoying snag: the arm.arm target (aka "armv4"). This has been dropped
in FreeBSD 13 and later, due to it being old and missing lots of
features, in favor of armv6 and armv7.

I have been able to work around things such as missing atomic support,
and some other issues. But the big problem with this target is that it
was frankenstein'd into using clang as its compiler, while still using
the ancient BFD ld 2.17.50!

With clang 11 and higher, I quickly ran into issues that this ancient
version of BFD ld chokes on ELF files with more than 64k sections, and
these are now produced pretty often, especially if you are using
-ffunction-sections -fdata-sections. However, C++ programs also emit
lots of comdat sections, and due to some arm specific shenanigans, this
costs 4 sections per symbol! So any largish C++ program goes over 64k
sections, for example clang and lld themselves, but also the googletest
suite we have in-tree.

I did some spelunking in the binutils git repository, and I found that
there is an upstream commit that fixes the 64k sections limit.
Unfortunately, this is a big commit that is also under GPLv3, so we
cannot import it as-is. And even if we got permission from the original
author to use it under GPLv2, it would still not solve the problem that
the old BFD ld is *extremely* slow on these files. Case in point,
linking clang.full with it took >60 minutes, even on a fairly fast
machine!

The obvious solution would seem to be to switch to lld as the default
linker, however that also runs into a snag. Since lld uses the blx
instruction for interworking, it prints a warning "lld uses blx
instruction, no object with architecture supporting feature detected".
And because we link with the --fatal-warnings option, this stops
buildworld dead in its tracks. Obviously, this warning could be ignored
(by removing --fatal-warnings, or patching it out), but I think this
might result in issues when the actual binaries are run.

Unfortunately I have no way to verify whether this is the case, so my
current working solution is to switch arm.arm back to good old gcc
4.2.1, which should work fine in combination with the equally ancient
BFD ld 2.17.50. Since this combination has bitrotted a little in mean
time, it was required to add a few minor patches to the tree to make it
build. But I expect the produced binaries should run OK on actual armv4
hardware.

So the executive summary is:
* clang 11 and higher (up to 13) can most likely be backported to
  stable/12
* except for arm.arm (armv4) which will have to choose between:
  1. clang/ld.lld >11 with possible blx instructions, no idea if this
     will work on actual hardware
  2. gcc/ld.bfd which should always have worked, but no shiny new clang
     or lld

That last option would also mean that many ports can't be built of
course. But it seems hardly relevant, as nobody is going to natively
build Chromium or Firefox on those low-powered machines?

In any case, I would like to solicit some feedback about this, so I can
continue merging. Personally my preferred solution would be to switch
arm.arm to gcc, so it can deorbit in peace. Otherwise somebody with
access to this hardware should verify that a world linked with lld
actually runs; that might not be trivial.

-Dimitry