Re: (solved) Re: undefined symbol: __aarch64_ldadd8_acq_rel since llvm12 (mongodb44)

On 3 Sep 2021, at 10:31, Ronald Klop <ronald-lists@klop.ws> wrote:
> 
> Van: Ronald Klop <ronald-lists@klop.ws>
> Datum: zondag, 25 juli 2021 08:10
> Aan: FreeBSD Ports <freebsd-ports@freebsd.org>, Michal Meloun <mmel@FreeBSD.org>, Dimitry Andric <dim@FreeBSD.org>
> Onderwerp: (solved) Re: undefined symbol: __aarch64_ldadd8_acq_rel since llvm12 (mongodb44)
> 
> Van: Dimitry Andric <dim@FreeBSD.org>
> Datum: 19 juli 2021 11:01
> Aan: Ronald Klop <ronald-lists@klop.ws>
> CC: FreeBSD Ports <freebsd-ports@freebsd.org>, Michal Meloun <mmel@FreeBSD.org>
> Onderwerp: Re: undefined symbol: __aarch64_ldadd8_acq_rel since llvm12 (mongodb44)
> 
> >
> >
> > On 19 Jul 2021, at 09:29, Ronald Klop <ronald-lists@klop.ws> wrote:
> > > > > Does anybody have a clue why I get this error since clang 12 on aarch64/arm64?
> > > Do other ports have the same issue?
> > > > > Regards,
> > > Ronald.
> > > > > Van: Ronald Klop <ronald-lists@klop.ws>
> > > Datum: vrijdag, 16 juli 2021 09:43
> > > Aan: freebsd-arm@freebsd.org
> > > Onderwerp: undefined symbol: __aarch64_ldadd8_acq_rel since llvm12 (mongodb44)
> > >> Hi,
> > >> I'm also maintaining databases/mongodb44 and this gives undefined symbols since llvm12 (I think).
> > >> See: http://www.ipv6proxy.net/go.php?u=http://ampere2.nyi.freebsd.org/data/main-arm64-default/pf44e1c1de734_s63ca9ea4f3/logs/errors/mongodb44-4.4.6.log
> > >> And look for:
> > >> ld.lld: error: undefined symbol: __aarch64_ldadd8_acq_rel
> > >> There are a bunch of similar symbols not found while linking.
> > >> This compiles fine using llvm11 or on amd64.
> >
> > It turns out clang 12 now enables -moutline-atomics by default, but we
> > don't yet expose the necessary functions from compiler-rt. As a
> > temporary workaround, compile with -mno-outline-atomics. Meanwhile,
> > adding these functions to the compiler-rt lib is on my TODO list.
...
> Hi Dimitry,
> 
> I upgraded my poudriere and tried building mongodb50 with -moutline-atomics enabled.
> [00:00:56] Host OSVERSION: 1400032
> [00:00:56] Jail OSVERSION: 1400032
> ...
> 
> [09:33:20] ld.lld: error: undefined hidden symbol: __aarch64_have_lse_atomics
> [09:33:20] >>> referenced by outline_atomic_cas1_1.S:121
> [09:33:20] >>>               outline_atomic_cas1_1.o:(__aarch64_cas1_relax) in archive /usr/lib/libgcc.a
> [09:33:20] >>> referenced by outline_atomic_cas1_1.S:121
> [09:33:20] >>>               outline_atomic_cas1_1.o:(__aarch64_cas1_relax) in archive /usr/lib/libgcc.a
> [09:33:20] >>> referenced by outline_atomic_cas1_2.S:121
> [09:33:20] >>>               outline_atomic_cas1_2.o:(__aarch64_cas1_acq) in archive /usr/lib/libgcc.a
> [09:33:20] >>> referenced 57 more times
> [09:33:20] c++: error: linker command failed with exit code 1 (use -v to see invocation)
> [09:33:22] scons: *** [build/opt/mongo/shell/mongo] Error 1
> [09:33:58] ld.lld: error: undefined hidden symbol: __aarch64_have_lse_atomics
> [09:33:58] >>> referenced by outline_atomic_cas1_1.S:121
> [09:33:58] >>>               outline_atomic_cas1_1.o:(__aarch64_cas1_relax) in archive /usr/lib/libgcc.a
> [09:33:58] >>> referenced by outline_atomic_cas1_1.S:121
> [09:33:58] >>>               outline_atomic_cas1_1.o:(__aarch64_cas1_relax) in archive /usr/lib/libgcc.a
> [09:33:58] >>> referenced by outline_atomic_cas1_2.S:121
> [09:33:58] >>>               outline_atomic_cas1_2.o:(__aarch64_cas1_acq) in archive /usr/lib/libgcc.a
> [09:33:58] >>> referenced 57 more times
> [09:33:59] c++: error: linker command failed with exit code 1 (use -v to see invocation)
> [09:33:59] scons: *** [build/opt/mongo/db/mongod] Error 1

It turns out that this function is defined in
contrib/llvm-project/compiler-rt/lib/builtins/cpu_model.c, which we have
compiled only for x86 until now. But upstream added the
__aarch64_have_lse_atomics symbol and an initialization function for it.

I tried simply enabling this file for the libcompiler-rt build on
aarch64, but it turns out that the implementation is rather Linuxy, and
does not compile on FreeBSD:

contrib/llvm-project/compiler-rt/lib/builtins/cpu_model.c:778:25: error: implicit declaration of function 'getauxval' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
  unsigned long hwcap = getauxval(AT_HWCAP);
                        ^
1 error generated.

The implementation looks like:

// LSE support detection for out-of-line atomics
// using HWCAP and Auxiliary vector
_Bool __aarch64_have_lse_atomics
    __attribute__((visibility("hidden"), nocommon));
#if defined(__has_include)
#if __has_include(<sys/auxv.h>)
#include <sys/auxv.h>
#ifndef AT_HWCAP
#define AT_HWCAP 16
#endif
#ifndef HWCAP_ATOMICS
#define HWCAP_ATOMICS (1 << 8)
#endif
static void CONSTRUCTOR_ATTRIBUTE init_have_lse_atomics(void) {
  unsigned long hwcap = getauxval(AT_HWCAP);
  __aarch64_have_lse_atomics = (hwcap & HWCAP_ATOMICS) != 0;
}

but in FreeBSD we don't have getauxval(), and I'm unsure if AT_HWCAP
works on aarch64.

As this is all outside my domain of knowledge I'd appreciate if somebody
knows how to retrieve this information on our AArch64 implementation. :)

-Dimitry