Re: Any known way to build devel/llvm* ( such as devel/llvm19 ) with --threads=1 for its linker activity during the build?

On Aug 6, 2024, at 01:56, meloun.michal@gmail.com wrote:

> On 05.08.2024 23:35, Mark Millard wrote:
>> On Aug 5, 2024, at 08:57, meloun.michal@gmail.com wrote:
>>> On 05.08.2024 11:09, Mark Millard wrote:
>>>> On Aug 5, 2024, at 00:44, meloun.michal@gmail.com wrote:
>>>>>> . . .
>>>>> Tegra has 4 Cortex-A15 cores and 2 GB of RAM.
>>>> OrangePi+ 2ed: Cortex-A7 with 4 cores and 2 GiBytes of RAM.
>>>> I wonder if the 2483 MiBytes would end up being about the
>>>> same on the Tegra variation indicated.
>>> 
>>> Yep, it must be +/-same. The 2/2 GB for userland/kernel is defined by HW.  Only size of shared libraries may affect (lower) usable user space for given program.
>>>>> All ports are built with default options. The only non-standard item is the swap size -> I have 16GB of swap on a swap partition on the SSD.
>>>> Wow, 16 GiBYtes of swap space for 2 GiBytes of RAM. I guess
>>>> when the swap is added that you get a notice-pair of the
>>>> structure:
>>>> QUOTE
>>>> warning: total configured swap (. . . pages) exceeds maximum recommended amount (. . . pages).
>>>> warning: increase kern.maxswzone or reduce amount of swap.
>>>> END QUOTE
>>>> with a rather large difference between the two ". . ." figures.
>>>> Do you make other adjustments to deal with the otherwise-reported
>>>> potential mistuning? It appears to make tradeoffs in the kernel
>>>> internal memory handling, if I understand right.
>>> The above message should be interpreted as: warning, the kernel may in word, rear case need to allocate additional
>>> memory when swapping some object(memory) out. This may leads to deadlock/panic. But again, event is this warning valid,
>>> resulting deadlock/panic is very rare. I newer see it in past many years...
>> Interesting.
>>>>> But I guess that's not important in this case.
>>>> At least for my context, it appears that memory allocations
>>>> are failing to find a big enough free area inside the
>>>> process's address space --without running out of system
>>>> RAM+SWAP space overall.
>>>> For the OrangePi+ 2ed ( and devel/llvm18 18.1.7 ) it was
>>>> during the earlier linker run for:
>>>> FAILED: bin/lli-child-target
>>>> . . .
>>>> LLVM ERROR: out of memory
>>>> Allocation failed
>>>> That much finished just fine on the Windows DevKit
>>>> 2023 used via a armv7 jail ( devel/llvm18 18.1.8_1 ).
>>>> The failure point was in a later link ( matching what
>>>> I saw via devel/llvm19 ).
>>>>> I just started build of llvm19 - but it takes few hours to complete..
>>>> Probably fewer hours than on the OrangePi+ 2ed but
>>>> more than on the Windows DevKit 2023 (if they were
>>>> completing, anyway).
>>> 
>>> The native build is still running (60% in fact), arm32 jail build has been stopped on my Honeycomb (killed by OOM).Unfortunately this is an old problem and is common on all platforms. The current LLVM cannot be built without additional tricks on machines that have less than 2GB (RAM + swap) per core.....
>> FYI: A small RAM+SWAP test for amd64-as-amd64 for devel/llvm18 18.1.7 . . .
>> USE_TMPFS=no
>> Targetting BE_NATIVE without BE_AMDGPU and without MLIR. Avoiding being stripped.
>> (MLIR is resource and time expensive to build but I make no use of it.)
>> (I also make no use of BE_AMDGPU materials.)
>> (I've not been doing any cross builds in a very long time.)
>> (Not stripping can make for better failure reporting.)
>> (Leading whitespace might notn b e preserved in the copy of the diff:)
>> # git -C /usr/ports/ diff devel/llvm18
>> diff --git a/devel/llvm18/Makefile b/devel/llvm18/Makefile
>> index 1b1f759ba50e..779ddf3bea7e 100644
>> --- a/devel/llvm18/Makefile
>> +++ b/devel/llvm18/Makefile
>> @@ -87,7 +87,7 @@ CMAKE_ARGS+=  -DLLVM_ENABLE_TERMINFO=OFF
>>  CMAKE_ARGS+=   -DLLVM_VERSION_SUFFIX=
>>    OPTIONS_DEFINE=        BE_AMDGPU BE_WASM CLANG COMPILER_RT DOCS LLD STATIC_LIBS
>> -OPTIONS_DEFAULT=       BE_AMDGPU BE_WASM CLANG LLD
>> +OPTIONS_DEFAULT=       BE_WASM CLANG LLD
>>  OPTIONS_SINGLE=                BACKENDS
>>  OPTIONS_SINGLE_BACKENDS=BE_FREEBSD BE_NATIVE BE_STANDARD
>>  OPTIONS_EXCLUDE_armv6= COMPILER_RT
>> @@ -103,7 +103,7 @@ OPTIONS_DEFINE_powerpc=     GOLD
>>  OPTIONS_DEFINE_powerpc64=      GOLD
>>  OPTIONS_DEFINE_powerpc64le=    GOLD
>>   -OPTIONS_DEFAULT+=      BE_STANDARD COMPILER_RT EXTRAS LIT LLDB MLIR OPENMP \
>> +OPTIONS_DEFAULT+=      BE_NATIVE COMPILER_RT EXTRAS LIT LLDB OPENMP \
>>                         PYCLANG POLLY STATIC_LIBS
>>  OPTIONS_DEFAULT_amd64= GOLD
>>  OPTIONS_DEFAULT_powerpc=       GOLD
>> @@ -211,8 +211,8 @@ CONFLICTS_INSTALL=  ${PORTNAME}${LLVM_SUFFIX} ${PORTNAME}${LLVM_SUFFIX}-lite
>>    .if defined(WITH_DEBUG)
>>  CMAKE_BUILD_TYPE=      RelWithDebInfo
>> -STRIP=
>>  .endif
>> +STRIP=
>>    PLIST_SUB+=    LLVM_MAJOR_MINOR=${LLVM_MAJOR_MINOR} \
>>                 LLVM_MAJOR=${LLVM_MAJOR} \
>> @@ -224,10 +224,10 @@ FIRST_COMMAND=    ${COMMANDS:C/^/XXXX/1:MXXXX*:C/^XXXX//}
>>    MAN1SRCS+=     ${LLVM_MAN1SRCS}
>>   -STRIP_LIBS=    BugpointPasses.so \
>> -               LLVMHello.so \
>> -               ${LIBNAME}.0 \
>> -               libLTO.so
>> +#STRIP_LIBS=   BugpointPasses.so \
>> +#              LLVMHello.so \
>> +#              ${LIBNAME}.0 \
>> +#              libLTO.so
>>    EXTRAS_LIBS=   \
>>                 libclangApplyReplacements \
>> Note: This amd64 testing is not using --threads=1 for linking.
>> A Hyper-V context for amd64-as-amd64 set up for small memory testing with 4 Hyper-V cores:
>> real memory  = 2147483648 (2048 MB)
>> avail memory = 2029309952 (1935 MB)
>> Swap: 3584Mi Total
>> So: AVAIL_RAM+SWAP == 5519 MiBytes
>> Building just llvm18-18.1.7 :
>> [00:00:22] [01] [00:00:00] Building   devel/llvm18@default | llvm18-18.1.7
>> [00:55:27] [01] [00:55:05] Finished   devel/llvm18@default | llvm18-18.1.7: Success ending TMPFS: 0.00 GiB
>> (My poudriere-devel is patched to report the ending TMPFS figures.)
>> From  my patched-up top's output from monitoring during the build:
>> Mem:  . . .                , 1462Mi MaxObsActive, 582432Ki MaxObsWired, 1940Mi MaxObs(Act+Wir+Lndry)
>> Swap: 3584Mi Total , . . . , 1090Mi MaxObsUsed, 2455Mi MaxObs(Act+Lndry+SwapUsed), 2995Mi MaxObs(A+Wir+L+SU),
>>                              2996Mi (A+W+L+SU+InAct)
>> So, slightly under 3 GiBytes RAM+SWAP observed as used, well under 8 GiBytes.
>> 3GiByte[RAM+SWAP]/4Core == 0.75 GiByte[RAM+SWAP]/Core, well under 2 GiByte[RAM+SWAP]/Core.
>> From this and the like of your OOM reporting, I get an idea how much MLIR, BE_AMDGPU, and BE_STANDARD contribute to the RAM+SWAP use. (I've not tested such directly in a long time.)
>> For reference:
>> # uname -apKU
>> FreeBSD 7950X3D-UFS 15.0-CURRENT FreeBSD 15.0-CURRENT #144 main-n271413-1f7df7570174-dirty: Sat Jul 27 16:01:52 UTC 2024     root@7950X3D-ZFS:/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/sys/GENERIC-NODBG amd64 amd64 1500021 1500021
>> # ~/fbsd-based-on-what-commit.sh -C /usr/main-src
>> 1f7df7570174 (HEAD -> main, freebsd/main, freebsd/HEAD) LinuxKPI: move __kmalloc from slab.h to slab.c
>> Author:     Bjoern A. Zeeb <bz@FreeBSD.org>
>> Commit:     Bjoern A. Zeeb <bz@FreeBSD.org>
>> CommitDate: 2024-07-26 11:51:04 +0000
>> branch: main
>> merge-base: 1f7df757017404011732196e65981d9325f7a89f
>> merge-base: CommitDate: 2024-07-26 11:51:04 +0000
>> n271413 (--first-parent --count for merge-base)
>> # ~/fbsd-based-on-what-commit.sh -C /usr/ports
>> 4e0616a2066d (HEAD -> main, freebsd/main, freebsd/HEAD) graphics/libavif: drop maintainership
>> Author:     Jan Beich <jbeich@FreeBSD.org>
>> Commit:     Jan Beich <jbeich@FreeBSD.org>
>> CommitDate: 2024-07-13 01:31:18 +0000
>> branch: main
>> merge-base: 4e0616a2066dc9e45cef47de812efdea4e09709c
>> merge-base: CommitDate: 2024-07-13 01:31:18 +0000
>> n671165 (--first-parent --count for merge-base)
>> Note: This /usr/ports/ is from before devel/llvm19 or /devel/freebsd-gcc14 . I've not synchronized this UFS environment with my experimention with a more recent ports vintage. devel/llvm18 is at 18.1.7 still here.
>> Hyper-V is actually using the same UFS boot media as the native UFS FreeBSD boot does --but is using a smaller swap partition.
> 
> So, I'm finally able to build llvm19, on native armv7 and also on arm32 jail.
> First It needs the not yet promoted patch
> 
> https://www.mail-archive.com/lldb-commits@lists.llvm.org/msg95950.html

Good to know to expect that. Thanks.

> But the more serious problem is that llvm's memory requirements are out of control.
> llvm ports with default configuration causes OOM on a 4-core armv7 with 4G RAM/16G swap, on a 4-core aarch64 with 4G RAM/16G swap and on a 16-core aarch64 with 16G RAM/16G swap.
> It's very hard to name this as correct/expected behavior. (Added dim@ just FYI).

As I understand it, the primary purpose of OPTIONS_DEFAULT is
to produce the set of packages intended to be published via the
package mirrors that contribute to https://pkg.freebsd.org/ .
So, for example, if any such item for a platform is to involve
MLIR from a devel/llvm* then that devel/llvm* needs to have MLIR
in OPTIONS_DEFAULT for that platform. (MLIR is just an example
here, but is a good one.)

As far as I can tell, there is no implication that folks building
their own packages (or just ports) should build with, for example,
MLIR unless a dependency involved requires such.

Going the other way, I do not see a reason to require all port
options to be "cheap enough" for general use for most everyone
that builds packages or ports.

So, if the build servers used to populate https://pkg.freebsd.org/
mirrors build what OPTIONS_DEFAULT indicates without OOM activity
problems, for example, that is all that is required relative to
OPTIONS_DEFAULT and OOM as far as I can tell. This might mean that
devel/llvm19 for armv7, for example, needs to not have MLIR in
OPTIONS_DEFAULT if the ampere* servers can not build devel/llvm19
for armv7 when MLIR is included. But it would appear that the
64-bit platforms are likely okay with MLIR included as long as
there are other packages (or FLANG use) dependent on MLIR that are
intended to be supplied as well.

Such contributes to why I tailor the OPTIONS_DEFAULT for the
devel/llvm* that I use (and, so, build) to avoid the likes of 
MLIR (that I do not use but has non-trivial consequences if
it is built): I do not expect the OPTIONS_DEFAULT's in the ports
git repository to happen to be appropriate for my context when I
run into some property that does not fit my context.

> Mark, I think that root case of your problems is that you're trying to link with libraries (shared or local) with debug information (unstriped).

Seems likely that more needs to be stripped (or never generated)
than before. I'll probably manage to regenerate the armv7 context
and test that at some point. It is definitely true that I've
historically kept more of such information around than FreeBSD
does on its own for optimized code: with symbols or debug
information. But it seems that such no longer fits in the armv7
context for what I've historically done.

> This causes a huge memory consumption. I don't remember if it just wastes address space (because  mmapping a large library but never accessing the debugging information) or if it accessing the debugging information (so the link needs physical memory for it as well).


===
Mark Millard
marklmi at yahoo.com