From nobody Sun Jun 05 00:28:07 2022 X-Original-To: freebsd-arm@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 6EFC71BF68D0 for ; Sun, 5 Jun 2022 00:28:26 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic306-21.consmr.mail.gq1.yahoo.com (sonic306-21.consmr.mail.gq1.yahoo.com [98.137.68.84]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4LFyCF0NY7z3H7b for ; Sun, 5 Jun 2022 00:28:24 +0000 (UTC) (envelope-from marklmi@yahoo.com) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1654388896; bh=YwuC9ZyixVPy+Btx7B3nruDDp6cHOzO9Ps3YvjTOJpM=; h=Subject:From:In-Reply-To:Date:Cc:References:To:From:Subject:Reply-To; b=giI3DyVUGfcP2JjTyx9/2ZaODzsaBjI2Lc3oLT3Z3Enixc6YLdl/luPlTBafHisXB0GohvDk1r83aSc8oCYrMLAFjgJwJ4Nu64yRNuJKSc2BDRhOsVa0rZXuJP0/4xvrR5GIISdS7ZPLxcwaF5lGrGpoOJEOysTT7F4iNxsCSMk72hAP9Jgzh6SBM85ibZXZwtUUp522Ac1c80JfApZcitxQtnZEoy8nQMtBhQrkK3XXxgDNAeV07DgZdcA04ofOd0GbUT8E8QLY8NjhHaWSElb/3SiLSuMTHjfoALmkjxgmsF74qHALw2G3djE6798lXqfQFQguanVoQTEjcKeKQQ== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1654388896; bh=DKRBDfdooq13AgRWM4VReoZOfhqdf/nhThdD65e5Uqw=; h=X-Sonic-MF:Subject:From:Date:To:From:Subject; b=cP8jGRDbWEOn/Xe74g2mJXnL8vdFjK23f6ZcnpefbNZuW7KSNH3LB7QfzWpqzvj3JZipcLp1huJtYXRKgp8St20iGzrLiCedZPVb/0iQhRhKyDffhCTeRapP6+qkLaKnkoKDU/+LPZWncK5dx+9Ea8FqCR6s3UY5zOv6Opgt9vHUzwwxV8fLHsx+38BB55Nq7ALu44EcDcg+0mFqqvzAiaF4OK2mNv2lfs5oIhf2vLS7n9CeEnLFVDqa+cj+ZjD74ZjlQVgmi4RuDA92EsXUKtvrO8NId+oTPdoNpzPjfVY88AYM7KKxT6ekpgjY9Z/hnJPR9WEQB0Ei3cG1LBaAgg== X-YMail-OSG: Fn7BOdIVM1kOSLGpon.Jo7fuo3avRFdWyRZ2j_qLONLolmpeLgmdAu5QI8.g6ko aZ59bq3JZE5.7Nf7lgl9JHDbC9pJe8hJxZk93ielW0i4exwrSa1Hk3tDozCw3m1A_RWuQbwiNJY9 ud2ndEI8sl6WrOpA5O5ZyJLpbd7tDBZe5YE309blt0oEhwAi.O2JVCsC5ZCaluWDqkROkjBeu0Xn xvT5y88QZmIZ6wtB2ePpvBJy2A0m3y0WV_zx07Y6WUwLlAg7m97cw_.vLi23yFAwQ3T6tRvuS9ZG s8GhIYRINYDThbGFNRKhW8Z3k3lYVGiSMxhS0gJc6wZn2VYZLWwkwz73VPTmHCQ7Ln8.j._iG8KX yQvoLiKoFdrXz2nNNEvVgTEuNDMc0hAVyMFaiI._44iPtCDc0odpgXa0Ra.u6uNhRVdQ5p_x6t7Y oxYVVsIGrfTT3kKqGHerrD3xNAAmNYH.nEAKC7x.P6TaiuuxfM3h1H3ZIpujtVfrp2UshbzOOt4E 5TTV_nUxp8tlp8nxzXGmxd_EaksN.4hiajtnpbsOs_Pvs3ubsbDIoAtoO.VMFdIHA.yAhJ8YGogk oiSAERKX3e94PN8RxlCzU36aEzRN8qrdHMn7R66dU6rIHVYzenZtaKA4JxPv2cXQrduF2XzOuvNA LR7.5bOOoi51OotKIN1lVOVghofQsnj.tV5cyeUgLQupD7ojZPrG7tZkBeHeP0XCt7SlHPNDNy2G Q3OXZGPkv7H8OzE3Kf5IZawT9eKPc8_u2AaGOv0GQS6tA5W82uGcbO_h9et_4kokr1_tm04gPUo9 f7mTNmuMjC26oY_Hi_xzSfG3_YcfSiFvunYu79yBbl7kxLS2eBIBramqokwwF7ugA35SUAyWmeCQ fUzWhwhHwgWNqc1Z5zwrryxWG3v7sZQ70Ig3nV_YzvEda_loRwVC4eGWcSE.Wrexd.SwkaK5Camu XOSbrqzFe6XvrwFnUdDUfzS6wcq.zdALcGoyvNeMiUugXMwKRQj_HSv9XIouROAWCR_WxhfCQ2Ay 6jqjDV6OMygQfrWJCWIKcoJC1y6QUfLrOkJk75.oGEiQzO41K9UlcfG40ba8ztiHliOU_1o7uVJ2 jBt2QJMMXhKwvysqCvgs6oVnU02L6DrZy78jCPn9q365pfa9SoPoa8BxytXsXofmgU7mxYrcJpE1 LC3yRbzXCjtD32fejTpLebyPuc5Sm2sWxjPB4o6GvTR6sRqRXeA3nrNwa3JhG.qW9ZME0oZ5zAK1 .Vw1Xu26LrItKsLffSWgzIAcHXGi57Xthv81yP72fXAWmX929p.xq6SJeM8CLO9.1V22BTIG330M 9V3L4tQk40Z4gdax5b4xiDoAll9fRqwfnAaW5.eKcIyhnmnqLB2SazZqLECtDCuvJjcqB0znMbZY SrJYnoYp0SxzyjoxE30d18Mo429Xrkdtp9_uarnuGGHXEYUOkHtLloWOK.HQ.8XtPImobBXiYBFe yuitxVok1QjDsfdfuJwYc_qGqYQepwbF7yOL90v5iu5jQLk6vBwxQe9x8S5J0IvjkUbrcXWHf1jF S2nVcblCtAql9ibxMmgTHADaPbMvy4igct2HtEva5NOt9jok5Ck._Rho_kEblrR_8gZy_QP22XZu N6aSkkzf62135ETgSPu13OQX7TlWluLMCy7Qk.YzA06DshAvYZvQMTt9yVeE89.B6M_zu0loxB.R nZaBrnbRyDrQf49aWPqRYr7sxJlgVo0oFNuchN6UU.shsvUUIw5CUKFra2dAd4zLo_5gn6q3k4_Q uwHVLQCNCI6KplzFJLq4rfVwB1SlyhmJVRsXecSYFyeSFOwD.EpC3gB2ZAIi0QS1k_m18W_owXcr lpMrZw3HOGEDPWs9SVPNP4j98X5PCioMKH5rdOQWNai2EI4aE7pPhvMYQihbz.uF81AKDQAsq87x gmZ593qjdrcEjeRy1ZAAaFLGG.km8tp3W2yjgyD.IEF3Q3gUyLOFSSvIvF1wlgrhGC5PGB6rpV3p S7YLXNqC6szTEMFpsn.DR6H6tCvJ0B.A2zwk2neLLDLCrhSX0SVlK4hqq_ZU4g4xxv5PS7Q0V.ax 6RoG3TwQZvHa0obSibu1tPLTgmqZTkVhz9JpJmE0Xbda2.YjFrIdv0vkIhvfPQOnNrHN6RqVDHAg gdzBbLfhCNQowTyPK6Iy.nlq7QgPEmyNCoqNzmrrHisexXHo8Xy2hmgm0rw4HVRZAb3wZXL2q_hk FM9axOwnzkIrtgxnU1dem40V3vzIs6XHdXh9AhX91bxmZodJCPFwH8cNaxl3b2TcsIfx0L6b1E4p ZNZyMWBWHR2jvXUCp X-Sonic-MF: Received: from sonic.gate.mail.ne1.yahoo.com by sonic306.consmr.mail.gq1.yahoo.com with HTTP; Sun, 5 Jun 2022 00:28:16 +0000 Received: by hermes--canary-production-bf1-856dbf94db-lhjkg (Yahoo Inc. Hermes SMTP Server) with ESMTPA ID d4cfc0f019c7bcda5784e3a734314e0d; Sun, 05 Jun 2022 00:28:10 +0000 (UTC) Content-Type: text/plain; charset=utf-8 List-Id: Porting FreeBSD to ARM processors List-Archive: https://lists.freebsd.org/archives/freebsd-arm List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-arm@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 14.0 \(3654.120.0.1.13\)) Subject: Re: devel/llvm13 failed to reclaim memory on 8 GB Pi4 running -current From: Mark Millard In-Reply-To: <99rror60-3pnn-s3q6-2q70-5ss2p968r658@fncre.vasb> Date: Sat, 4 Jun 2022 17:28:07 -0700 Cc: Free BSD , bob prohaska , Mark Johnston Content-Transfer-Encoding: quoted-printable Message-Id: <324C9134-542F-4954-A0E6-E75A08491A8D@yahoo.com> References: <20220127164512.GA51200@www.zefox.net> <99rror60-3pnn-s3q6-2q70-5ss2p968r658@fncre.vasb> To: Marcin Cieslak X-Mailer: Apple Mail (2.3654.120.0.1.13) X-Rspamd-Queue-Id: 4LFyCF0NY7z3H7b X-Spamd-Bar: - Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yahoo.com header.s=s2048 header.b=giI3DyVU; dmarc=pass (policy=reject) header.from=yahoo.com; spf=pass (mx1.freebsd.org: domain of marklmi@yahoo.com designates 98.137.68.84 as permitted sender) smtp.mailfrom=marklmi@yahoo.com X-Spamd-Result: default: False [-1.59 / 15.00]; RCVD_TLS_LAST(0.00)[]; ARC_NA(0.00)[]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; NEURAL_HAM_MEDIUM(-0.98)[-0.983]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[4]; FREEMAIL_FROM(0.00)[yahoo.com]; MV_CASE(0.50)[]; MIME_GOOD(-0.10)[text/plain]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_SPAM_SHORT(0.90)[0.896]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[yahoo.com:+]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; RCVD_IN_DNSWL_NONE(0.00)[98.137.68.84:from]; MLMMJ_DEST(0.00)[freebsd-arm]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; RCVD_COUNT_TWO(0.00)[2]; MID_RHS_MATCH_FROM(0.00)[]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim] X-ThisMailContainsUnwantedMimeParts: N On 2022-Jun-4, at 14:49, Marcin Cieslak wrote: > On Fri, 28 Jan 2022, Mark Millard wrote: >=20 > [ Reviving old thread ] >=20 >> After that I intend runs with 30 GiBytes of swap (so RAM+SWAP 38 = GiBytes). >> Hopefully that will complete and I'll be able to report how much swap = was >> observed to have been used. >=20 > I thought I will get LLVM14 + OpenJDK built quickly so I fired up > a c6g.4xlarge AWS instance (16 vCPUs, 32 GB RAM), > or even c6g.8xlarge (32 vCPUs, 64 GB RAM) and even these > are unable to build llvm14 under FreeBSD 13.1-RELEASE > with poudri=C3=A8re enabling MAKE_JOBS for llvm build. It depends on the build options. Do you need fortran, a.k.a. flang ? Building flang requires building MLIR because building flang uses MLIR. Without needing flang, both can normally be turned off. flang does not build for armv7 and other 32-bit environments, so recently the /usr/ports/devel/llvm14/Makefile was modified to not have flang as a default for armv7 (or armv6). My understanding, the intent is to later also turn off MLIR for these. I have built devel/llvm14 with FLANG and MLIR disabled on a 8 GiByte RPi4B in an armv7 poudriere jail. This was as part of an ongoing "bulk -a -c" on the RPi4B. (I will not be able to let it run to completion but am testing how things go until I stop it.) The below notes are somewhat biased by my also having used BE_NATIVE for the devel/llvm14 build: ---Begin OPTIONS List--- =3D=3D=3D> The following configuration options are available for = llvm14-14.0.2: BE_AMDGPU=3Don: AMD GPU backend (required by mesa) BE_WASM=3Don: WebAssembly backend (required by firefox via wasi) CLANG=3Don: Build clang DOCS=3Don: Build and/or install documentation EXTRAS=3Don: Extra clang tools LIT=3Don: Install lit and FileCheck test tools LLD=3Don: Install lld, the LLVM linker LLDB=3Don: Install lldb, the LLVM debugger MLIR=3Doff: Multi-Level Intermediate Representation PYCLANG=3Don: Install python bindings to libclang =3D=3D=3D=3D> Options available for the single BACKENDS: you have to = select exactly one of them BE_FREEBSD=3Doff: Backends for FreeBSD architectures BE_NATIVE=3Don: Backend(s) for this architecture (ARM) BE_STANDARD=3Doff: All non-experimental backends =3D=3D=3D> Use 'make config' to modify these settings ---End OPTIONS List--- The RPi4B has 4 cores and I had poudriere using 4 builders and ALLOW_MAKE_JOBS=3D with no explicit constraint on the make jobs per builder, so implicitly 4. Thus it can have periods with load averages around 16 when when things do as I said to do. (Some ports do not limit themselves to the HWthread count.) Result: [54:41:45] [01] [13:40:48] Finished devel/llvm14 | llvm14-14.0.2: = Success Note that that is with 2 GiBytes of RAM per core, the same ratio that your report indicates. I will also report that I used a UFS context, not ZFS, and that my patched top indicated little swap usage during any of my "bulk -a -c" atttempt so far. (The top tracks and reports various MaxObs???? figures, i.e., "Maximum Observed" figures.) My environments do have /boot/loader.conf , /etc/sysctl.conf , and /usr/local/etc/poudriere.conf settings appropriate to avoiding OOM kills for long running [but bounded duration] build activity for the hardware that I have access to --and that is also part of why I prefer UFS for 2 GiBytes/HWthread for doing builds. (I do have access to 3 systems with 4 GiBytes per HWthread and on 2 of them I normally use ZFS, including for build activity.) > The build proceeds on 1 CPU only now Such was not true of my build reported above. Using one core at a time, the RPi4B would have taken much longer to do the build. > - and casual observation > with top confirms that compilation of certain C++ files > requires 7..8 GB of RAM. My understanding is that such has a known status for flang/MLIR being in use. > If 16 of them are built concurrently, > no wonder that there is not enough memory. >=20 > Files tha crashed my compiliation: >=20 > = /wrkdirs/usr/ports/devel/llvm14/work/llvm-project-14.0.4.src/flang/lib/Eva= luate/tools.cpp > = /wrkdirs/usr/ports/devel/llvm14/work/llvm-project-14.0.4.src/clang/lib/Sem= a/SemaOpenMP.cpp > = /wrkdirs/usr/ports/devel/llvm14/work/llvm-project-14.0.4.src/flang/lib/Sem= antics/check-omp-structure.cpp > = /wrkdirs/usr/ports/devel/llvm14/work/llvm-project-14.0.4.src/flang/lib/Eva= luate/fold-integer.cpp You do not report the related console messages related (or dmsg -a output or /var/log/messages content). I can think of multiple distinct possibilities for the reports that would have different implications. (But such need not be of any importance for you.) I see 3 /flang/ paths in the list of 4 paths, not surprising. (Your /usr/ports/ context is more recent than mine.) > Sure, poudri=C3=A8re could get Kubernetes-like capabilities to control > resources during the build one day, but aren't amounts of memory > needed to build the compiler excessive a bit? "the compiler" ignores a fair mount of what is built. Part of what I'm suggesting above is to configure the devel/llvm14 build to build fewer compilers/toolchains (no flang, no MLIR). You might not want, say, EXTRAS, or some other things I have "on". BE_NATIVE may not fit your usage context well but BE_FREEBSD could be considered. And so on. Hopefully my notes above are of some use unless you are required to use the default OPTIONS. Side note: The RPi4B bulk actually built llvm13 and rust with a vast part of the time frames overlapping and other things going through the other 2 builders. It later worked on llvm12, ghc, and suitesparse-graphblas with the vast part of the time being overlapped. Again: all 4 builders doing something. (suitesparse-graphblas does not last as many hours as the other 2.) No kernel OOM process kills. Under 300 MiByte swap used at any point. (ghc's haddock runs out of 32-bit process memory in a normal core-dump way, towards the end the ghc build, so ghc failed. But the build had been going for most of 24 hours before that point, as had llvm12.) =3D=3D=3D Mark Millard marklmi at yahoo.com