From nobody Thu Apr 06 20:53:04 2023 X-Original-To: freebsd-arm@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Pstxv5lNYz43MCB for ; Thu, 6 Apr 2023 20:53:23 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic313-20.consmr.mail.gq1.yahoo.com (sonic313-20.consmr.mail.gq1.yahoo.com [98.137.65.83]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4Pstxt4bwpz3qth for ; Thu, 6 Apr 2023 20:53:22 +0000 (UTC) (envelope-from marklmi@yahoo.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yahoo.com header.s=s2048 header.b=Punfhy2f; spf=pass (mx1.freebsd.org: domain of marklmi@yahoo.com designates 98.137.65.83 as permitted sender) smtp.mailfrom=marklmi@yahoo.com; dmarc=pass (policy=reject) header.from=yahoo.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1680814400; bh=XZvlY7kpB5UbNoEXiw5xoIy7lmqlmdfRg6yDsEwb3Bg=; h=Subject:From:In-Reply-To:Date:Cc:References:To:From:Subject:Reply-To; b=Punfhy2fjaaAODk2+86Cg88ZkVtRD+GWDUWTwOan5Dj+yfl6Ey8I+hhJy4wJkvMB5+Sx8NbPqFYgYgSeCHkDIxaQUjI9iafLbVmgUaeV2jx1pzZMKydxjSa3Nzxn8gOIjKWUS6wX7LagwpClO0jvgAuJ4UIJNPQlEt6FFR+en6g7KaCj1x0xLp3XWpMMIdTwagKrlOq7oihP5Tu6MRi5rgCZUvgI1OgoTBTjhR7KiLCvoGJa0lam4VbMZ/cuiTAhx1kZXSFQ67AucFEJR2pgy8hU0pu6Xn+Miv6Ut3rYEVFjpFAGg9nFmHnPkmsm8aIT/dMTHvRT129ozuiL0lkoCA== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1680814400; bh=0jaX7OF/ojUDIcVmHBtADXHJ5HBYxYU5pG3Lkaf+edk=; h=X-Sonic-MF:Subject:From:Date:To:From:Subject; b=ZWDgi2KGNI1dQur1naIQzgx3ONsKqMarvZyX/c6N3GlIdQjF4G4tJumM14zdmBIKDZJY2Akks8uzc6Fz9P+EyGHOcZlgF9mXpnDmGH7DuRAK1hWA+ZBSi/i9WCuuPUGCWU8GHCOzD+D3+RfQXuYBhMmUT5zQCt/9PbvUGG1fMFRb0/i2TqeED5a22eYiANNrhXoZU9wviUMctz7YyDff3Rj0VUAdFP/w2DoHN7g/J77gO6wmKg6PxQWIjg/1hbivdkYFsXnotPJ5p/2m1FY2fzr95rMNERtlhSQTXCgEBESKt23UoUAVGZCnjqGqSlIEsiwAdarpcpTG4Djw6MhDlA== X-YMail-OSG: Wz7Oa.MVM1liX_fNvYHN75rHXIDhqLvUo1DojjyIgFIiEiGP477J0yoUJzoul1R rIyDs0V5qnh4uk4_US4MK03ban8P6ndcW6m2vnHZmL0Iu5bgBiufb1Zzd5ageW4t1Sr8oLDYpGBu fVAvCK5bOTb9OGxtVU0V..fy_EiTW6V9km1OerzdVOrqL.xKu_MqjviB9CqwKhvrc65UtbLMcyIt 2ickliJfniqiF3j8vLjl6BhtxjBSm6fvm5oit3nPQcYK6l0gr6YFFkXeiSVpa4ZhHzSprUmRND0S Wcq2dPXT7ZJekqzy2lmsctkq7zeVxZzbREhzm5ijneElvi55jcyCjDHwBZ.IszCj09N0x6FIur5N Ri2PZ5jdeBtyVhqwX1j9ekh5cXom23hGk0vVS3fcfCrHB1cS_xgwl4EWcedh27iIbmuxy4u8Qc9o UENx2C0_mB5W3egVM_EKMMoyOiQqTtLhfoLBII.kVGq8LElucvrVWS4q798uDGBE10F8Pkkq4tNb JslIYHXYSPV12Tsil2cUlqnbOhvkOJkTMIcxPRDAYRrB1eoRx_ElULPdORIzVFM3DPAwtneXBFPF svWdUFD3HZsqEANQuSy8Q1z_amMPTLGOG78199G9ts9T3IelF0AzJWqsdieqMYiNeLBe8waYpp4U SySnx628DZ00rZmt5Xt3N0IA9RhQh614Wdur.yFZtgs1qXsCmNqXAoo08KIUzhHVXzRhWEEZJ63t xGI__lv1rXd7QUXzACXGnf9ryFIEx.thPx.Cdua9Rf7TBz3jc5Z9xwue7ditr5vzYVWxLaf9VLT1 3bSGJkGMER6dROWB5GFg1khGHpS1N2ae.w5goEWBUMbzZdSekHGCQSKIOIt.EeGumNdUQ1vng5RA aoAgH6bmJpbd_Yr963B.Wv6NKWeuhKgNB3Eq4EwOWWCv13ZZ91IB914V5ISxU1ALkrp7Jskcb2Sw ASDOJTaoTRfzhLHjZjMKFrMsC_1Ktd4nzYeZiCS6JIHsl.Deg0TUjs0UNtXtNzJhnLhhQNsztRlW Oa3QxQkg6ZTkMB1orxaLs7pVlrT9FJ5hgcyx.zaSdUjciUxP9NC3zsYQnTiVtLZCJsjVAw7KJLqm MAQtR0w8kmKd2SpPxy.LNO0Jv9dKk7pGAj2SLSxvOAytGaL3LmAVM.wsCJAFCU2mZpqJLijBRO97 mzqLBDVbwJ8RGG6rhwMa0PHtynayFzPS26mwXaOlM3aJFEMZ2V9si80M36vnJAn4zcaDiBvUm7T5 T6fDYD6qJx_ChhfKr8cwISGhIbvxVPmHA_6FgsFUMkvTORB_m4xvoxVq1GUdt3ltPE_A2F.nnKCP RgZFTqLiUzjxUbCUayXsxuSFzDf5dpC_N4PFGzCWS4zaAMV7vR_6Q8Ps67rymqioXFNWiXDLCp07 8EK.NpEuiEp1Yk9VO18BkSv6VIOVEQNPy.1NHfBT5w4RSiCwYxtIONZt0YW1kT2YQLjk_kd2QXPg K1YTsDGLEusZlV86qa5CLCIkexxm_xjhrHM.eHteoAs8103GJpcRNufdpvu.cq74sLMYQvTKDrDf cZ5cfYpDXHDEmatTIhvNA0mDytegqSSkxkT5OpnadQCfMV_fHWHtfweaQtTnmw_hXRR4llx3RlK_ j3_u.ilTJB.ZXSVo07ibRXtN5IKT5iS7_ErkyLEtP5UdgwGE9X3o2NKt11xqFGKAUf7KJiO9f8UX bpxdCCYQtkzO7PrQOO_DfTBItz.gCcN3dSg9chJ4xGzUpaXw_SxhIwUsHPLhOGD2VD_0ZHkSWVCO RslDRRlAV117JbjVPWp95jSa2Tuy97H4e5qc05LA2DpF1hldh1faRCENq8_QQHcnuoUpyHU9QA4O 93Qw79ds3LBTX.J8uyOv5x_k4yP6LwQOW5EKPDRenVEWs883dbZo9ilD06ZiAoh6F_fUwmu.wLBB mA3ZAfqv3vlRpl9z__wmrMqckLcPFD9qwwI9vjqayogxq8XIpGGP09mm7uDdxllMnGdHHUnrHIrB 4xoFbotjilm92Y.BXjZ5X3Ww.uBVv7Tf56r1idXDdKIqowNeT1I6Gfy_4Q2FqfMclkdnGBFNjQWD s0xkB.Z3EGsAimIHYlf3WI.A1sJAFokHVPonl55xNc_EyNqskOLBX860Jf19hMn5bkF5RzSe96VS JGb1fQnELKY55DCam0bmOy0pAmvezlhl9gX4USzAn9Cd_J732HBkJ.rldEdBSJoZPDdRoHfrl4Ao g0..vyZnpx51FsfI7J34cCZB2cUc9mjGedcOl0YCJ7sH0PQ_q2532V3xOp4WiSEA75yXWXiWpbvo - X-Sonic-MF: X-Sonic-ID: 3ffa6bee-4aec-4333-913b-1d9441ddc105 Received: from sonic.gate.mail.ne1.yahoo.com by sonic313.consmr.mail.gq1.yahoo.com with HTTP; Thu, 6 Apr 2023 20:53:20 +0000 Received: by hermes--production-ne1-7dbd98dd99-84p8v (Yahoo Inc. Hermes SMTP Server) with ESMTPA ID a90a6844e893d7549de2d7cc12038123; Thu, 06 Apr 2023 20:53:16 +0000 (UTC) Content-Type: text/plain; charset=us-ascii List-Id: Porting FreeBSD to ARM processors List-Archive: https://lists.freebsd.org/archives/freebsd-arm List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-arm@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.400.51.1.1\)) Subject: Re: Expected native build times on an RPI4? From: Mark Millard In-Reply-To: <2E1AFF79-9016-4331-8A81-67160DC1F299@yahoo.com> Date: Thu, 6 Apr 2023 13:53:04 -0700 Cc: freebsd-arm@freebsd.org Content-Transfer-Encoding: quoted-printable Message-Id: References: <2E1AFF79-9016-4331-8A81-67160DC1F299@yahoo.com> To: Joseph Koshy X-Mailer: Apple Mail (2.3731.400.51.1.1) X-Spamd-Result: default: False [-2.49 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; SUBJECT_ENDS_QUESTION(1.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.99)[-0.995]; MV_CASE(0.50)[]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; MIME_GOOD(-0.10)[text/plain]; ARC_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_VIA_SMTP_AUTH(0.00)[]; FROM_HAS_DN(0.00)[]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; RCVD_IN_DNSWL_NONE(0.00)[98.137.65.83:from]; RCVD_COUNT_THREE(0.00)[3]; FREEMAIL_FROM(0.00)[yahoo.com]; TO_DN_SOME(0.00)[]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; DKIM_TRACE(0.00)[yahoo.com:+]; FROM_EQ_ENVFROM(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; RCVD_TLS_LAST(0.00)[]; MLMMJ_DEST(0.00)[freebsd-arm@freebsd.org] X-Rspamd-Queue-Id: 4Pstxt4bwpz3qth X-Spamd-Bar: -- X-ThisMailContainsUnwantedMimeParts: N On Apr 5, 2023, at 19:04, Mark Millard wrote: > On Apr 3, 2023, at 13:42, Joseph Koshy wrote: >=20 >> A 'make -j3 buildworld' of a freshly checked out -current tree >> took over 15+ hours on an RPI4 before eventually running out >> of space (/usr/obj had reached 7G by then). >>=20 >> The CPU(s) ran at a top speed of 1500Mhz during the build, >> per 'sysctl dev.cpu.0.freq'. >>=20 >> Even so, the build hadn't managed to cross the 'building >> libraries' step. >>=20 >> I'm wondering how best to provision for building -current: >> how long does 'buildworld' take on this device usually, and >> how much disk space does a build of -current usually need? >=20 > I looked and I'd not recorded any buildwork buildkernel > timings notes since back in very late 2021. So what I > had need not be appropriate for now. I've finally got > around to starting a from scratch build, on a 8 GiByte > RAM "C0T" RPi4B. (My normal build are done on a different > type of aarch64 system.) This is a from-scratch build, > but of note are: >=20 > make[1]: "/usr/main-src/Makefile.inc1" line 327: SYSTEM_COMPILER: = Determined that CC=3Dcc matches the source tree. Not bootstrapping a = cross-compiler. > make[1]: "/usr/main-src/Makefile.inc1" line 332: SYSTEM_LINKER: = Determined that LD=3Dld matches the source tree. Not bootstrapping a = cross-linker. >=20 > Sometimes bootstrapping build activity is required and > that would mean more time (and space) than for what I'm > timing. >=20 > (I've no clue if the build attempt that you mentioned > involved building a bootstrap compiler or bootstrap > linker or both.) >=20 > [Timings added after much of the other text had been > typed in already.] >=20 > World build completed on Wed Apr 5 17:52:47 PDT 2023 > World built in 26009 seconds, ncpu: 4, make -j4 >=20 > So, for world, 26009sec*(1min/60sec)*(1hr/60min) =3D=3D 7.2247_2222... = hr < 7.3 hr. >=20 > Kernel build for GENERIC-NODBG-CA72 completed on Wed Apr 5 18:27:29 = PDT 2023 > Kernel(s) GENERIC-NODBG-CA72 built in 2082 seconds, ncpu: 4, make -j4 >=20 > So, for kernel, 2082sec*(1min/60sec)*(1hr/60min) =3D=3D 0.578_3333... = hr < 0.6 hr. >=20 > So, for total, somewhat under 8 hr. >=20 > (An example of needing bootstrapping would happen for > jumping from main being 14.0 to being 15.0 . Another > could be jumping from system clang 15 to system clang > 16 . The additional time would not be trivial.) >=20 >=20 > Notes . . . >=20 > The RPi4B has heatsinks and a case with a fan. The > config.txt has the following added, among other > things: >=20 > [pi4] > over_voltage=3D6 > arm_freq=3D2000 > sdram_freq_min=3D3200 > force_turbo=3D1 >=20 > (I do not use FreeBSD facilities to manage arm_freq .) >=20 > The result has no temperature problems during such > builds. I picked arm_freq=3D2000 based on it working > across 7 example RPi4B's (mix of 8 GiByte "B0T" > and "C0T" and older 4 GiByte "B0T" variants). 2100 did > not prove to always work, given the other 3 settings. > I avoided system-specific tailoring in normal > operation and so standardized what they all use. >=20 > The media is a USB3 NVMe drive, not spinning rust, > nor a microsd card. The drive is powered from > just the RPi4B. The media has a UFS file system. > I avoid tmpfs use that competes for RAM. (I've also > got access to ZFS media around but that is not what > I'm testing with in this example.) >=20 > The power supply used for the RPi4B has more margin > than is typical: 5.1V, 3.5A. >=20 > A serial console is set up. >=20 > For a 8 GiByte RAM system I normally have 30 GiBytes > or so of swap space active (not special to buildworld > buildkernel activities but I'll not get into details > of why-so-much here). However, for this timing I'm > running without swap since I've not tested that on a > 8GiBYte RPi4B in a long time. (Most of the potential > swap usage is tied to how I build ports into > packages, not buildworld buildkernel .) >=20 > The FreeBSD context is (output line split for > better readability): >=20 > # uname -apKU > FreeBSD CA72_UFS 14.0-CURRENT FreeBSD 14.0-CURRENT #90 > main-n261544-cee09bda03c8-dirty: Wed Mar 15 20:25:49 PDT 2023 > = root@CA72_16Gp_ZFS:/usr/obj/BUILDs/main-CA72-nodbg-clang/usr/main-src/arm6= 4.aarch64/sys/GENERIC-NODBG-CA72 > arm64 aarch64 1400082 1400082 >=20 > The build is building that same version from scratch > (after a "rm -fr" of the build-tree area). I do not > use ccache or the like. So: an example of a possible > upper bound on the required build time for a specific > configuration that is built, but no bootstrap compiler > or linker build involved. >=20 > I do this because comparing timings of incremental builds > that need not be doing the same increments is problematical. > (However configuring for allowing incremental instead of > only full builds is important to spending less total time > building across builds.) >=20 > I'll list various settings that I use. There are non- > obvious contributions too. For example I use a EtherNet ssh > session instead of the serial console: The serial console > can lead to waiting for fast scrolling output to finish. > (Matters more consistently for installworld and > installkernel scrolling output.) I run headless, avoiding > some competition for RAM and such. I do not load the > RPi4B with additional activities not even nice'd ones. >=20 > Note that it is a non-debug system that is running and > it is building a matching non-debug world and kernel. >=20 > In /boot/loader.conf I have: >=20 > # Delay when persistent low free RAM leads to > # Out Of Memory killing of processes: > vm.pageout_oom_seq=3D120 > # > # For plunty of swap/paging space (will not > # run out), avoid pageout delays leading to > # Out Of Memory killing of processes: > vm.pfault_oom_attempts=3D-1 > # > # For possibly insufficient swap/paging space > # (might run out), increase the pageout delay > # that leads to Out Of Memory killing of > # processes (showing defaults at the time): > #vm.pfault_oom_attempts=3D 3 > #vm.pfault_oom_wait=3D 10 > # (The multiplication is the total but there > # are other potential tradoffs in the factors > # multiplied, even for nearly the same total.) >=20 > (I'd not expected the 8 GiByte build to need > to page out to swap space so I left in place > my normal setting for vm.pfault_oom_attempts .) >=20 > In /etc/sysctl.conf I have: >=20 > # Together this pair avoids swapping out the process kernel stacks. > # This avoids processes for interacting with the system from being > # hung-up by such. > vm.swap_enabled=3D0 > vm.swap_idle_enabled=3D0 >=20 > (But, absent any active swap space, such would not > happen. However, the lack of active swap space is > not my normal context and the above is what I have > in place for normal use as well.) >=20 > Part of the below indicates that I avoid building > MIPS, POWERPC, RISCV, and X86 targeting materials > because I do not intend to target anything but > aarch64 and armv7 from aarch64 systems. This is not > the default. Going in the other direction, I build > CLANG_EXTRAS that builds more than what is default. > This combination makes my build timings ball-park > figures relative to your context. >=20 > An oddity is that I avoid much of the stripping so > my builds are somewhat bigger than normal for the > materials produced. (I like the somewhat better > backtraces from leaving symbols in place, even if > the build is optimized and avoids full debug > information.) >=20 > I use: >=20 > TO_TYPE=3Daarch64 > # > KERNCONF=3DGENERIC-NODBG-CA72 > TARGET=3Darm64 > .if ${.MAKE.LEVEL} =3D=3D 0 > TARGET_ARCH=3D${TO_TYPE} > .export TARGET_ARCH > .endif > # > WITH_SYSTEM_COMPILER=3D > WITH_SYSTEM_LINKER=3D > # > WITH_ELFTOOLCHAIN_BOOTSTRAP=3D > #Disables avoiding bootstrap: WITHOUT_LLVM_TARGET_ALL=3D > WITH_LLVM_TARGET_AARCH64=3D > WITH_LLVM_TARGET_ARM=3D > WITHOUT_LLVM_TARGET_MIPS=3D > WITHOUT_LLVM_TARGET_POWERPC=3D > WITHOUT_LLVM_TARGET_RISCV=3D > WITHOUT_LLVM_TARGET_X86=3D > WITH_CLANG=3D > WITH_CLANG_IS_CC=3D > WITH_CLANG_FULL=3D > WITH_CLANG_EXTRAS=3D > WITH_LLD=3D > WITH_LLD_IS_LD=3D > WITH_LLDB=3D > # > WITH_BOOT=3D > # > # > WITHOUT_WERROR=3D > #WERROR=3D > MALLOC_PRODUCTION=3D > WITH_MALLOC_PRODUCTION=3D > WITHOUT_ASSERT_DEBUG=3D > WITHOUT_LLVM_ASSERTIONS=3D > # > # Avoid stripping but do not control host -g status as well: > DEBUG_FLAGS+=3D > # > WITH_REPRODUCIBLE_BUILD=3D > WITH_DEBUG_FILES=3D > # > # Use of the .clang 's here avoids > # interfering with other CFLAGS > # usage, such as ?=3D usage. > CFLAGS.clang+=3D -mcpu=3Dcortex-a72 > CXXFLAGS.clang+=3D -mcpu=3Dcortex-a72 > CPPFLAGS.clang+=3D -mcpu=3Dcortex-a72 > ACFLAGS.arm64cpuid.S+=3D -mcpu=3Dcortex-a72+crypto > ACFLAGS.aesv8-armx.S+=3D -mcpu=3Dcortex-a72+crypto > ACFLAGS.ghashv8-armx.S+=3D -mcpu=3Dcortex-a72+crypto >=20 > Those last 6 lines lead to the code generation being > tuned for Cortex-A72's. (The code still works on > Cortex-A53's.) I expect such lines are rarely used > but I happen to. >=20 > I'll note that avoiding WITHOUT_LLVM_TARGET_ALL is > tied to old observed behavior that I've not > revalidated. >=20 > In the past, I've had examples where RPi4B -j3 built > in less time than -j4 for such full-build timing tests. > On a RPi4B, I've never had -j5 or higher build in less > time. (Some of this is the RPi4B RAM/RAM-cache > subsystem properties: easier than normal to > saturate the RAM access and the caching is small. > Another contribution may be the USB3 NVMe media > latency being small. Spinning rust might have > different tradeoffs, for example.) I've also never > had -j2 or less take less time for full builds. >=20 > (Folks that do not use vm.pageout_oom_seq to avoid > kills from happening may use -j2 or such to better > avoid having parts of some build attempts killed > sometimes.) >=20 > Unfortunately, I forgot to set up monitoring of > MaxObsActive, MaxObsWired, and MaxObs(Act+Wir+Lndry). > ("MaxObs" is short for "Maximum Observed".) So I > do not have such figures to report. (I use a > modified top to get such figures.) >=20 > The build-tree size: >=20 > # du -xsm /usr/obj/BUILDs/main-CA72-nodbg-clang/usr/ > 13122 /usr/obj/BUILDs/main-CA72-nodbg-clang/usr/ >=20 > But such is based on details of what I build vs. > what I do not, as well as lack of stipping. So, in > very round numbers, 20 GiBytes would be able to hold > a build. You might want notable margin, in part because > as FreeBSD and the toolchain progress, things have > tended to get bigger over time. Plus the figure is a > final size. If the peak size is larger, I do not know. > A debug build would take more space than my non-debug > build. Also, the 13122 does not include the build > materials for a bootstrap compiler or a bootstrap > linker (or both). Thus my rounding to 20 GiBytes as a > possibility for illustration. >=20 > Again: no ccache-like use. Otherwise there would be > more space someplace to consider overall. >=20 I repeated the "rm -fr", rebooted, and did a -j3 buildworld buildkernel . The result was: World build completed on Thu Apr 6 03:31:43 PDT 2023 World built in 28858 seconds, ncpu: 4, make -j3 So, for world, 28858sec*(1min/60sec)*(1hr/60min) =3D=3D 8.016_1111... hr = < 8.1 hr. Kernel build for GENERIC-NODBG-CA72 completed on Thu Apr 6 04:10:26 PDT = 2023 Kernel(s) GENERIC-NODBG-CA72 built in 2323 seconds, ncpu: 4, make -j3 So, for kernel, 2323sec*(1min/60sec)*(1hr/60min) =3D=3D 0.6452_7777... = hr < 0.7 hr. So, for total, somewhat under 8.8 hr. So 31181sec/28091sec ~=3D 1.11 times what than -j4 took. I did remember to get MaxObs figures for this: load averages: . . . MaxObs: 3.59, 3.21, 3.09 1404Mi MaxObsActive, 1155Mi MaxObsWired, 2383Mi MaxObs(Act+Wir+Lndry) (Note: Laundry did end up non-zero, despite the lack of swap space.) So this combination looks like it would not need swap space for a 4 GiByte RPi4B but likely would need such for a 2 GiByte RPI4B. Looks like the same could be true of a -j4 build. After that I repeated the "rm -fr", rebooted, and did a -j5 buildworld buildkernel . The result was: World build completed on Thu Apr 6 12:42:04 PDT 2023 World built in 25940 seconds, ncpu: 4, make -j5 So, for world, 25940sec*(1min/60sec)*(1hr/60min) =3D=3D 7.20_5555... hr = < 7.3 hr. Kernel build for GENERIC-NODBG-CA72 completed on Thu Apr 6 13:16:50 PDT = 2023 Kernel(s) GENERIC-NODBG-CA72 built in 2086 seconds, ncpu: 4, make -j5 So, for kernel, 2086sec*(1min/60sec)*(1hr/60min) =3D=3D 0.579_4444... hr = < 0.6 hr. So, for total, somewhat under 8 hr. So around 28026sec/28091sec ~=3D 0.998 times what -j4 took. Note a small scale example of a tradeoff that can occur based on the details of what is being built: buildworld took less time but buildkernel took more. I did remember to get MaxObs figures for this: load averages: . . . MaxObs: 5.57, 5.29, 5.17 1790Mi MaxObsActive, 1157Mi MaxObsWired, 2775Mi MaxObs(Act+Wir+Lndry) (Note: Laundry did end up non-zero, despite the lack of swap space.) So this combination looks like it would not need swap space for a 4 GiByte RPi4B but would need such for a 2 GiByte RPi4B. Incremental builds (META_MODE) variability, ccache avoidance of compiles variability, and media access timing properties could all lead to other tradeoffs in specific builds for what -jN's work better. ZFS would be a significant change of context because of Wired memory handling. (The ARC leads to a far more widely variable Wired-memory usage pattern, for example.) I'm not claiming the above indicates some universal answer to what is optimal across a range of contexts. One thing that was different for a time for my older timings was that some Google test build used to take large amounts of RAM and time compared to the figures I report above. If I remember right, this stopped when FreeBSD adjusted the specific test's build to generate unoptimized code, avoiding the bad-case in the LLVM toolchain's optimization handling for generating the test involved. Note: The ZFS ARC's Wired memory usage makes any "MaxObs" that includes a Wired memory contribution not readily comparable to the same "MaxObs" for a UFS context. =3D=3D=3D Mark Millard marklmi at yahoo.com