From nobody Thu Apr 06 02:04:40 2023 X-Original-To: freebsd-arm@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PsPvq46h1z43FLB for ; Thu, 6 Apr 2023 02:04:55 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic316-55.consmr.mail.gq1.yahoo.com (sonic316-55.consmr.mail.gq1.yahoo.com [98.137.69.31]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4PsPvn4RPGz3K5W for ; Thu, 6 Apr 2023 02:04:53 +0000 (UTC) (envelope-from marklmi@yahoo.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yahoo.com header.s=s2048 header.b=IBpCm1Bv; spf=pass (mx1.freebsd.org: domain of marklmi@yahoo.com designates 98.137.69.31 as permitted sender) smtp.mailfrom=marklmi@yahoo.com; dmarc=pass (policy=reject) header.from=yahoo.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1680746691; bh=kO7d7XKz5bB5XJZYR3Qdef4ap0S2/xsajUATQ3VcbRo=; h=Subject:From:In-Reply-To:Date:Cc:References:To:From:Subject:Reply-To; b=IBpCm1BvqTdkYk5/2o3R53d0zemUyT6+wyrCQgyqWL19hB0v8zDTwkNkeW4tUno4RE2h/j9RwqZo/D3fUuvkHkdQ5nV5+EAT1wRKrx967S9Bp7GbinceEdQ6h6IE/+MzdJ0xyrXM7tAfsbR5wq3yV0/y+AgQ+M5rPAI7NrfVtpcwxSlb/OH9nxg6hN+1t52YX3SWFXtXTpOSBzEl5azfcCIrAiT+OK4DtyeA/W3Ry0PEjGoNAXhak1JAOwKgCrw8JG5Q2bYuPFUROUjwD1mhEeD1eGLAGlnKmY8xphVsdcXb+crNCE2db8mTlwWKxfX3gzy9USNlEdWxgdeuZ2VRAw== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1680746691; bh=FzBIPVp+BbnPlu6ibpxpEjDpVpwDB5TbF3d+0sGH7uD=; h=X-Sonic-MF:Subject:From:Date:To:From:Subject; b=BxpRZwqOVdLORcc3KrrlUFWgQDUnKf6bhflXuUCKHbxGURgzBoWRVShCgiQbPfyZDQdbNOEkeMut1NEYhmP4GZ22slW5usjv2zvQ3C1eYoT1J/wV/iHQeS6tpv92aRS0dUhTZ8IlO/7cJRp8b+T2v+jVkZv8LPoEepeNvF21Pen1/Ja/3TZuXo82SnbeItjp6lA4iGy6L0kVNwTvcQJhrZFXIEm91GEddVZ1UO9HJQ0dVkuk1CABkmotRvfGelIpG2iIIeHFk5NPWmQ65iK7xBHuRKBM4L36eSnGjI43XpVQcA2zvbMAsX8yWHIIlejKt4wFFwT3aXnKnYqL3HxRyg== X-YMail-OSG: bYhVWf4VM1mSha0ZjyYW4lyWq05oFHdeVGEbQJbICMxVYk5sl4ncYERv7LdU1Dm 1nI.A8ba0hLk9df9lrIGhfa2WRWRcpL1i_.ovtYlsS6OZVxbZbyuSocAy560AxoEvn9OiDDYI8L9 gtsGMirQilNFwhq02b.ENZsl5hqAZAf07oEMmQVkm8r7kHEgGYMDXZwVXA16YWE3s9DvLot5PQu. mX4cg9ghqfIKwCFZ6VlOJRlMO76B4Yunw3l_m66ZQXeAz60IcwxaKYIV_klmF8m9BM347bia7W5C 7T_qhZqx1NzBfvosa3JexVwLhJSiLHkqXtJaZJbgS4XZ.npnXlOwLnJlXjefmi2HE.Kcz94QqH1F IvNhRN1J.NJaHAdWfzsflZcxJXQXFbYmojFmXjGPbiCngrIFEiej_uDnbanQH3.r99TEIVJR4o.C TLRPg7BgK.7gFHXOFlS1tPQO1QF0ACZH68GPbIj1dBjAUyn2rrmZzsp47zuWTPFNm47EzSymTecK TmLLa3XVVGbW86HdsLI..lAMu.rHheMNbEydYVxAsntBGHysBO5InzGST1TCceRRYi1sfZdiFZ.4 qEP6.q_QmtRwQK9CFGpPboaldns9iO87uBJd9npNWIQogGiY6K9urTvrxJ9WJyNJVFPD6DAABliu zUgTAmDKnAGXKela2wC9VUeYyi10Seay8XKi_69Urg7x53e2YFO3e9_84._KBU2xw69x8AaXFvCM S8HrqQNHdQcxf65YpxLN7oj9s2EKh6CmCICic8ltKSKl8H4knv48DrhxQpM8jNnXuTvdBDet45KT BWTSA2rre0F4OdY0BSaG1Vq0zypIJildc2TQWUAwbduP8uMgk48HGJKvCsH2KNJoR.c3cDDgIjuI ZK6dD23gcJ9YfUzF8GWGAsI2jL3cs1SxOvYuDUL5ssIckCr7HYIO2RrLVT67QCwbjEzfHvGrvxNH UXmH45MfnIqbgfo0_Cj8Yt4ALclgWZgTjLjSFb3SJ4Mgt.kCwqA6Aj4WmbQT4k4cad8cqs6qzXo0 mKeKWY0kwBmKc82iSG3EC6Fu2.C7nz.Xu1fESPhnK_IxaL1qxWoJWnXkcgf_aQYxV.NwN4iAMd.L CQ_e.oMGyUQDI700PSIUjixkLLZ.NiXqLgwJGsnFabVyyH4Di7ocS9EXZD6f.NpLbCb8ICcZsEJG 6jF7SWqIgZDm.nSAHI3ZZdOs9q1fMeatALv.hgO0PhsZXSdpLZJAUMOc_mczng0Vn030cyIqiCgm MLpP_CPK5Zeu16MGF59iHR.4FUU3x27pVddHu_V3ozeaYHzfXM3Gozu5.2p_bli5W8LzuJ4PZW4c V00X6lhnzNkPTRSQe4JE1ib9jHw0vX_2DTx6oHtoBT6u0oylTlV0SmvUO4D9KIC4odT5RtqKz7TB f0Mh4lUQUbOvQ.GgnwUOflLrhm2VXFnvVu8.h9WFUrqXMArFeoz9xjSbJrXG8zFiK8j5aa16QHC. pV5vS0SJVbsaz9HXG1oBqzFTRhk36IkR3ZmFc2JuqwmqJkUak7oFLIzWk59c6O2H0YqOd8FvbMfg CxGq38gVqzFL4Kt4AigkDBxx7Sgt.4R94.D.asTlRftFYZCxuTI.WLjM0qtxUGp58HUenmhdnedM gDmmj.jtv7HGDh6OtLn2e3L5VwIzRdVBkq_vnYEIfstAkhvpGyCkGJX9SFaJxp3e4r_Rv2QBh0m2 SbuYBTfIWMtGs1aKEmFAFsIkmOvgH.VzZ62541iNYDnp26LQsReYSwBCIssF5tWIBFqmjl3IifSB wQcz8n2KvbnolOaYNio3DEkgBkBP_fWL0Dma9ARPwL1sNTdVgZCXppCQKaPStNGgqE9fpWwOvf7V p2KQtZg2FbGDxq5wqtY2QCXVfovYRKZ0G6z8ge9lkujyoQV9nnL3BdDp1RmvJynHbBTPgvT1bBrv C5AyWK145evtoSfhJ2NTHhpBfVDRLpv8ErLlYTn1wlJi342FeP.JDY8057w.BeJIcjs_dBaLksse 0RTAQyXg0.nK1FG4c8rzpCVRo1pOI_qKOM2_eLqG31kzOQyOlLlQpOAn66TSgQunD6aXGiAtHcGg pdaiFK9awt8F6flFGJP.WjvbfY9HYhk5u8HkU_o9FdI4UIU93PYy_V.IiguPJr2JqkiGKrVlgRDX Ifl54Wxe7NGjYWRL.uuvjIlsIaxRsccR3_U6FuawwmzrMT7Reh.E8H.6eNG7HxVWKqJe42oSFUzW fcrNoEnrqdWkjgS2giYUA.2xkE1kppXx_3wLpAW3CRzWDjP7ONW_ttPrUEog3QZnGmPxq5Js- X-Sonic-MF: X-Sonic-ID: b18fa305-d203-4ab8-8692-78e62cbef6e4 Received: from sonic.gate.mail.ne1.yahoo.com by sonic316.consmr.mail.gq1.yahoo.com with HTTP; Thu, 6 Apr 2023 02:04:51 +0000 Received: by hermes--production-gq1-546798879c-sq6s2 (Yahoo Inc. Hermes SMTP Server) with ESMTPA ID 3dd7cc0d57d4e368cfec1056bf1d47fa; Thu, 06 Apr 2023 02:04:51 +0000 (UTC) Content-Type: text/plain; charset=us-ascii List-Id: Porting FreeBSD to ARM processors List-Archive: https://lists.freebsd.org/archives/freebsd-arm List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-arm@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.400.51.1.1\)) Subject: Re: Expected native build times on an RPI4? From: Mark Millard In-Reply-To: Date: Wed, 5 Apr 2023 19:04:40 -0700 Cc: freebsd-arm@freebsd.org Content-Transfer-Encoding: quoted-printable Message-Id: <2E1AFF79-9016-4331-8A81-67160DC1F299@yahoo.com> References: To: Joseph Koshy X-Mailer: Apple Mail (2.3731.400.51.1.1) X-Spamd-Result: default: False [-1.56 / 15.00]; SUBJECT_ENDS_QUESTION(1.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-0.86)[-0.865]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; MV_CASE(0.50)[]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; NEURAL_HAM_SHORT(-0.19)[-0.191]; MIME_GOOD(-0.10)[text/plain]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_IN_DNSWL_NONE(0.00)[98.137.69.31:from]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; RCVD_COUNT_THREE(0.00)[3]; FREEMAIL_FROM(0.00)[yahoo.com]; TO_DN_SOME(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; DKIM_TRACE(0.00)[yahoo.com:+]; FROM_EQ_ENVFROM(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; MLMMJ_DEST(0.00)[freebsd-arm@freebsd.org]; RCVD_TLS_LAST(0.00)[]; RWL_MAILSPIKE_POSSIBLE(0.00)[98.137.69.31:from] X-Rspamd-Queue-Id: 4PsPvn4RPGz3K5W X-Spamd-Bar: - X-ThisMailContainsUnwantedMimeParts: N On Apr 3, 2023, at 13:42, Joseph Koshy wrote: > A 'make -j3 buildworld' of a freshly checked out -current tree > took over 15+ hours on an RPI4 before eventually running out > of space (/usr/obj had reached 7G by then). >=20 > The CPU(s) ran at a top speed of 1500Mhz during the build, > per 'sysctl dev.cpu.0.freq'. >=20 > Even so, the build hadn't managed to cross the 'building > libraries' step. >=20 > I'm wondering how best to provision for building -current: > how long does 'buildworld' take on this device usually, and > how much disk space does a build of -current usually need? I looked and I'd not recorded any buildwork buildkernel timings notes since back in very late 2021. So what I had need not be appropriate for now. I've finally got around to starting a from scratch build, on a 8 GiByte RAM "C0T" RPi4B. (My normal build are done on a different type of aarch64 system.) This is a from-scratch build, but of note are: make[1]: "/usr/main-src/Makefile.inc1" line 327: SYSTEM_COMPILER: = Determined that CC=3Dcc matches the source tree. Not bootstrapping a = cross-compiler. make[1]: "/usr/main-src/Makefile.inc1" line 332: SYSTEM_LINKER: = Determined that LD=3Dld matches the source tree. Not bootstrapping a = cross-linker. Sometimes bootstrapping build activity is required and that would mean more time (and space) than for what I'm timing. (I've no clue if the build attempt that you mentioned involved building a bootstrap compiler or bootstrap linker or both.) [Timings added after much of the other text had been typed in already.] World build completed on Wed Apr 5 17:52:47 PDT 2023 World built in 26009 seconds, ncpu: 4, make -j4 So, for world, 26009sec*(1min/60sec)*(1hr/60min) =3D=3D 7.2247_2222... = hr < 7.3 hr. Kernel build for GENERIC-NODBG-CA72 completed on Wed Apr 5 18:27:29 PDT = 2023 Kernel(s) GENERIC-NODBG-CA72 built in 2082 seconds, ncpu: 4, make -j4 So, for kernel, 2082sec*(1min/60sec)*(1hr/60min) =3D=3D 0.578_3333... hr = < 0.6 hr. So, for total, somewhat under 8 hr. (An example of needing bootstrapping would happen for jumping from main being 14.0 to being 15.0 . Another could be jumping from system clang 15 to system clang 16 . The additional time would not be trivial.) Notes . . . The RPi4B has heatsinks and a case with a fan. The config.txt has the following added, among other things: [pi4] over_voltage=3D6 arm_freq=3D2000 sdram_freq_min=3D3200 force_turbo=3D1 (I do not use FreeBSD facilities to manage arm_freq .) The result has no temperature problems during such builds. I picked arm_freq=3D2000 based on it working across 7 example RPi4B's (mix of 8 GiByte "B0T" and "C0T" and older 4 GiByte "B0T" variants). 2100 did not prove to always work, given the other 3 settings. I avoided system-specific tailoring in normal operation and so standardized what they all use. The media is a USB3 NVMe drive, not spinning rust, nor a microsd card. The drive is powered from just the RPi4B. The media has a UFS file system. I avoid tmpfs use that competes for RAM. (I've also got access to ZFS media around but that is not what I'm testing with in this example.) The power supply used for the RPi4B has more margin than is typical: 5.1V, 3.5A. A serial console is set up. For a 8 GiByte RAM system I normally have 30 GiBytes or so of swap space active (not special to buildworld buildkernel activities but I'll not get into details of why-so-much here). However, for this timing I'm running without swap since I've not tested that on a 8GiBYte RPi4B in a long time. (Most of the potential swap usage is tied to how I build ports into packages, not buildworld buildkernel .) The FreeBSD context is (output line split for better readability): # uname -apKU FreeBSD CA72_UFS 14.0-CURRENT FreeBSD 14.0-CURRENT #90 main-n261544-cee09bda03c8-dirty: Wed Mar 15 20:25:49 PDT 2023 = root@CA72_16Gp_ZFS:/usr/obj/BUILDs/main-CA72-nodbg-clang/usr/main-src/arm6= 4.aarch64/sys/GENERIC-NODBG-CA72 arm64 aarch64 1400082 1400082 The build is building that same version from scratch (after a "rm -fr" of the build-tree area). I do not use ccache or the like. So: an example of a possible upper bound on the required build time for a specific configuration that is built, but no bootstrap compiler or linker build involved. I do this because comparing timings of incremental builds that need not be doing the same increments is problematical. (However configuring for allowing incremental instead of only full builds is important to spending less total time building across builds.) I'll list various settings that I use. There are non- obvious contributions too. For example I use a EtherNet ssh session instead of the serial console: The serial console can lead to waiting for fast scrolling output to finish. (Matters more consistently for installworld and installkernel scrolling output.) I run headless, avoiding some competition for RAM and such. I do not load the RPi4B with additional activities not even nice'd ones. Note that it is a non-debug system that is running and it is building a matching non-debug world and kernel. In /boot/loader.conf I have: # Delay when persistent low free RAM leads to # Out Of Memory killing of processes: vm.pageout_oom_seq=3D120 # # For plunty of swap/paging space (will not # run out), avoid pageout delays leading to # Out Of Memory killing of processes: vm.pfault_oom_attempts=3D-1 # # For possibly insufficient swap/paging space # (might run out), increase the pageout delay # that leads to Out Of Memory killing of # processes (showing defaults at the time): #vm.pfault_oom_attempts=3D 3 #vm.pfault_oom_wait=3D 10 # (The multiplication is the total but there # are other potential tradoffs in the factors # multiplied, even for nearly the same total.) (I'd not expected the 8 GiByte build to need to page out to swap space so I left in place my normal setting for vm.pfault_oom_attempts .) In /etc/sysctl.conf I have: # Together this pair avoids swapping out the process kernel stacks. # This avoids processes for interacting with the system from being # hung-up by such. vm.swap_enabled=3D0 vm.swap_idle_enabled=3D0 (But, absent any active swap space, such would not happen. However, the lack of active swap space is not my normal context and the above is what I have in place for normal use as well.) Part of the below indicates that I avoid building MIPS, POWERPC, RISCV, and X86 targeting materials because I do not intend to target anything but aarch64 and armv7 from aarch64 systems. This is not the default. Going in the other direction, I build CLANG_EXTRAS that builds more than what is default. This combination makes my build timings ball-park figures relative to your context. An oddity is that I avoid much of the stripping so my builds are somewhat bigger than normal for the materials produced. (I like the somewhat better backtraces from leaving symbols in place, even if the build is optimized and avoids full debug information.) I use: TO_TYPE=3Daarch64 # KERNCONF=3DGENERIC-NODBG-CA72 TARGET=3Darm64 .if ${.MAKE.LEVEL} =3D=3D 0 TARGET_ARCH=3D${TO_TYPE} .export TARGET_ARCH .endif # WITH_SYSTEM_COMPILER=3D WITH_SYSTEM_LINKER=3D # WITH_ELFTOOLCHAIN_BOOTSTRAP=3D #Disables avoiding bootstrap: WITHOUT_LLVM_TARGET_ALL=3D WITH_LLVM_TARGET_AARCH64=3D WITH_LLVM_TARGET_ARM=3D WITHOUT_LLVM_TARGET_MIPS=3D WITHOUT_LLVM_TARGET_POWERPC=3D WITHOUT_LLVM_TARGET_RISCV=3D WITHOUT_LLVM_TARGET_X86=3D WITH_CLANG=3D WITH_CLANG_IS_CC=3D WITH_CLANG_FULL=3D WITH_CLANG_EXTRAS=3D WITH_LLD=3D WITH_LLD_IS_LD=3D WITH_LLDB=3D # WITH_BOOT=3D # # WITHOUT_WERROR=3D #WERROR=3D MALLOC_PRODUCTION=3D WITH_MALLOC_PRODUCTION=3D WITHOUT_ASSERT_DEBUG=3D WITHOUT_LLVM_ASSERTIONS=3D # # Avoid stripping but do not control host -g status as well: DEBUG_FLAGS+=3D # WITH_REPRODUCIBLE_BUILD=3D WITH_DEBUG_FILES=3D # # Use of the .clang 's here avoids # interfering with other CFLAGS # usage, such as ?=3D usage. CFLAGS.clang+=3D -mcpu=3Dcortex-a72 CXXFLAGS.clang+=3D -mcpu=3Dcortex-a72 CPPFLAGS.clang+=3D -mcpu=3Dcortex-a72 ACFLAGS.arm64cpuid.S+=3D -mcpu=3Dcortex-a72+crypto ACFLAGS.aesv8-armx.S+=3D -mcpu=3Dcortex-a72+crypto ACFLAGS.ghashv8-armx.S+=3D -mcpu=3Dcortex-a72+crypto Those last 6 lines lead to the code generation being tuned for Cortex-A72's. (The code still works on Cortex-A53's.) I expect such lines are rarely used but I happen to. I'll note that avoiding WITHOUT_LLVM_TARGET_ALL is tied to old observed behavior that I've not revalidated. In the past, I've had examples where RPi4B -j3 built in less time than -j4 for such full-build timing tests. On a RPi4B, I've never had -j5 or higher build in less time. (Some of this is the RPi4B RAM/RAM-cache subsystem properties: easier than normal to saturate the RAM access and the caching is small. Another contribution may be the USB3 NVMe media latency being small. Spinning rust might have different tradeoffs, for example.) I've also never had -j2 or less take less time for full builds. (Folks that do not use vm.pageout_oom_seq to avoid kills from happening may use -j2 or such to better avoid having parts of some build attempts killed sometimes.) Unfortunately, I forgot to set up monitoring of MaxObsActive, MaxObsWired, and MaxObs(Act+Wir+Lndry). ("MaxObs" is short for "Maximum Observed".) So I do not have such figures to report. (I use a modified top to get such figures.) The build-tree size: # du -xsm /usr/obj/BUILDs/main-CA72-nodbg-clang/usr/ 13122 /usr/obj/BUILDs/main-CA72-nodbg-clang/usr/ But such is based on details of what I build vs. what I do not, as well as lack of stipping. So, in very round numbers, 20 GiBytes would be able to hold a build. You might want notable margin, in part because as FreeBSD and the toolchain progress, things have tended to get bigger over time. Plus the figure is a final size. If the peak size is larger, I do not know. A debug build would take more space than my non-debug build. Also, the 13122 does not include the build materials for a bootstrap compiler or a bootstrap linker (or both). Thus my rounding to 20 GiBytes as a possibility for illustration. Again: no ccache-like use. Otherwise there would be more space someplace to consider overall. =3D=3D=3D Mark Millard marklmi at yahoo.com