From nobody Sun May 21 07:55:53 2023 X-Original-To: freebsd-arm@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4QPCZK2N0jz4CCQr for ; Sun, 21 May 2023 07:56:09 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic303-23.consmr.mail.gq1.yahoo.com (sonic303-23.consmr.mail.gq1.yahoo.com [98.137.64.204]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4QPCZH713wz3s7l for ; Sun, 21 May 2023 07:56:07 +0000 (UTC) (envelope-from marklmi@yahoo.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yahoo.com header.s=s2048 header.b=huPZBbui; spf=pass (mx1.freebsd.org: domain of marklmi@yahoo.com designates 98.137.64.204 as permitted sender) smtp.mailfrom=marklmi@yahoo.com; dmarc=pass (policy=reject) header.from=yahoo.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1684655765; bh=hLx07SNu3WMRGNIspeWadmSe6uNIZndjX4KFD7pTi64=; h=Subject:From:In-Reply-To:Date:Cc:References:To:From:Subject:Reply-To; b=huPZBbuiseEIsRu0gZlUfQG308X5h0xojG5XnFIcZZzBe+lhTDfCjGNIjE+kY5KfTGZ1c9YcYuBGhllIj3EPeZoR2+uebFLKZHhwUrNCD6FS7VfRtXXOnwi0bRCbSrgvAam01tqAURB2VrHt6SL2JW0xvfsIfDaXjXXXgWRuedSjW85DnVzHL7RsV+kY1QV7RjFLi4UxHMlOxED1FcOooeQK2xjP4Kw19zz5rnKX3nInoKXjSFC/awcsq+qEr378lK07ymwu2hsocOosPJQP4OxYJEFZAA8VP8ORngqAPiAqmwRq+NSZeBsu8xFMzD+FdKzthiiEoHQmpH7YKz05wg== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1684655765; bh=NpjpPJ96rj+BxFYPU9jCHwV+IrgL0ZrfzlKmqVkkU1D=; h=X-Sonic-MF:Subject:From:Date:To:From:Subject; b=K+CrgqzkM++gu8buGDivF3etOTER/7WpMb9QQTOJEh4gvmkrdEu8GV+77mepumbIqHsLL1h8qUF7FUgcdxUpyQ8QM2GUrbzBBMFU8zPFVBXemAJpoW7Z0cjGK81+cDiztg5koEm6afTxNpsa6PfiNxf47yERep7PpflGDTwkQYO2QVzlEu1bHriqNoFzW3bGtOhL5wutFvU7jBGwE2GEbU5Uz2N9sCdGbb4JGsl3qztKihtEqYVTRpJTk78FidoZfXQDQC1NQk7/OCQKt5weWTJBgjTvcLXSZkV1VjKS4lIB2MXsObzi/8p4jqUgHNt5gSozH/EanwWPNcwLHj7h+g== X-YMail-OSG: uqgrhOoVM1kkRvi8hJMDNy3XA26DEQkI29VNtJopcRNPbgV28goPNrRG52PUUeq 8yN4HXCzdEJBjeVi59hkRVenIUvEZ7eHe7171n79v7lvnjtW56Q7ps8QEOXlS.wa6L1X7.Eeq3p. nr3El58RaOP6SOknDPmJ1t3baYousHlU_VbCfOyQTeJk._YPMLIaSqATihyGm5jAGVfEm69HNECj vAtPL17Gl81Ic.H5uZFQTS9q9.EbdZ1MKYAk55JiXa4RfAQvcWpHlXLR8x3PMFBhv0SwQSOLTAP8 Of78Kd_8fRDNnVFEaFlsTqcYJwlBpd2ap0K7W9.vgtz648Qub4XEnUCYxaS0BNxvITI2MvTCAuHF CQirzvMopYgv9joy0JSoxL3PcMMDPrmjaK2guGicyZXgRadS4g9ft5LMKZ1oqUf2PwIpiURhlBaO T7O6NVSFC8eJ7OYGOWbmjP1cEa1JDuFers8VJyFjNwLDGhl7DajnHWQ.UqX_bNHz_6i5Pt8vFJv2 JmopoBZ7UtBJHdtjCMMHHHBYFOVa_a2VlPt94BLeP0hoPSKfanwPMCtYb_zFBaaoygsPOcz8hsso HVff98uNx6bJFmUMlh_MTsyCr8RhPQ63cuqQIV_27mu9kIhZe62Omx6DM2yuem_hNMSn514iSPCm 9dywlFXrFlFO.r.9FPOTn.uBmxFc8k3ND4O8PB9JnVB6mcL0ZQL_xpLldmDVb3Ga6vdHSlO9oYDP 69AKx_1oulEd.RuadtLTUpWXLdKUjEQHfQz2RDiWr7CJ_C6pihYgoHUYBMWsbQlpTkVT7fKYzQzh F_WNYbFuLzj_qeigAW2gFbQSg0yEWUCwwziLTImZT1Crn0pwezBMWq8NnunaDviPcAqivpsS_r1B xlVucU6cCP5_5Pb0EOwntH6yjeaBfN69SayHsrAataekvwsx0jK27TLzxfO.z.b4_kOSJU1oPtSW .9mP0UbGM6YW55bKpi8zD4Eyf2ihuySf2I8ccngQOD0nei9u2G71GxuiZhxJT7R6YADOp4nHh11b JnIjH4bG7Px519NhpSl5ITrY3UEpthkCCwET9cycn8qZl50kh2pk_dSw6lwWJ_C9hqVAymxqxoRA N6uA3p73JadnvV4bpwX2wUvMi5WNwTxLPVSa_3PPEiVARSzpUzcCRtzv0UDwX0xJC6c9N9qZCHv8 V2jgDMRw9cp.LvNx9.2GoAZxXnV3woBltTVjPV.Gix9IGnmQiYBHC5TDyZd36BwGq1SqgsH1e_lM Ve2SlVmz3iNDQDoKp3UtRS.BVtfEWKj47zvbvV5fVikzbipooZhu1OQdWA0NEOMKaHEvU8Expj4F vza7E.MMpBBRY2ZZi6z8RZF0otqyTlub6hC3TbqxdlJqy_1Xzt6tfib.loUNxq40ZxtRlgLictCe 4k7Fh.lzbCXUWNR8w4CaF5zDocUQ0dlBPQ_hd_taDRb3jkgExeEjXwlni4yzEBd1KyovKQWXcxtu L1RQoee3hTIPkidCDcmwHat3_XSfG5YU4cDLtxTwJo4KETXez3HMnSBjwfmVYdhAaIagGzN9EabS I8vBgsqVVt.UcjTZ5WJ5XOOuEuWDGsJED0zVMy6TlQodxObBo9hYNCWDZcJnofRuJeXSi7y1XnU0 9v10LqgFAnzoWWk78gN51XQPfL.CBzycCPhpNKanus6dmgOYGddpK9cFQXHUwqbXvCMa0YPXkLd5 OwSnJSFSZjXMsz6DtFSMZdml3_GokHqFrdQS9ofDOnL9dNT6iLEYDlVvM2beqB.WGcwS8ORZJODq DYYiXLiC6WsemKshCqSgQlqFitGMSsDr53kmQ2oIEZRf2GgVjbs4INDFN_H18Th2R6sjtDl8kwQG ARAsnrqi1z4oDADI6T9La6Llk_7dRLBh.L82bT1XZjb5CIEn7SnykD2CKWbDu.oyFMnekKGLR_yP ST0t.7ok01rjuXh_ykluLxd4BCLGqh63.5bUlFWe.3K0uvI7uwytU7KUYPrJ3FaDk.m3qREwzvQY CvAuIYTNoQBgFuZT8pM7YzZmXY8xyWgBl0nr2GamOyLFa1Al_dl_OQC1uQUoRc537Eja050AWXmP p84vLazuImOQiPHqLpiPyinruTo9qFlcx_oTZBn4yY5iqZgovezdF1RNUiyMrFPZyynJYYJJzzFA dkUItUhHbKeoFs3N8yMBMPF.iz.pWkcaxVqEurUeXuGCasj2otcjHt7KXh5OhcXMT_aFsewiL6wj 5eBJDwhBo7wbKi4iHWSBH6eYy3zM07eoja3MqzhkWhaS4IN0Go6XjuuHEY7q5LG.M0KlT72ds3Q- - X-Sonic-MF: X-Sonic-ID: 4cedd67f-3c3b-4fba-b4fc-7fb9295703b5 Received: from sonic.gate.mail.ne1.yahoo.com by sonic303.consmr.mail.gq1.yahoo.com with HTTP; Sun, 21 May 2023 07:56:05 +0000 Received: by hermes--production-gq1-6db989bfb-xtcqt (Yahoo Inc. Hermes SMTP Server) with ESMTPA ID 3fde5a7ab40eb35d51efb77ec595fb7a; Sun, 21 May 2023 07:56:03 +0000 (UTC) Content-Type: text/plain; charset=us-ascii List-Id: Porting FreeBSD to ARM processors List-Archive: https://lists.freebsd.org/archives/freebsd-arm List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-arm@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.400.51.1.1\)) Subject: Re: Armv7 (rpi2) getting stuck in buildworld for -current From: Mark Millard In-Reply-To: Date: Sun, 21 May 2023 00:55:53 -0700 Cc: freebsd-arm Content-Transfer-Encoding: quoted-printable Message-Id: <7AA87E52-AB3D-48E4-B62A-EE73EBB23003@yahoo.com> References: <7758885B-3115-47F0-A453-1C010313D4B8@yahoo.com> <03F330A1-35E4-40D4-B9C6-407041BBEC58@yahoo.com> To: bob prohaska X-Mailer: Apple Mail (2.3731.400.51.1.1) X-Spamd-Result: default: False [-3.47 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.97)[-0.970]; MV_CASE(0.50)[]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; MIME_GOOD(-0.10)[text/plain]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; FROM_HAS_DN(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_TLS_LAST(0.00)[]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; RCVD_IN_DNSWL_NONE(0.00)[98.137.64.204:from]; TO_DN_ALL(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; FREEMAIL_FROM(0.00)[yahoo.com]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; DKIM_TRACE(0.00)[yahoo.com:+]; FROM_EQ_ENVFROM(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; MIME_TRACE(0.00)[0:+]; MLMMJ_DEST(0.00)[freebsd-arm@freebsd.org] X-Rspamd-Queue-Id: 4QPCZH713wz3s7l X-Spamd-Bar: --- X-ThisMailContainsUnwantedMimeParts: N On May 20, 2023, at 11:59, Mark Millard wrote: > I set up the RPi2B v1.1 and started a -j4 buildworld buildkernel > from-scratch rebuild on/of: >=20 > # uname -apKU # long output line split for readability > FreeBSD OPiP2E_RPi2v1p1 14.0-CURRENT FreeBSD 14.0-CURRENT #74 > main-n262658-b347c2284603-dirty: Fri Apr 28 23:07:41 PDT 2023 > = root@CA72_16Gp_ZFS:/usr/obj/BUILDs/main-CA7-nodbg-clang/usr/main-src/arm.a= rmv7/sys/GENERIC-NODBG-CA7 > arm armv7 1400088 1400088 >=20 > (The original build was done on another machine.) >=20 >=20 > At somewhat under 18 hrs it finished with the large swap > use during the overlapping time frame for these 4 builds: >=20 > clang/libclang/CodeGen/CodeGenAction.o > clang/libclang/CodeGen/CodeGenFunction.o > clang/libclang/CodeGen/CodeGenModule.o > clang/libclang/CodeGen/CodeGenPGO.o >=20 > These are my notes from the information for somewhat after > the swap use dropped off. >=20 >=20 >=20 > My armv7 builds disable targeting other architectures but > I also have WITH_CLANG_EXTRAS=3D . (Not that the build has > gotten that far yet.) I show the controlling file content > later below. >=20 > I used no assignments for: >=20 > #vm.pfault_oom_attempts=3D-1 > #vm.pfault_oom_attempts=3D 10 > #vm.pfault_oom_wait=3D ??? >=20 > but did/do have: >=20 > vm.pageout_oom_seq=3D120 >=20 > vm.swap_enabled=3D0 > vm.swap_idle_enabled=3D0 >=20 > in use for this experiment. >=20 > FYI: > make[1]: "/usr/main-src/Makefile.inc1" line 326: SYSTEM_COMPILER: = Determined that CC=3Dcc matches the source tree. Not bootstrapping a = cross-compiler. > make[1]: "/usr/main-src/Makefile.inc1" line 331: SYSTEM_LINKER: = Determined that LD=3Dld matches the source tree. Not bootstrapping a = cross-linker. >=20 > (Those 2 have significant time implications for the overall > build.) >=20 > Based on (my modified) top, sampling every 3 seconds, >=20 > Mem: . . ., > 754020Ki MaxObsActive, > 186756Ki MaxObsWired, > 923356Ki MaxObs(Act+Wir+Lndry) >=20 > Swap: 1740Mi Total, . . ., > 756828Ki MaxObsUsed, > 1442Mi MaxObs(Act+Lndry+SwapUsed), > 1615Mi MaxObs(Act+Wir+Lndry+SwapUsed) >=20 > So: slightly over 739 MiBytes of swap observed to have been > in use at one time. >=20 > As for the overlapping time's duration: file creation and > modification times, in time order were: >=20 > (via extraction from ls -TldU output:) > 09:37:28 creation of clang/libclang/CodeGen/CodeGenAction.o > 09:38:44 creation of clang/libclang/CodeGen/CodeGenFunction.o > 09:40:19 creation of clang/libclang/CodeGen/CodeGenModule.o > 09:41:28 creation of clang/libclang/CodeGen/CodeGenPGO.o >=20 > (via extraction from ls -Tld output:) > 09:47:15 modification of clang/libclang/CodeGen/CodeGenFunction.o > 09:49:53 modification of clang/libclang/CodeGen/CodeGenAction.o > 09:50:10 modification of clang/libclang/CodeGen/CodeGenPGO.o > 09:54:49 modification of clang/libclang/CodeGen/CodeModule.o >=20 > So: >=20 > 09:41:28 . . . 09:47:15 (under 6 min) for the overlapping time > frame and the highest swap space use happened inside that > interval. >=20 > During this, there were times mixes of CPUn and "swread" STATE > for the compiles. But at no point were all observed to be > blocked waiting, at no point was only 1 observed to show a CPUn > with a large WCPU. >=20 > This is largely attributable to the USB media having tiny > latencies compared to spinning rust and having reasonable > transfer rates for the type of I/O: NMVe USB3 media (that is > also USB2 compatible for USB powered usage). >=20 > My use of: >=20 > # > # Delay when persistent low free RAM leads to > # Out Of Memory killing of processes: > vm.pageout_oom_seq=3D120 >=20 > and: >=20 > # > # Together this pair avoids swapping out the process kernel stacks. > # This avoids processes for interacting with the system from being > # hung-up. > vm.swap_enabled=3D0 > vm.swap_idle_enabled=3D0 >=20 > did not lead to any problems so far. >=20 >=20 > For reference: >=20 > Via systat -vmstat I monitored . . . >=20 > VN PAGER SWAP PAGER > in out in out > count =20 > pages > ioflt . . . > . . . > intrn . . . >=20 > Both VN in and SWAP in can contribute to ioflt, faults > that required I/O. The ioflt number would be before the > "ioflt" text. >=20 > There is a later line that lists intrn (somewhat below > ioflt): "in-transit blocking page faults". The intrn > number would be before the "intrn" text. >=20 > The figures varied under 600 for ioflt and intrn for > what I saw during the large swap space use, with > matching SWAP activity, no significant VN activity. > (The figures are for an about 5 second update interval, > as I remember.) (I watched the on screen updates. > I did not try to capture the material in a file.) >=20 > I expect that these figures would be large for a > sustained period in your context. >=20 >=20 > I also monitored with "gstat -spod". I assume that stat is > more familiar. (I use -spod even when "d" happens to not be > going to show any activity.) >=20 >=20 > [I do not recommend leaving "systat -swap" running: it > accumulates a large set of memory leaks and so can > mess up tracking swap space use by being a signficant > contributor. I did not put it to significant use other > than discovering that problem.] >=20 >=20 > Configuration points . . . >=20 >=20 > /boot/efi/config.txt has: >=20 > enable_uart=3D1 > dtoverlay=3Dmmc > # > # Local addition that avoids (at least) USB3 SSD boot failures that = look like: > # uhub_reattach_port: port ? reset failed, error=3DUSB_ERR_TIMEOUT > # uhub_reattach_port: device problem (USB_ERR_TIMEOUT), disabling = port ? > initial_turbo=3D60 > # > # Local additions: > uart_2ndstage=3D1 > dtdebug=3D1 > kernel=3Du-boot.bin.2023.01.armv7 > kernel7=3Du-boot.bin.2023.01.armv7 > dtoverlay=3Ddisable-bt > # > force_turbo=3D1 >=20 > ( /etc/rc.conf has powerd commented out. ) > (I build u-boot with a couple of settings added.) > (Leaving initial_turbo in place allows disabling > force_turbo independently --but still allowing > the USB booting to work during the temporary > turbo status. intial_turbo is not required when > force_turbo is enabled --but does not hurt.) >=20 > /boot/loader.conf has : >=20 > # > # Delay when persistent low free RAM leads to > # Out Of Memory killing of processes: > vm.pageout_oom_seq=3D120 > # > # For plunty of swap/paging space (will not > # run out), avoid pageout delays leading to > # Out Of Memory killing of processes: > #vm.pfault_oom_attempts=3D-1 > # > # For possibly insufficient swap/paging space > # (might run out), increase the pageout delay > # that leads to Out Of Memory killing of > # processes: > #vm.pfault_oom_attempts=3D 10 > #vm.pfault_oom_wait=3D ??? > # (The multiplication is the total but there > # are other potential tradoffs in the factors > # multiplied, even for nearly the same total.) >=20 > (As I understand you are now using defaults for > vm.pfault_oom_attempts and vm.pfault_oom_wait . > So I did as well for those 2 for this experiment.) >=20 >=20 > /etc/sysctl.conf has: >=20 > # > # Together this pair avoids swapping out the process kernel stacks. > # This avoids processes for interacting with the system from being > # hung-up. > vm.swap_enabled=3D0 > vm.swap_idle_enabled=3D0 >=20 >=20 > # more ~/src.configs/src.conf.CA7-nodbg-clang-alt.aarch64-host=20 > TO_TYPE=3Darmv7 > # > KERNCONF=3DGENERIC-NODBG-CA7 > TARGET=3Darm > .if ${.MAKE.LEVEL} =3D=3D 0 > TARGET_ARCH=3D${TO_TYPE} > .export TARGET_ARCH > .endif > # > WITH_SYSTEM_COMPILER=3D > WITH_SYSTEM_LINKER=3D > # > WITH_ELFTOOLCHAIN_BOOTSTRAP=3D > #Disables avoiding bootstrap: WITHOUT_LLVM_TARGET_ALL=3D > WITHOUT_LLVM_TARGET_AARCH64=3D > WITH_LLVM_TARGET_ARM=3D > WITHOUT_LLVM_TARGET_MIPS=3D > WITHOUT_LLVM_TARGET_POWERPC=3D > WITHOUT_LLVM_TARGET_RISCV=3D > WITHOUT_LLVM_TARGET_X86=3D > WITH_CLANG=3D > WITH_CLANG_IS_CC=3D > WITH_CLANG_FULL=3D > WITH_CLANG_EXTRAS=3D > WITH_LLD=3D > WITH_LLD_IS_LD=3D > # > WITH_LLDB=3D > # > WITH_BOOT=3D > # > WITHOUT_WERROR=3D > MALLOC_PRODUCTION=3D > WITH_MALLOC_PRODUCTION=3D > WITHOUT_ASSERT_DEBUG=3D > WITHOUT_LLVM_ASSERTIONS=3D > # > # Avoid stripping but do not control host -g status as well: > DEBUG_FLAGS+=3D > # > WITH_REPRODUCIBLE_BUILD=3D > WITH_DEBUG_FILES=3D > # > XCFLAGS+=3D -mcpu=3Dcortex-a7 > XCXXFLAGS+=3D -mcpu=3Dcortex-a7 > # There is no XCPPFLAGS but XCPP gets XCFLAGS content. >=20 > (An armv7 host does not need differing content than an > aarch64 host, thus the use of the *.aarch64-host file.) >=20 > However long the overall build ends up taking, the above > is part of why the details end up as they will end up. >=20 >=20 > /etc/crontab notes: >=20 > I do not know if you leave the following enabled during the long > builds or not ( from /etc/crontab ): >=20 > # Perform daily/weekly/monthly maintenance. > 1 3 * * * root periodic daily > 15 4 * * 6 root periodic weekly > 30 5 1 * * root periodic monthly >=20 > that can run things like "/usr/local/sbin/pkg check -qsa" > (daily example) that would compete for resources. I left them > active, so daily competed with the build for a while, but it > did not happen to overlap with the high swapspace use time > frame. >=20 > I commonly disable these for builds that will span into the > hours it indicates, at least when I'm monitoring builds for > comparisons and such. >=20 FYI: the buildworld just completed a bit ago: World built in 117913 seconds, ncpu: 4, make -j4 So, somewhat under 33 hrs for what and how I build, given the media I use. The buildkernel is in process. =3D=3D=3D Mark Millard marklmi at yahoo.com