From nobody Fri Jan 28 05:55:47 2022 X-Original-To: freebsd-arm@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 2BEA1197669F for ; Fri, 28 Jan 2022 05:56:01 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic311-25.consmr.mail.gq1.yahoo.com (sonic311-25.consmr.mail.gq1.yahoo.com [98.137.65.206]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4JlRXH6bcyz3pdS for ; Fri, 28 Jan 2022 05:55:59 +0000 (UTC) (envelope-from marklmi@yahoo.com) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1643349352; bh=ntRQ7LJQ4cKIvgdNuJRRqjYJkaxw/ekQ2K7lmSojTpk=; h=Subject:From:In-Reply-To:Date:Cc:References:To:From:Subject:Reply-To; b=e798nUNaMnXLSfeCyAehTifglvPh75IikBLybGOlpj/wam25juGVbHBNXqkyOH5Zri0h9g5QpfZe8RT8TvaXcUHQ/Y6FiPMePOmQtKg4bPOYR0S23Br+B+DwH38uOyM7kluit0oJFLQsfwE5KUfKNjjZDd21UQJ+/Nz8rjJtWG9Kb9mEcx04VNJjVW0lC3giwj4Vp6xpEb+fAjRZXkGE2MjVlDG8lc4C25ZDoLdn/zprqs4V1Z+S+fqn4l5vMcTA2WUCF+b9Bnxw9YZ2qdh+uiiiho24tVJhxRcXrsubLhiDXQ8UwdeVoebZKDudhHdzx2gyEs9rb846Ud52U3mzhg== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1643349352; bh=/TbDRL5vrM7t4E6PcsatD1KIbPY6TgUMe95mpiY6rT/=; h=X-Sonic-MF:Subject:From:Date:To:From:Subject; b=jM0USl7ArS3Be3G/WGIHMmvb8tPNLMJi+D72ni+h9/bxteyy6OgRb15lUdRrfXsZlOz8GTBL1OPUW6zlAO/8w0qKXSfmTCqclKwIFybY17Tmj9VTxW6JD19pJgdFx1+cY8YmcFQigmdhVs/6tCzGK/C8P3rs1ie4I81+3TuzztxLUpVRpLF0iVegQe8s9pTR3EELHOGnKxbDXqGBuBbyTBTCrlQs0teQJkCpXL1TLUjdeLf+HRfZo+tTxkC685dkwUziWS2PvIL/kyJWL+ISdoIEaKRWBuwqbFNRkvXxbHW+uapQOI41ioYdPeBbDLzhD8GVa+gdH+i7WmgHz6H08Q== X-YMail-OSG: .Y_3XAAVM1lsl4VB0rlg6QoclkBD8KMX3o8Gn7pJzHlejeQlPlQySAbNtlAbZ7G dW1_cx0v2BlG1DZD1kebatygEBE7oMlgf1YzHnT2DQqGlLWear4u.d5iWedp0lJtcpbtNtw3XFQu DDxNNC_P9gmQh0PsZsBdYm4_U63n0nE6J0bQxLZ0RskCWSGbn7NHRCzjs.QLF0mo4X8NLuYQ8UD_ Dtc7SQEJd0euX6fPPcnSIe1FFgv0OznyA3J0o0tPcyhpK7mamhDuSSxHZpxz4plM2_HL1Zc9LRkx jXZ.5bNbjfBHFwSuD3jx9TuipEI3StMJLIJKDswFETC2.cEEuOtXQlE6NBN8D.N5zJzv_48fWdoO CU.Qpd2UoFquejH7rZPmh8oDFPzELgIUpZqGrRN4jIYHo03M60Rin9T0uXGpjqReFbnAbKYdOmvS .jrTXrGdQH8Ypsj871tAKJT.KG9pBkYSNqa3d77ySriV_ehsrmjbqSpAZFYl0X_Z2t3Ptf6h62L. uja5.Bdfb18B8vdKJqydU71k9XKktux9wZpPJYcQoK6E0W_7i0vNi6HskpBl4PbhMJWvl3KKu32E PGnjHq4ee6g3vNF9XFmp7GGgVOHQx.e1K.f1YGYpX6d_XdMiZJalVOUwCeZMDHUadw2vJCnX6_OO tPJlsPji5nvfx2ShDYsgyU7pnwZHe9ECRsDeTaHG3NAgnta1jHOYH3Fr_gHymY4FsSrBxldwtvTQ i8RmkmpmHvVIlN2xLjmLtOfM8w96sp4WH68425afxCrx1W7g3b5rhC231shnVCbswsFTpyHNweGu Tlk3jjVWZzLbjev9TZYBdwFHrl8UCLjXaKOqiSoevgzZlcaJIEZnigdUM.BDoX.UGuUn_BAT5TuA BZYiYX9Lf8hgEzcmkeBK8a7NiIiN0JpyujZRifgNdaS04CX3i3.ApLs5s3ICwQB7yB7IZ36iLUbb hN4Y4BO9lgSkyi44WBWBlOBYTEZZGgmHL7p.gUaiGaoXbZSuoTnpnQpw9T8RRMnuHKUvV1lk_XTw .Z65KQV4MZ1SlgXQ2rv27IMru2IhDlSFUtPEbmF5tIgFoiC1v9X5UlBwTUe9U2d.nKLUNy.AOeDQ nTySQacptPnGExk0ksKV7W4lCCOuP67VIQONSklke8KCWNC1zpq23IZgrjUJSDcFzxCL9lWrli5s h20TFk63A0GJR7m92XeH9RhjwTPX5j0vhp1_ScqqvZESz1WLFlnfXiHPQdb.zJnhILj_R7a3qZ1w 38M_PUc3njL6UxsWvnXk5z8hgFcQ8e2X_79cqKkxkgOh.JLiwyVLhlNjTj3XZD9T0Zs5mpTZuZnj _Se39hLTlxC37UmP8bjfe.E_kjmAumtrnpjVBetEt4itH8cfBlkQW9J2wtsogzsBF0mYGpfZpFIB lvjBjwTuWXgr0QtEv3fLAx3v7g2E1VU9OlDMi14YyrL7GH8cwusO.D.a3LVhI8SzSVoDdM.hCumh L593Gt54z.4wInmcTDeBDKk87gWY3UHZu2YWB3gO9_DOJlTOFLvK.nhIEhuIbwsVvkKc4sAhxTDJ __gKXTLnIgTb2mXfULqk1.4iEJ50vl0aEQXzmZbF__cqj.sbENrza2Tqe8yIUFbm64GKwiGpbSfV AcAVmtZ7RMuI2KwdN03PoZINJoRVvxVnz9hpOyHzuYkmz6eDb2oP3DWsBtMgcMw8un1EIHbNkYE_ O9Nnpo6UtZSnOBQZeUXkdPKhIfcSoyYei7EEOLw4KrJJZph1uiKRR32BiYWZlpU4Pxbyp2NRtJvG Wd4T5GT1GD91mirR_W7iWJ0zsPz2i9wcr7HFU1MOtpCU05rWnbHPd9enj9bE7TC_5JitKOUDPQP5 o1EhvSSI5lKJT5rX_RvgotjLFuyeUVsfsxx02ohk9LspIH9_j7WtzyTozL6yhi.Z2QvU6c1pxD8R D_n1BafWcwvVzXcffBZP5yyERLlrXnTLJpFmVrzJ.h484ICcliWfjUKVU6Kw9lgzuHH7aqwZB9gV eWG_pywR8IeumykxO9oXJPPS7XE3iZO395Q9NAtkeJAsWl7bdZs6gom9WJMy7oApHv.aAq0CGKls lohmlHzl2lVlNfXg1tceEAgScYA1VR0Dkk9hWCkdVCy7dkMHv5adqpBC7RYvrZ6oMMurWwh5iEe8 v7r547MysgaBE4rJD4TeZVwNQSUiynz0xB3W6p70ThVEjUHsSwns_UIr.AmsK3AoHZ_2U6gUc5zL hC8Ib.cDS1D0W6wUXtY0yWWYIzeLSerdfkk.NpNQ2p0WpGsqwEInSDUlTIzFooVwai3tJFIMUvXY L4QdE1m_2eph1gzblWQ-- X-Sonic-MF: Received: from sonic.gate.mail.ne1.yahoo.com by sonic311.consmr.mail.gq1.yahoo.com with HTTP; Fri, 28 Jan 2022 05:55:52 +0000 Received: by kubenode521.mail-prod1.omega.ne1.yahoo.com (VZM Hermes SMTP Server) with ESMTPA ID 0c7342698c51d073dbc089e98026a5b9; Fri, 28 Jan 2022 05:55:49 +0000 (UTC) Content-Type: text/plain; charset=us-ascii List-Id: Porting FreeBSD to ARM processors List-Archive: https://lists.freebsd.org/archives/freebsd-arm List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-arm@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 14.0 \(3654.120.0.1.13\)) Subject: Re: devel/llvm13 failed to reclaim memory on 8 GB Pi4 running -current [ZFS context: used the whole swap space] From: Mark Millard In-Reply-To: <6528ED25-A3C6-4277-B951-1F58ADA2D803@yahoo.com> Date: Thu, 27 Jan 2022 21:55:47 -0800 Cc: Free BSD , Mark Johnston Content-Transfer-Encoding: quoted-printable Message-Id: <10B4E2F0-6219-4674-875F-A7B01CA6671C@yahoo.com> References: <20220127164512.GA51200@www.zefox.net> <2C7E741F-4703-4E41-93FE-72E1F16B60E2@yahoo.com> <20220127214801.GA51710@www.zefox.net> <5E861D46-128A-4E09-A3CF-736195163B17@yahoo.com> <20220127233048.GA51951@www.zefox.net> <6528ED25-A3C6-4277-B951-1F58ADA2D803@yahoo.com> To: bob prohaska X-Mailer: Apple Mail (2.3654.120.0.1.13) X-Rspamd-Queue-Id: 4JlRXH6bcyz3pdS X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yahoo.com header.s=s2048 header.b=e798nUNa; dmarc=pass (policy=reject) header.from=yahoo.com; spf=pass (mx1.freebsd.org: domain of marklmi@yahoo.com designates 98.137.65.206 as permitted sender) smtp.mailfrom=marklmi@yahoo.com X-Spamd-Result: default: False [-3.50 / 15.00]; FREEMAIL_FROM(0.00)[yahoo.com]; MV_CASE(0.50)[]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[yahoo.com:+]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; NEURAL_HAM_SHORT(-1.00)[-1.000]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; MID_RHS_MATCH_FROM(0.00)[]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[98.137.65.206:from]; MLMMJ_DEST(0.00)[freebsd-arm]; RWL_MAILSPIKE_POSSIBLE(0.00)[98.137.65.206:from]; RCVD_COUNT_TWO(0.00)[2] X-ThisMailContainsUnwantedMimeParts: N On 2022-Jan-27, at 17:43, Mark Millard wrote: > On 2022-Jan-27, at 15:30, bob prohaska wrote: >=20 >> On Thu, Jan 27, 2022 at 02:21:44PM -0800, Mark Millard wrote: >>>=20 >>> Okay. I just started a poudriere bulk devel/llvm13 build >>> in a ZFS context: >>>=20 >>> . . . >>> [00:00:37] Pkg: +BE_AMDGPU -BE_FREEBSD +BE_NATIVE -BE_STANDARD = +BE_WASM +CLANG +DOCS +EXTRAS -FLANG +LIT +LLD +LLDB +MLIR -OPENMP = -PYCLANG >>> [00:00:37] New: +BE_AMDGPU -BE_FREEBSD -BE_NATIVE +BE_STANDARD = +BE_WASM +CLANG +DOCS +EXTRAS +FLANG +LIT +LLD +LLDB +MLIR +OPENMP = +PYCLANG >>> . . . >>> [00:01:27] [01] [00:00:00] Building devel/llvm13 | llvm13-13.0.0_3 >>>=20 >>=20 >> Is this ARM hardware, or an emulator? >=20 > 8 GiByte RPi4B, USB3 NVMe media with a ZFS partition. The content > is a slightly modified copy of the HoneyComb's PCIe slot Optane > media. >=20 > The UFS-based 8 GiByte RPi4B is also based on copying from the > same Optane media, both for the system materials and various > ports/packages/pouriere related materials. (Not, necessarily, > other things.) >=20 >> I've been using plain old make in /usr/ports/devel,=20 >> might it be informative to try a poudriere build as well? >=20 > The Pkg:, New:, and llvm13 lines I listed are poudriere(-devel) > output. I am doing my builds via poudriere. ALLOW_PARALLEL_JOBS=3D > and USE_TMPFS=3D"data" in use. >=20 > I have a context in which almost all prerequisites had already > been built. (The change in options lead to 2 very small ports > to build before devel/llvm13's started in a builder.) >=20 > (You might not have a jail that already has the prerequisites.) >=20 >> One would expect the added overhead to increase memory use. >>=20 >=20 > Well, from the context I started in, only devel/llvm13 is being > built once it starts. Once it gets to the build phase (after > dependencies and such are set up), there is not much overhead > because the only activity is the one builder and it is only > building llvm13 --via make in the builder. At the end there > would be extra activity as poudriere finishes up. During the > build phase, I only expect minor overhead from poudriere > monitoring the build logs and such. >=20 > I expect that the mere fact that a poudriere jail is in use > for the builder to execute in does not contribute to > significantly increasing the system's memory use or changing > the system's memory use pattern. >=20 >=20 > There are some other differences my context. The instances of > main [so: 14] are non-debug builds (but with symbols). The > builds are optimized for the RPi4B (and others) via use of > -mcpu=3Dcortex-a72 usage. My /usr/main-src/ does have some > personal changes in it. (Some messaging about the kills is > part of that.) >=20 > The RPi4B's are using: >=20 > over_voltage=3D6=20 > arm_freq=3D2000=20 > sdram_freq_min=3D3200=20 > force_turbo=3D1=20 >=20 > (There are heat-sinks, fans, and good power supplies.) >=20 > The media in use are USB3 1 TB Samsung Portable SSD T7 > Touch's. I'm unlikely to see "swap_pager: indefinite > wait buffer:" notices if the cause was based on the > media performance. (You have spinning rust, if I > remember right.) >=20 > I do not have a monitoring script making a huge log file > during the build. So less is competing for media access > or leading to other overheads. (But, as I remember, > you have gotten the problem without having such a script > running.) ZFS context: Well, the ZFS example used up all the swap space, according to my patched top. This means that my setting of vm.pfault_oom_attempts is not appropriate for this context: # Delay when persistent low free RAM leads to # Out Of Memory killing of processes: vm.pageout_oom_seq=3D120 # # For plunty of swap/paging space (will not # run out), avoid pageout delays leading to # Out Of Memory killing of processes: vm.pfault_oom_attempts=3D-1 # # For possibly insufficient swap/paging space # (might run out), increase the pageout delay # that leads to Out Of Memory killing of # processes (showing defaults at the time): #vm.pfault_oom_attempts=3D 3 #vm.pfault_oom_wait=3D 10 # (The multiplication is the total but there # are other potential tradoffs in the factors # multiplied, even for nearly the same total.) I'll need to retest with something more like the commented out vm.pfault_oom_attempts and vm.pfault_oom_wait figures in order to see the intended handling of the test case. What are you using for each of: vm.pageout_oom_seq ? vm.pfault_oom_attempts ? vm.pfault_oom_wait ? For reference, for ZFS: last pid: 380; load averages: 1.50, 3.07, 3.93 MaxObs: 5.71, = 4.92, 4.76 = up 0+07:23:14 21:23:43 68 threads: 1 running, 65 sleeping, 2 waiting, 19 MaxObsRunning CPU: 13.3% user, 0.0% nice, 4.9% system, 0.9% interrupt, 80.8% idle Mem: 4912Mi Active, 167936B Inact, 1193Mi Laundry, 1536Mi Wired, 40960B = Buf, 33860Ki Free, 6179Mi MaxObsActive, 6476Mi MaxObsWired, 7820Mi = MaxObs(Act+Wir+Lndry) ARC: 777086Ki Total, 132156Ki MFU, 181164Ki MRU, 147456B Anon, 5994Ki = Header, 457626Ki Other 59308Ki Compressed, 254381Ki Uncompressed, 4.29:1 Ratio Swap: 8192Mi Total, 8192Mi Used, K Free, 100% Inuse, 19572Ki In, 3436Ki = Out, 8192Mi MaxObsUsed, 14458Mi MaxObs(Act+Lndry+SwapUsed), 15993Mi = MaxObs(Act+Wir+Lndry+SwapUsed) Console: (Looks like I misremembered adjusting the "out of swap space" wording for the misnomer message.) swap_pager: out of swap space swp_pager_getswapspace(18): failed swap_pager: out of swap space swp_pager_getswapspace(1): failed swp_pager_getswapspace(1): failed swap_pager: out of swap space swp_pager_getswapspace(1): failed swp_pager_getswapspace(7): failed swp_pager_getswapspace(24): failed swp_pager_getswapspace(3): failed swp_pager_getswapspace(18): failed swp_pager_getswapspace(17): failed swp_pager_getswapspace(1): failed swp_pager_getswapspace(12): failed swp_pager_getswapspace(23): failed swp_pager_getswapspace(30): failed swp_pager_getswapspace(3): failed swp_pager_getswapspace(2): failed . . . Then a bunch of time with no messages . . . swp_pager_getswapspace(5): failed swp_pager_getswapspace(28): failed . . . Then a bunch of time with no messages . . . Top again: last pid: 382; load averages: 0.73, 1.00, 2.40 MaxObs: 5.71, = 4.92, 4.76 = up 0+07:31:26 21:31:55 70 threads: 1 running, 65 sleeping, 4 waiting, 19 MaxObsRunning CPU: 0.1% user, 0.0% nice, 5.6% system, 0.0% interrupt, 94.3% idle Mem: 3499Mi Active, 4096B Inact, 2612Mi Laundry, 1457Mi Wired, 40960B = Buf, 34676Ki Free, 6179Mi MaxObsActive, 6476Mi MaxObsWired, 7820Mi = MaxObs(Act+Wir+Lndry) ARC: 777154Ki Total, 135196Ki MFU, 178330Ki MRU, 5995Ki Header, 457631Ki = Other 59520Ki Compressed, 254231Ki Uncompressed, 4.27:1 Ratio Swap: 8192Mi Total, 8192Mi Used, K Free, 100% Inuse, 409600B In, 4096B = Out, 8192Mi MaxObsUsed, 14458Mi MaxObs(Act+Lndry+SwapUsed), 15993Mi = MaxObs(Act+Wir+Lndry+SwapUsed) I then used top to kill ninja and the 4 large compiles that were going on. I'll change: vm.pfault_oom_attempts vm.pfault_oom_wait and reboot and start over. I expect that the ongoing UFS test will likely end up similarly and that similar adjustments and restarts will be needed because of actually running out of swap space. =3D=3D=3D Mark Millard marklmi at yahoo.com