From nobody Mon Jan 22 19:56:34 2024 X-Original-To: dev-commits-src-main@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TJgwQ2DV4z586X3 for ; Mon, 22 Jan 2024 19:56:54 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic312-24.consmr.mail.gq1.yahoo.com (sonic312-24.consmr.mail.gq1.yahoo.com [98.137.69.205]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4TJgwP3GJFz45kh for ; Mon, 22 Jan 2024 19:56:53 +0000 (UTC) (envelope-from marklmi@yahoo.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yahoo.com header.s=s2048 header.b=pUBtdLXu; dmarc=pass (policy=reject) header.from=yahoo.com; spf=pass (mx1.freebsd.org: domain of marklmi@yahoo.com designates 98.137.69.205 as permitted sender) smtp.mailfrom=marklmi@yahoo.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1705953410; bh=ffOgC+s71+NIxokuHfqfhqN0fgVQB45vKImS7Xf43to=; h=From:Subject:Date:References:To:In-Reply-To:From:Subject:Reply-To; b=pUBtdLXubbS8a172um97pA3GNvN7h3fiOmPZVC+yP5JiXD+RfqHKzdNpT+ze1Reu6FmObkQlT9sreZRL6j8n+pxZQtW4zk/FTS7cNHvqXYFSdjDE5LsJC8Iuli4wng8iJP9Bbvtz4AYQiWotk/KW4fkNu0O2HVC8ciHEObihvoHFilzl4Ndz3dSlRWbiJX7yPxtlR5HF58ao8lY7414ySValZx4RiuVUarOuGe8r/I9Luwp6hv+KlHvHl/vX1zT/BydU9l72K9LK0klWgRicZJjXhPQKkJ7w7hn1YnyKdYp8i9WwrRbqymUaS2vdgx3NHWe5e2GIiNXAlyMk1nEQOQ== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1705953410; bh=ZgzTNX4wTUhVD3GToVE+tkwItG0Pswcvz3JSonlWlNf=; h=X-Sonic-MF:From:Subject:Date:To:From:Subject; b=Rz3U0yhuHSXTliKDNfV8V05xn/SsYLO0gG322nJYfQ6eTcP1KeZw6F7OgWLPvxhR3Bf2kq13XKE3JHwEvJqHB+ro/0wTDqx0Soosy8OhxrVzwMXyeyNtjVNKA0YJDnjQx9gvoGTRvzzf14/dYfB/2CgqdydISNUGtdulo5F5JTOrqsJoKttnbVLOnZIlKuXmQyyTVn0Tzy3n+fUz22PTdvIBA8nQITeJ6l7IROTYlI+vyqT77xHicmmA18ACla5VTLaTLCuYHUzpzGgAbdtmJzXW7LiiIVnNaz2JmzSWr5UOXd5EkfKHbDiNiFVijW9jXquprX0B9f/TKlb2NyV0mQ== X-YMail-OSG: Vu1hWD8VM1mzDny2wKXyHsDF7kOs1Q6Mp2c4DjqmY4iRPi2H_ywetxdFiL7p72o llB8.c7fwtm76XGlh23KOIvC3EZDHaVr36_WRhWJLPyVFYJYKv4wq_4Ai3jYYZLJox.BiAfmrbWP BkXfiaSp8Um8iJZZ.48sX00AV1DP0Vb8UBoAtgHspEwCclDoYfT6ILp302uARFc2hXScBWRb4gaP wXrb6ZQdXieL6d37rNjfx26wyLp8GHCjaLIZT7ctoIpFi__bDJnycXWO95krJeGZ0HlnWBmIDxSZ GauKVZGQBYd4MkoH.QGfq0P2aCGKji_5fOG.iehHOV5GES4fZJZrYVwpQDHyBQY7ZAZ8XOZEBQ9e PUIlZJRkOCk0zjczKsEiQqvISVPD_exLuuXdsDuxn7TJnBKsDCQww_8j1mKp50G2Js.7A_Ql2gXu uI4imM.yLSbBU75Eoqx2rDB4tsxwRyCSG1srH6fXq78NAYOXEX9brH82payPg13WBuDGzyWT0VYP pha1cdKav_EHHGNoc_Ye0_ItF2i62EWlR48JoV40I2mC1TSpWhpbX4L6OmwO_L9Z9pdixKU9cH9e MGRyZ9Kva4S.kAP26mjhyMLoFgasSTqPnPlWzDbJA58765.GSJFRePGfGyRk.c7BkyTvvhafOGHE dYUFpcu_Ki2ihSbsuoKmWOZ56AUmfnd6aIOfX8ZAMr.5LqLDD2VTpYngaF93W0Uaz6dMfuLwsUYW ZRUi7xMUWQ_u5o6GCYxj8jJG6lE1HqeJgEE80qCr9c0_.A4dlq7ZfJg3n.bInv2T2cT8tfHMVC9T bCD3yZKBbS4hRXBh6r8AxUecxlVdOVRNVqliog9Ybjn3iv1BLhd_7EdVZ5MxPBukS1blUJY7QPiD 64vKlUPXhmthdeKVNFkhytDcO1mijQ16XDuESFqqXCc40MzwGVmLZzyO1BrUE.gtz6eSbIVfpeo2 vq4tWy3UbO.re1f0NXVtum0iBstKCOu4uFnoh7m2KrhRnx452T3gMqq_E0AV2sgl3FxIlORUI2rn JDgs7laZUb68ArwrMf_qUgyGFOrzb2vVRxfcb1yeD3pcgydevTsMWnDQhKVtAoBxrsaY9Bn9Lqio Mn32V7i32hjTv24lHKtJ.DvoiD7eoRgugshUaALT30v_P_uZEpPrODyFRmt_YSJYK1v0JUWvwK6U jsmhJa3c7U9PPz4DQQ91hLOkP5y_SHQgDoSLwOvE8rUzOmehIN3ymp5mqzxDu779p5qnBb4ZuHYV Wli_sFdSXoRWpnfHyYznUmBqAdWY91ZxzUK709Si9fEuOX5n6kah_A46XmWr8q0DScfBtaK5oeBH G4wUy.woeMMhUCjpEHbfWGViJlDyQSPJzGdK580yL1DqO9Y6q58foajjKMw1p_z4rX_Ss_bKgxnp wPJgKYU9gnlLWOz6rruSjgBqcFql6p74RYL0dXU0WbKHFpx6nZh.ffjBkgsdeuCQDXOxdXyN7i.H xNeY8HgHURTJSC4PkraXswNODA8r8v44LAJs7jQKOtdRjZY47RwEpqpDOxv6PPKHt9u6Tba_OgLr zcpymx10.6dNygV7gBF3Sd83fZ1IrCKNDVwhUwvKybmqZhVtmKnvwASBQegKAZBv7v5QWCpBFvbG 9Ws2en_9qTEROQzN4DQHnyO8q_yFd1BBLxnqQRkPBGSJA9lLxqvoMsmW3LGQOq4iomL.HK1Ahuu0 kW4AWGrxxp8aQVEVKxuXZLadAWqmVBy.unt2FEoYJ0Cm6.s_igx9om9P40MOWL30VME_wI.kDkOr RiEg577U4w6Mz8Q5Qt6Suk.zh7sTuzMUaS1B4l9XUAwjlZghvPcbJCMusH2PzUDuweuvnUddAJJT K0a7QvHKXI.s3oNuYGhW0gJvvNeYO_o4f1VhFzYBUaHd1HjmOH9uuxpS5TUoWOYowaRsbnOwNSUx zpmkY.zcKNNe5EjUWKMjhsWIhi6RX_ya_3kfPNjfQddU1dBZ_bw2e3QBXydqbyXXCH_ghEwkXRlF N5VnVeadLiXR35FM7rTYCqlzrpcf_oYsH4ky1mr7NsJ95q8TnAzLgjao5kkaLseMRqH5NI3Zm811 ZCRWujRfYx.45euz8xui1fVLcaA.8IA4QxWL3GZ8RfOhLA7HBN7DRbLcT19ldxZhZDJRc9y5K03A B_fGbH_8zdToIRhigcHnePbpe_rTRi4qY8UQMM1_bTTVMIBX1rV6gX0rwoysxYe_MVaUugiOqn_7 QkGA6r.ZxSqILR9tHKSl9cSz7TrmUH4BY4.BrKuxwVtimaywO.hf2s5z.BzwaDOA_IvovKUh07tW 8YTz6EPwc9cSbOQH85g-- X-Sonic-MF: X-Sonic-ID: a12f69ff-2ef0-4164-aac2-dddaed5d75ed Received: from sonic.gate.mail.ne1.yahoo.com by sonic312.consmr.mail.gq1.yahoo.com with HTTP; Mon, 22 Jan 2024 19:56:50 +0000 Received: by hermes--production-gq1-78d49cd6df-c95sd (Yahoo Inc. Hermes SMTP Server) with ESMTPA ID 88f9494b0c4555b12a55aca88dba7b05; Mon, 22 Jan 2024 19:56:45 +0000 (UTC) From: Mark Millard Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable List-Id: Commit messages for the main branch of the src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-main List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-dev-commits-src-main@freebsd.org X-BeenThere: dev-commits-src-main@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3774.300.61.1.2\)) Subject: Re: git: 636592343c3e - main - tmpfs: increase memory reserve to a percent of available memory + swap Date: Mon, 22 Jan 2024 11:56:34 -0800 References: <33F50320-DA8F-4913-9197-5AED3D916E7A@yahoo.com> To: Baptiste Daroussin , dev-commits-src-main@freebsd.org In-Reply-To: <33F50320-DA8F-4913-9197-5AED3D916E7A@yahoo.com> Message-Id: X-Mailer: Apple Mail (2.3774.300.61.1.2) X-Spamd-Bar: --- X-Spamd-Result: default: False [-3.50 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; MV_CASE(0.50)[]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; MIME_GOOD(-0.10)[text/plain]; RCPT_COUNT_TWO(0.00)[2]; RCVD_TLS_LAST(0.00)[]; MIME_TRACE(0.00)[0:+]; TO_DN_SOME(0.00)[]; ARC_NA(0.00)[]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; FREEMAIL_FROM(0.00)[yahoo.com]; FROM_HAS_DN(0.00)[]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; FROM_EQ_ENVFROM(0.00)[]; DKIM_TRACE(0.00)[yahoo.com:+]; MLMMJ_DEST(0.00)[dev-commits-src-main@freebsd.org]; RWL_MAILSPIKE_POSSIBLE(0.00)[98.137.69.205:from]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[98.137.69.205:from] X-Rspamd-Queue-Id: 4TJgwP3GJFz45kh On Jan 22, 2024, at 11:37, Mark Millard wrote: >=20 > Baptiste Daroussin wrote on > Date: Mon, 22 Jan 2024 13:23:37 UTC : >=20 >> On Mon, Jan 22, 2024 at 07:15:22AM -0600, Mike Karels wrote: >>> On 21 Jan 2024, at 21:48, Alexey Dokuchaev wrote: >>>=20 >>>> On Mon, Jan 22, 2024 at 03:27:57AM +0000, Jessica Clarke wrote: >>>>> On 19 Dec 2023, at 15:34, Mike Karels wrote: >>>>>> commit 636592343c3ec0feb61a4d8043676381384420dd >>>>>>=20 >>>>>> tmpfs: increase memory reserve to a percent of available memory + = swap >>>>>>=20 >>>>>> [...] >>>>>>=20 >>>>>> Add sysctl for vfs.tmpfs.memory_percent and the pre-existing >>>>>> vfs.tmpfs.memory_reserved to tmpfs(5). >>>>>>=20 >>>>>> PR: 275436 >>>>>> MFC after: 1 month >>>>>> Reviewed by: rgrimes >>>>>> Differential Revision: https://reviews.freebsd.org/D43011 >>>>>=20 >>>>> I just got: >>>>>=20 >>>>> make[6]: Could not create temporary file /tmp/makeDl4i9S: >>>>> No space left on device >>>>>=20 >>>>> after doing a buildworld -j4 on a 4 core, 16 GiB system where /tmp = is a >>>>> tmpfs, with the prior buildworld having taken me past this commit, >>>>> which I strongly suspect to be the cause. Whilst I understand the >>>>> motivation behind your commits, this is a sad regression to have; = it's >>>>> not exactly a particularly strange situation. >>>=20 >>> Bug 276402 is about the interaction of this change with ZFS ARC, = which >>> seems to be problematical. I spent some time investigating = yesterday. >>> Depending on the dynamics, I get different results. If tmpfs grows, >>> the ARC will be reduced if needed. But if the ARC grows to the point >>> that tmpfs sets free space to 0 or quite low, attempts to write to >>> tmpfs fail without any reduction in ARC. >>>=20 >>> Jess, two quesions: >>>=20 >>> 1. Are you using ZFS on this system? >>>=20 >>> 2. Can you try with vfs.tmpfs.memory_percent set to 100? >>>=20 >>>> FWIW, I've seen two people complain about this last week, = apparently >>>> this kernel OOM protection doesn't work very well. :-/ >>>=20 >>> So far, I have only seen OOM kills when memory + swap are quite = short, >>> with a tmpfs memory_percent above 95. But with ZFS, it can happen >>> when the ARC could have been reduced. >>>=20 >>> I will investigate more today. If I don't find a way to resolve = this, >>> I'll probably commit a change to set vfs.tmpfs.memory_percent to 100 >>> by default, at least for now, and if that works around the problem. >>>=20 >> This is easily reproducible with poudriere, if you try to build some >> ports/packages which are big memory consumers, for example any chrome = variant or >> just lang/rust, I have a machine with 64G of ram >=20 > How many hardware threads? > How much swap space? For got to ask: How many parallel builders allowed in the bulk run at once? > You specified: USE_TMPFS=3Dall > ALLOW_MAKE_JOBS=3Dyes ? > MAKE_JOBS_NUMBER use (or the like)? > I assume ZFS but any ARC specific tuning? >=20 > Not that you would want to do what I do, but it > illustrates why I ask about the contexts explored. >=20 >=20 > 16 hardware thread aarch64 with 64 GiBytes of RAM context: >=20 > I've historically used: Forgot to say: Number of parallel builders allowed in the bulk run equal to the count of hardware threads, so 16 here. > ZFS untuned ARC context (but I also do the same in UFS contexts) > USE_TMPFS=3Dall > ALLOW_MAKE_JOBS=3Dyes > no MAKE_JOBS_NUMBER other than for editors/lapce* > RAM+SWAP =3D=3D 310 GiBytes (so a little over RAM+SWAP=3D=3D4.8*RAM) >=20 > Basically SWAP is preventing tmpfs use from running > out of space, despite the likes of 25 GiByte+ use > of tmpfs by the rust build, for example. >=20 > Historically multiple toolchains building in parallel > in the resulting high load average context have > completed just fine, including when rust is one of > them. I've not come close to running out of RAM+SWAP > so far. >=20 > Although my testing of "bulk -a" is very rare, such > has worked when I tried it, also using the hig load > average style. >=20 > Why ~4.8? Because somewhere between there and 5.0 the > binding of the swap space starts puttting out a notice > about potentially being mistuned. (It reports on a > potential change that I do not know the other tradeoffs > for. So I stay in the range where no notice is made.) >=20 >> and it is impossible to get >> lang/rust build in poudriere (USE_TMPFS=3Dall) without setting >> vfs.tmpfs.memory_percent to 100. >=20 > I'm guessing this is implicitly: without having a huge > RAM+SWAP space. >=20 >> the poudriere dies with plenty of no space left on device. >=20 > I will note that I've not tested my configuration > with the change yet, holding back on updating FreeBSD > for other reasons. But I doubt that I'd run out of > RAM+SWAP, given the huge RAM+SWAP that I've > historically used for the context. >=20 > I do analogously on 2 other systems: > 32 GiBytes of RAM and 8 hardware threads > 192 GiBytes of RAM and 32 hardware threads > (Both having both ZFS and UFS media.) >=20 > On the various small RAM systems, I use > USE_TMPFS=3Ddata or USE_TMPFS=3Dno and avoid > ZFS. >=20 > I also always avoid spinning rust. >=20 =3D=3D=3D Mark Millard marklmi at yahoo.com