From nobody Mon Jan 22 19:37:23 2024 X-Original-To: dev-commits-src-main@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TJgVF1dczz584l5 for ; Mon, 22 Jan 2024 19:37:41 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic305-20.consmr.mail.gq1.yahoo.com (sonic305-20.consmr.mail.gq1.yahoo.com [98.137.64.83]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4TJgVC57y7z44Bb for ; Mon, 22 Jan 2024 19:37:39 +0000 (UTC) (envelope-from marklmi@yahoo.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yahoo.com header.s=s2048 header.b=lUxkTIVN; dmarc=pass (policy=reject) header.from=yahoo.com; spf=pass (mx1.freebsd.org: domain of marklmi@yahoo.com designates 98.137.64.83 as permitted sender) smtp.mailfrom=marklmi@yahoo.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1705952257; bh=0g3hRfuf0rJ36XR4M0MW/y+RIm9bdZlpkfA+MIzmBW8=; h=From:Subject:Date:To:References:From:Subject:Reply-To; b=lUxkTIVNPMQrJUmZHINTA2iwfBzRd6ynhShLQND7bM/IVXnSwir3pIkS7oOH3NZjAG0ef1AHCCExFJ7nR73Om42ObYPiAMX3JHZZFAAhaYXDVhwutaCv0KSQwJFWkqv1y6GdqoSH33lgBMzB6RRjoh1VhcqKg6/eTz56OLSDsDg9LKMjf8SmafA1FpQocFvGhbWaPK8Ptp8yZVSIUVsgC2K1mRqRbB+87fuPXOAJBO35k+PxMHHKUJa3GNp9PbVV1AgRm2vind2iAuKE3FCH6jt2ka33GJKHxfkXTK+RTGOvDYz5l134wqbqExH5+6ZM38SFtlw8FZwnnliTUTCaUA== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1705952257; bh=49mJ1DEE09C8/WkRDa9FD52j0Q05c9VtnxVxu4b7h3s=; h=X-Sonic-MF:From:Subject:Date:To:From:Subject; b=UIPaziFzH9DnJppcata250hWUoMZMLpXilx0jd7kztAeVsu6D5IHZZzXS8IA98BO+FLTMrQUt8Mnw4pSDFSzdW/szd1bXjZu6ux2e/hEPWkQblK+lElNkxJTsJb/ZR71QBYfC+NhxOs+JroVtqRdW7vJyMGASXAOIC818aX/XN6dc+XPIYCorsJM15C2jo5r8BbV3Il8P4D6o2s3ihqHnwhPERghTDlx9EksrEbaWNXnDBatJk1LNKsAOVOP/BVV+crPdnYIrO4ZFGB0gOlYuw5buL+Olv7Gpt0sDbBQzbTD3AMUPJ1e0C8mkw74xGpcI93xNriq4uCf3LA6AHRddA== X-YMail-OSG: 09vmQvIVM1lCb.cOEEa5hMokD6QfJ.qSEf353FIjWDOoZPICEcYOKUiTuZwA6Ar Q3oXPz03j2pOT49wO908lpUb6ZHGvH6jIAiWfdm1785VqusXYXapEbeFDFAjtaAQKoxMvXUJ2tCh 5HRwtX1fbirzc31yBUuMoN5gdnUdK4L12uc8vzih_efULnX1he.XLsFgronFcXLUOe.OriqQQ2mX trsHW4_ZG1pHZYLEoWcfLQ9K8lyjiXXcqFcF7pZFlm10V.hflt5mTRLNRhlz_usqIoY9JI.4in3G R3NNwMrXXAjBjq7gCc6mzsqtf1rHpvJPGfVnd.AlrxhoYcSIXi.X8K0IqOMN_ne3A_0sHewl62Uw kUJUAdDWuwJ.C7_lBJt5BlKZTEGJSbERxDSdKISpnJ4KWfMA2o6uNSigRNnfuij3k4xlsvIvhXO9 RG4mLD1LITTr_133mv8pMe0PnBaPHajUOzTf7N4v_KZCDqDWeXzoruLc3V7yVz9j_1l9EZq9dNN0 kEI_Kp39eirJVzov46V_E0s133zbkV6XDdokJPCXYNqwy52jr9v5cQotAxA8x0zakrkdpRjsRk_a _4hrVMjquaPkf9iboBLNUdIGD21gWIXmR.uHwHk6EJLi22h5Wj6xOuvzGMGIsPp6aURIDcerLQvB ZAsVMtdetOMCfT4xM_rPKL.1DD76CMx8xQgA6rNEY9A2N2iUdS.pY9Fy3TkY6PQ6637i.oCozzAh 0Iv0DXKHaolvBs3PbjB8DDT37hp_hWfHC9lKW9wnfQSNpGl5p21B8eTAsGaQYhnfaDGtFCWPMNv0 113uM8hWZVowFGb3HFasutlVqiuNFMJa1CapCDaWtmr9V5JBC1.Q4ak8lqeDe1ZM8.teQwCBBWq. Gg.NBhP0mezsUZyxhwyMKLzPyiu.dRUbCxC5VKpo2u0CgmJOi_dzJW7CbLrkh3YekKwy7_Ehgp_2 qTCYWNI6b._zt3Mc405Qw4FIDv1sdOPXJNC6_q8gREVEONtE.KmnoPduAgiE4U3930l7009cuyCY I3Lcl10gfiQnF5riFAAghTTjgVDlFjE.e6MAIkrM81OzXS9I25apHbRGgK7ekdk.RMRk4z1JXt0z ZmEgARtqDEFEQrkqJh5f.KOgtBDJRBFVEaNhnQjZuBk02GRVl1vslAjUw.d4VEHgTK84xtd8Bzb4 xDqPREkQaCMygGFcP1asJr9.f0Md99bfDgK7B5_GSh.Zi7CeCp6O2Ein51MwjC0hFn4Gur_AMxf4 tGFTbUGUEmdpBJDtHfr6BzGlpzLqdgFdNwQgSL_PVlRIGfbA2X4o7hxyFqlTw0lciBC0Hw36VOK3 8P9bMbIiHG8a_ns8WGToSLojIUcAn7xb0fcM11ERHaJk1aBi20no31NWFbJ.UCNWKReBNuYmIVdV CbjfvhW.ei1fleDBcD7_PrXMLZvR0tndQiX3pFK.dsixbV4if4ajP5AlO5OddwCUCGf736C0Y1iZ OR3ApCMM_5O.EnjKjnQXSI1NbNQQuONSQBBqXJvDx7CatIdYQ40yT.U7YLhESQtsg543j1EdMqkK dOVk3ACmweMaDDUxCd.UIr_ufGGfDZAaEERZccDmQ74VXpm8fyuRGXG0TQX85b1UxT5aDDMMW2me kc0knYpHGie.Fgs3lM31kyOYoC7XNUV3eWcCG2GMhgEyWKh2A6NJQSyNmrIGoKD4vYexwkIgl4t. cWcA3FyBzA2Zkf8rJHZSphoudPeA1l.KKv0bucZlQCi6RUaV1V.RAz6B9PBRE4T2jkzIxFIh9BAN 8XYYfNBWeKdysgFxL1XXsIDDBCKVoxHhH93foDhRmgImUJu.itxYTexsCH51Zt6G0vJXNwNW8QEs qCxCK_EprhPBDklpwhl1P7aFEw23CJIHA.NThjfPq13JteddT5hIKzTxyg_5lHo4A5J8i6AqnITY 8aO_b0Y.d.rSwTGQonl91Ro_TMNvnNPNXCvQWZappNdFJFEI_dLIrclIgWO2pmQFGZ_6pipl6pW8 6mJgbLWD.ks9F5K5.7rBqNjbfIMnH8xsWQuoaa.Km7TtNLyPlohHICedXag67uWT2HJ3HX7lIrvZ CPC8aZuTHtHo8f83Zwn9GzW0F342qcwP9mHa1utcs9kd9nQiKEr4z1Db.4cgOqOtzTMpAclNEqWQ K.1yn8PvZudI9vPV.dI4pkjUw.eXdp7vkS5l71UQQdvaGqSp3eOfWusTeqeE6tpgbR_ioErsyY_8 ZKFOHQbLlxb0oEQ5Js1GmlGzPL3NlD99p4Vj1lLWtFSx3.bzBvM7xW.MYUXk.F48hE2Kba1P.22I M7P1fl2.M0H13L7ruNQ-- X-Sonic-MF: X-Sonic-ID: 65cc2774-6786-46bc-b2a9-64a09ecd0408 Received: from sonic.gate.mail.ne1.yahoo.com by sonic305.consmr.mail.gq1.yahoo.com with HTTP; Mon, 22 Jan 2024 19:37:37 +0000 Received: by hermes--production-gq1-78d49cd6df-k5hld (Yahoo Inc. Hermes SMTP Server) with ESMTPA ID c4a9d5619fe09eda10e706b90849506b; Mon, 22 Jan 2024 19:37:34 +0000 (UTC) From: Mark Millard Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable List-Id: Commit messages for the main branch of the src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-main List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-dev-commits-src-main@freebsd.org X-BeenThere: dev-commits-src-main@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3774.300.61.1.2\)) Subject: Re: git: 636592343c3e - main - tmpfs: increase memory reserve to a percent of available memory + swap Message-Id: <33F50320-DA8F-4913-9197-5AED3D916E7A@yahoo.com> Date: Mon, 22 Jan 2024 11:37:23 -0800 To: Baptiste Daroussin , dev-commits-src-main@freebsd.org X-Mailer: Apple Mail (2.3774.300.61.1.2) References: <33F50320-DA8F-4913-9197-5AED3D916E7A.ref@yahoo.com> X-Spamd-Bar: --- X-Spamd-Result: default: False [-3.50 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; MV_CASE(0.50)[]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; MIME_GOOD(-0.10)[text/plain]; RCPT_COUNT_TWO(0.00)[2]; RCVD_TLS_LAST(0.00)[]; MIME_TRACE(0.00)[0:+]; TO_DN_SOME(0.00)[]; ARC_NA(0.00)[]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim]; FREEMAIL_FROM(0.00)[yahoo.com]; FROM_HAS_DN(0.00)[]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; FROM_EQ_ENVFROM(0.00)[]; DKIM_TRACE(0.00)[yahoo.com:+]; MLMMJ_DEST(0.00)[dev-commits-src-main@freebsd.org]; RWL_MAILSPIKE_POSSIBLE(0.00)[98.137.64.83:from]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[98.137.64.83:from] X-Rspamd-Queue-Id: 4TJgVC57y7z44Bb Baptiste Daroussin wrote on Date: Mon, 22 Jan 2024 13:23:37 UTC : > On Mon, Jan 22, 2024 at 07:15:22AM -0600, Mike Karels wrote: > > On 21 Jan 2024, at 21:48, Alexey Dokuchaev wrote: > >=20 > > > On Mon, Jan 22, 2024 at 03:27:57AM +0000, Jessica Clarke wrote: > > >> On 19 Dec 2023, at 15:34, Mike Karels wrote: > > >>> commit 636592343c3ec0feb61a4d8043676381384420dd > > >>> > > >>> tmpfs: increase memory reserve to a percent of available memory = + swap > > >>> > > >>> [...] > > >>> > > >>> Add sysctl for vfs.tmpfs.memory_percent and the pre-existing > > >>> vfs.tmpfs.memory_reserved to tmpfs(5). > > >>> > > >>> PR: 275436 > > >>> MFC after: 1 month > > >>> Reviewed by: rgrimes > > >>> Differential Revision: https://reviews.freebsd.org/D43011 > > >> > > >> I just got: > > >> > > >> make[6]: Could not create temporary file /tmp/makeDl4i9S: > > >> No space left on device > > >> > > >> after doing a buildworld -j4 on a 4 core, 16 GiB system where = /tmp is a > > >> tmpfs, with the prior buildworld having taken me past this = commit, > > >> which I strongly suspect to be the cause. Whilst I understand the > > >> motivation behind your commits, this is a sad regression to have; = it's > > >> not exactly a particularly strange situation. > >=20 > > Bug 276402 is about the interaction of this change with ZFS ARC, = which > > seems to be problematical. I spent some time investigating = yesterday. > > Depending on the dynamics, I get different results. If tmpfs grows, > > the ARC will be reduced if needed. But if the ARC grows to the point > > that tmpfs sets free space to 0 or quite low, attempts to write to > > tmpfs fail without any reduction in ARC. > >=20 > > Jess, two quesions: > >=20 > > 1. Are you using ZFS on this system? > >=20 > > 2. Can you try with vfs.tmpfs.memory_percent set to 100? > >=20 > > > FWIW, I've seen two people complain about this last week, = apparently > > > this kernel OOM protection doesn't work very well. :-/ > >=20 > > So far, I have only seen OOM kills when memory + swap are quite = short, > > with a tmpfs memory_percent above 95. But with ZFS, it can happen > > when the ARC could have been reduced. > >=20 > > I will investigate more today. If I don't find a way to resolve = this, > > I'll probably commit a change to set vfs.tmpfs.memory_percent to 100 > > by default, at least for now, and if that works around the problem. > >=20 > This is easily reproducible with poudriere, if you try to build some > ports/packages which are big memory consumers, for example any chrome = variant or > just lang/rust, I have a machine with 64G of ram How many hardware threads? How much swap space? You specified: USE_TMPFS=3Dall ALLOW_MAKE_JOBS=3Dyes ? MAKE_JOBS_NUMBER use (or the like)? I assume ZFS but any ARC specific tuning? Not that you would want to do what I do, but it illustrates why I ask about the contexts explored. 16 hardware thread aarch64 with 64 GiBytes of RAM context: I've historically used: ZFS untuned ARC context (but I also do the same in UFS contexts) USE_TMPFS=3Dall ALLOW_MAKE_JOBS=3Dyes no MAKE_JOBS_NUMBER other than for editors/lapce* RAM+SWAP =3D=3D 310 GiBytes (so a little over RAM+SWAP=3D=3D4.8*RAM) Basically SWAP is preventing tmpfs use from running out of space, despite the likes of 25 GiByte+ use of tmpfs by the rust build, for example. Historically multiple toolchains building in parallel in the resulting high load average context have completed just fine, including when rust is one of them. I've not come close to running out of RAM+SWAP so far. Although my testing of "bulk -a" is very rare, such has worked when I tried it, also using the hig load average style. Why ~4.8? Because somewhere between there and 5.0 the binding of the swap space starts puttting out a notice about potentially being mistuned. (It reports on a potential change that I do not know the other tradeoffs for. So I stay in the range where no notice is made.) > and it is impossible to get > lang/rust build in poudriere (USE_TMPFS=3Dall) without setting > vfs.tmpfs.memory_percent to 100. I'm guessing this is implicitly: without having a huge RAM+SWAP space. > the poudriere dies with plenty of no space left on device. I will note that I've not tested my configuration with the change yet, holding back on updating FreeBSD for other reasons. But I doubt that I'd run out of RAM+SWAP, given the huge RAM+SWAP that I've historically used for the context. I do analogously on 2 other systems: 32 GiBytes of RAM and 8 hardware threads 192 GiBytes of RAM and 32 hardware threads (Both having both ZFS and UFS media.) On the various small RAM systems, I use USE_TMPFS=3Ddata or USE_TMPFS=3Dno and avoid ZFS. I also always avoid spinning rust. =3D=3D=3D Mark Millard marklmi at yahoo.com