Re: git: 636592343c3e - main - tmpfs: increase memory reserve to a percent of available memory + swap

On Jan 22, 2024, at 11:37, Mark Millard <marklmi@yahoo.com> wrote:
> 
> Baptiste Daroussin <bapt_at_freebsd.org> wrote on
> Date: Mon, 22 Jan 2024 13:23:37 UTC :
> 
>> On Mon, Jan 22, 2024 at 07:15:22AM -0600, Mike Karels wrote:
>>> On 21 Jan 2024, at 21:48, Alexey Dokuchaev wrote:
>>> 
>>>> On Mon, Jan 22, 2024 at 03:27:57AM +0000, Jessica Clarke wrote:
>>>>> On 19 Dec 2023, at 15:34, Mike Karels <karels@FreeBSD.org> wrote:
>>>>>> commit 636592343c3ec0feb61a4d8043676381384420dd
>>>>>> 
>>>>>> tmpfs: increase memory reserve to a percent of available memory + swap
>>>>>> 
>>>>>> [...]
>>>>>> 
>>>>>> Add sysctl for vfs.tmpfs.memory_percent and the pre-existing
>>>>>> vfs.tmpfs.memory_reserved to tmpfs(5).
>>>>>> 
>>>>>> PR: 275436
>>>>>> MFC after: 1 month
>>>>>> Reviewed by: rgrimes
>>>>>> Differential Revision: https://reviews.freebsd.org/D43011
>>>>> 
>>>>> I just got:
>>>>> 
>>>>> make[6]: Could not create temporary file /tmp/makeDl4i9S:
>>>>> No space left on device
>>>>> 
>>>>> after doing a buildworld -j4 on a 4 core, 16 GiB system where /tmp is a
>>>>> tmpfs, with the prior buildworld having taken me past this commit,
>>>>> which I strongly suspect to be the cause. Whilst I understand the
>>>>> motivation behind your commits, this is a sad regression to have; it's
>>>>> not exactly a particularly strange situation.
>>> 
>>> Bug 276402 is about the interaction of this change with ZFS ARC, which
>>> seems to be problematical. I spent some time investigating yesterday.
>>> Depending on the dynamics, I get different results. If tmpfs grows,
>>> the ARC will be reduced if needed. But if the ARC grows to the point
>>> that tmpfs sets free space to 0 or quite low, attempts to write to
>>> tmpfs fail without any reduction in ARC.
>>> 
>>> Jess, two quesions:
>>> 
>>> 1. Are you using ZFS on this system?
>>> 
>>> 2. Can you try with vfs.tmpfs.memory_percent set to 100?
>>> 
>>>> FWIW, I've seen two people complain about this last week, apparently
>>>> this kernel OOM protection doesn't work very well. :-/
>>> 
>>> So far, I have only seen OOM kills when memory + swap are quite short,
>>> with a tmpfs memory_percent above 95. But with ZFS, it can happen
>>> when the ARC could have been reduced.
>>> 
>>> I will investigate more today. If I don't find a way to resolve this,
>>> I'll probably commit a change to set vfs.tmpfs.memory_percent to 100
>>> by default, at least for now, and if that works around the problem.
>>> 
>> This is easily reproducible with poudriere, if you try to build some
>> ports/packages which are big memory consumers, for example any chrome variant or
>> just lang/rust, I have a machine with 64G of ram
> 
> How many hardware threads?
> How much swap space?

For got to ask: How many parallel builders allowed
in the bulk run at once?

> You specified: USE_TMPFS=all
> ALLOW_MAKE_JOBS=yes ?
> MAKE_JOBS_NUMBER use (or the like)?
> I assume ZFS but any ARC specific tuning?
> 
> Not that you would want to do what I do, but it
> illustrates why I ask about the contexts explored.
> 
> 
> 16 hardware thread aarch64 with 64 GiBytes of RAM context:
> 
> I've historically used:

Forgot to say:

Number of parallel builders allowed in the bulk run
equal to the count of hardware threads, so 16 here.

> ZFS untuned ARC context (but I also do the same in UFS contexts)
> USE_TMPFS=all
> ALLOW_MAKE_JOBS=yes
> no MAKE_JOBS_NUMBER other than for editors/lapce*
> RAM+SWAP == 310 GiBytes (so a little over RAM+SWAP==4.8*RAM)
> 
> Basically SWAP is preventing tmpfs use from running
> out of space, despite the likes of 25 GiByte+ use
> of tmpfs by the rust build, for example.
> 
> Historically multiple toolchains building in parallel
> in the resulting high load average context have
> completed just fine, including when rust is one of
> them. I've not come close to running out of RAM+SWAP
> so far.
> 
> Although my testing of "bulk -a" is very rare, such
> has worked when I tried it, also using the hig load
> average style.
> 
> Why ~4.8? Because somewhere between there and 5.0 the
> binding of the swap space starts puttting out a notice
> about potentially being mistuned. (It reports on a
> potential change that I do not know the other tradeoffs
> for. So I stay in the range where no notice is made.)
> 
>> and it is impossible to get
>> lang/rust build in poudriere (USE_TMPFS=all) without setting
>> vfs.tmpfs.memory_percent to 100.
> 
> I'm guessing this is implicitly: without having a huge
> RAM+SWAP space.
> 
>> the poudriere dies with plenty of no space left on device.
> 
> I will note that I've not tested my configuration
> with the change yet, holding back on updating FreeBSD
> for other reasons. But I doubt that I'd run out of
> RAM+SWAP, given the huge RAM+SWAP that I've
> historically used for the context.
> 
> I do analogously on 2 other systems:
> 32  GiBytes of RAM and  8 hardware threads
> 192 GiBytes of RAM and 32 hardware threads
> (Both having both ZFS and UFS media.)
> 
> On the various small RAM systems, I use
> USE_TMPFS=data or USE_TMPFS=no and avoid
> ZFS.
> 
> I also always avoid spinning rust.
> 




===
Mark Millard
marklmi at yahoo.com