Re: Re: git: 636592343c3e - main - tmpfs: increase memory reserve to a percent of available memory + swap

On Mon, Jan 22, 2024 at 07:15:22AM -0600, Mike Karels wrote:
> On 21 Jan 2024, at 21:48, Alexey Dokuchaev wrote:
> 
> > On Mon, Jan 22, 2024 at 03:27:57AM +0000, Jessica Clarke wrote:
> >> On 19 Dec 2023, at 15:34, Mike Karels <karels@FreeBSD.org> wrote:
> >>> commit 636592343c3ec0feb61a4d8043676381384420dd
> >>>
> >>>    tmpfs: increase memory reserve to a percent of available memory + swap
> >>>
> >>>    [...]
> >>>
> >>>    Add sysctl for vfs.tmpfs.memory_percent and the pre-existing
> >>>    vfs.tmpfs.memory_reserved to tmpfs(5).
> >>>
> >>>    PR:             275436
> >>>    MFC after:      1 month
> >>>    Reviewed by:    rgrimes
> >>>    Differential Revision:  https://reviews.freebsd.org/D43011
> >>
> >> I just got:
> >>
> >>   make[6]: Could not create temporary file /tmp/makeDl4i9S:
> >>   No space left on device
> >>
> >> after doing a buildworld -j4 on a 4 core, 16 GiB system where /tmp is a
> >> tmpfs, with the prior buildworld having taken me past this commit,
> >> which I strongly suspect to be the cause. Whilst I understand the
> >> motivation behind your commits, this is a sad regression to have; it's
> >> not exactly a particularly strange situation.
> 
> Bug 276402 is about the interaction of this change with ZFS ARC, which
> seems to be problematical.  I spent some time investigating yesterday.
> Depending on the dynamics, I get different results.  If tmpfs grows,
> the ARC will be reduced if needed.  But if the ARC grows to the point
> that tmpfs sets free space to 0 or quite low, attempts to write to
> tmpfs fail without any reduction in ARC.
> 
> Jess, two quesions:
> 
> 1. Are you using ZFS on this system?
> 
> 2. Can you try with vfs.tmpfs.memory_percent set to 100?
> 
> > FWIW, I've seen two people complain about this last week, apparently
> > this kernel OOM protection doesn't work very well. :-/
> 
> So far, I have only seen OOM kills when memory + swap are quite short,
> with a tmpfs memory_percent above 95.  But with ZFS, it can happen
> when the ARC could have been reduced.
> 
> I will investigate more today.  If I don't find a way to resolve this,
> I'll probably commit a change to set vfs.tmpfs.memory_percent to 100
> by default, at least for now, and if that works around the problem.
> 
This is easily reproducible with poudriere, if you try to build some
ports/packages which are big memory consumers, for example any chrome variant or
just lang/rust, I have a machine with 64G of ram and it is impossible to get
lang/rust build in poudriere (USE_TMPFS=all) without setting
vfs.tmpfs.memory_percent to 100.

the poudriere dies with plenty of no space left on device.

Best regards,
Bapt