Runaway processes freeze the system

[Yuri]

On 04/20/18 12:17, Stefan Blachmann wrote:
> Ideally rctl(8) should be the solution.
> Actions can be defined to refuse to allocate more memory or kill
> processes that demand too much memory.
>
> By turning off swap and restricting memory usage one can improve
> performance by preventing the swapper making the system sluggish.
>
> However, rctl is only of limited use for desktop users, as Xorg fails
> to start if vmemoryuse gets restricted (no matter how big the value).
> So rctl(8) appears of practical use only for example for protecting
> servers and embedded systems from crashing and damaging filesystems
> because of user programs eating up too much memory.
>
> I advise you to experiment with it, as the rctl(8) documentation/man
> page seems partly incorrect/misleading, for example regarding per-user
> resource limiting.


I am trying to understand what happens when both memory and swap are 
completely exhausted, and some process still tries to allocate more. 
This process calls mmap, but it can't succeed. There are only a few 
things that the kernel can do: 1. wait in a hope that some other process 
will return some memory soon 2. fail mmap 3. kill some process to free 
memory.


It seems like the current behavior is to wait indefinitely. First one 
large offending process freezes. Then other, smaller processes that 
attempt to allocate some more memory also freeze, one by one, and the 
whole system becomes unusable. I am trying to understand why this 
behavior occurs, instead of options 2 or 3? Are there any relevant 
sysctl variables? Is this a conscious choice of behavior? Is there some 
malfunction involved, and this isn't the intended behavior?


Yuri