Re: Out-of-swap killer and SIGTERM signal

Hi, Ambert.

On Wed, 12 Jan 2022 21:42:18 +0000
Ambert <ralph41096@protonmail.com> wrote:

> On 2022-01-12, Peter Jeremy <peterj_at_freebsd.org> wrote:
> > There have been lots of discussions about this in the past, starting
> > in about 1998, (though I agree that it's been about 4 years since the
> > last discussion).  I suggest you search for "freebsd+sigdanger" for
> > previous discussions.
> 
> Thank you for the keyword.  After a search, I find that the previous
> threads containing the keyword "sigdanger" discuss extensively two
> subjects:
>   1) How to exclude a process from the reach of the OOM killer.  And
>      how to ask the OOM killer to kill a given process first.
>   2) How to provide feedback about memory usage to processes, to give
>      them a hint that they should reduce their memory footprint, if
>      possible.  This feedback can be a signal sent to all processes, or
>      system-wide flags readable by any process who cares about a
>      potential memory shortage.
> 
> Those two subjects are interesting, but I am talking about something
> else.  My suggestion is almost never mentionned in previous threads.
> When it is mentionned [1] [2], there is no objection whatsoever, from
> anyone.  There is not even a comment about it.
> 
> Sending SIGTERM a few seconds before SIGKILL is useful because it
> allows a condemned process to exit gracefully, just like it would
> during a shutdown(8).  And it is simple to implement.  There is no need
> to change the algorithm selecting condemned processes, and there is no
> need to change a single line of code in userland.  For the
> administrator, only two tasks require extra work:
>  - set up a little bit of extra swap during the installation of
>    FreeBSD
>  - set a couple of sysctl values: the duration of the grace period
>    (vm.grace_period = zero milliseconds by default), and the amount of
>    extra swap that will not be usable normally (vm.grace_space = zero
>    bytes by default)


Based on my investigations in the past, delivery of signal takes
significantly long time in heavily slashing system such that
OOM initiate multiple iteration of shooting other processes.

In other words, SIGTERM may not reach to processes or also processes
may not get enough resources or time to shutdown cleanly.

Throughput of random I/O handling of swap devise impacts this a lot.
If you use SSD instead of HDD for swap devise, situation may be easier.

> Excerpt from a historical thread:
>     On 1998-04-27, Jordan K. Hubbard <jkh_at_time.cdrom.com> wrote:
>     > All the SIGDANGER (Will Robinson) signal is meant to do is give a
>     > process a little _warning_ before it's chosen as the designated
>     > sacrifice for the evening and terminated in an untimely fashion.
>     >
>     > I don't think the question here is "is this a good idea" - it's a
>     > perfectly reasonable idea and one which has been proposed before.
>     > The question here is really "what are the proposed semantics of
>     > this mechanism?", e.g. how long do you wait from the time you
>     > SIGDANGER the process and actually shoot it down, and what
>     > happens if you're also critically short of resources and don't
>     > have much time to wait?
> 
> [1]
> https://docs.freebsd.org/cgi/getmsg.cgi?fetch=209768+0+archive/1998/freebsd-hackers/19980426.freebsd-hackers
> 
> [2]
> https://lists.freebsd.org/pipermail/freebsd-current/2008-January/081743.html
> 
> 
> Ambert

I'm also interested in system monitoring and auto-turning based on sysctl and started below.
This is for swap space monitoring:
https://github.com/uyota/py-prdanlz/blob/main/examples/swap_usage.md

Installation:
https://github.com/uyota/py-prdanlz#how-to-install

You may be interested in.

Hiro