From nobody Wed Feb 02 00:38:48 2022 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id B7D9B198C4D5 for ; Wed, 2 Feb 2022 00:41:30 +0000 (UTC) (envelope-from ota@j.email.ne.jp) Received: from mail07.asahi-net.or.jp (mail07.asahi-net.or.jp [202.224.55.47]) by mx1.freebsd.org (Postfix) with ESMTP id 4JpNK40VNxz55jT; Wed, 2 Feb 2022 00:41:28 +0000 (UTC) (envelope-from ota@j.email.ne.jp) Received: from vmware13.advok.com (pool-96-225-64-148.nwrknj.fios.verizon.net [96.225.64.148]) (Authenticated sender: NR2Y-OOT) by mail07.asahi-net.or.jp (Postfix) with ESMTPSA id 08873C1F2D; Wed, 2 Feb 2022 09:41:18 +0900 (JST) Date: Tue, 1 Feb 2022 19:38:48 -0500 From: Yoshihiro Ota To: Ambert Cc: Peter Jeremy , "freebsd-hackers@freebsd.org" Subject: Re: Out-of-swap killer and SIGTERM signal Message-Id: <20220201193848.71043b06041aa90b541f8c13@j.email.ne.jp> In-Reply-To: References: X-Mailer: Sylpheed 3.7.0 (GTK+ 2.24.33; i386-portbld-freebsd13.0) List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 4JpNK40VNxz55jT X-Spamd-Bar: / Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of ota@j.email.ne.jp designates 202.224.55.47 as permitted sender) smtp.mailfrom=ota@j.email.ne.jp X-Spamd-Result: default: False [-0.91 / 15.00]; TO_DN_EQ_ADDR_SOME(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; RECEIVED_SPAMHAUS_PBL(0.00)[96.225.64.148:received]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; MV_CASE(0.50)[]; R_SPF_ALLOW(-0.20)[+ip4:202.224.55.0/24]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[email.ne.jp]; ARC_NA(0.00)[]; TO_DN_SOME(0.00)[]; NEURAL_SPAM_MEDIUM(0.75)[0.753]; NEURAL_HAM_LONG(-0.96)[-0.964]; TO_MATCH_ENVRCPT_SOME(0.00)[]; NEURAL_HAM_SHORT(-1.00)[-1.000]; MLMMJ_DEST(0.00)[freebsd-hackers]; FREEMAIL_TO(0.00)[protonmail.com]; RCVD_NO_TLS_LAST(0.10)[]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:4685, ipnet:202.224.32.0/19, country:JP]; RCVD_COUNT_TWO(0.00)[2]; MID_RHS_MATCH_FROM(0.00)[]; RWL_MAILSPIKE_POSSIBLE(0.00)[202.224.55.47:from] X-ThisMailContainsUnwantedMimeParts: N Hi, Ambert. On Wed, 12 Jan 2022 21:42:18 +0000 Ambert wrote: > On 2022-01-12, Peter Jeremy wrote: > > There have been lots of discussions about this in the past, starting > > in about 1998, (though I agree that it's been about 4 years since the > > last discussion). I suggest you search for "freebsd+sigdanger" for > > previous discussions. > > Thank you for the keyword. After a search, I find that the previous > threads containing the keyword "sigdanger" discuss extensively two > subjects: > 1) How to exclude a process from the reach of the OOM killer. And > how to ask the OOM killer to kill a given process first. > 2) How to provide feedback about memory usage to processes, to give > them a hint that they should reduce their memory footprint, if > possible. This feedback can be a signal sent to all processes, or > system-wide flags readable by any process who cares about a > potential memory shortage. > > Those two subjects are interesting, but I am talking about something > else. My suggestion is almost never mentionned in previous threads. > When it is mentionned [1] [2], there is no objection whatsoever, from > anyone. There is not even a comment about it. > > Sending SIGTERM a few seconds before SIGKILL is useful because it > allows a condemned process to exit gracefully, just like it would > during a shutdown(8). And it is simple to implement. There is no need > to change the algorithm selecting condemned processes, and there is no > need to change a single line of code in userland. For the > administrator, only two tasks require extra work: > - set up a little bit of extra swap during the installation of > FreeBSD > - set a couple of sysctl values: the duration of the grace period > (vm.grace_period = zero milliseconds by default), and the amount of > extra swap that will not be usable normally (vm.grace_space = zero > bytes by default) Based on my investigations in the past, delivery of signal takes significantly long time in heavily slashing system such that OOM initiate multiple iteration of shooting other processes. In other words, SIGTERM may not reach to processes or also processes may not get enough resources or time to shutdown cleanly. Throughput of random I/O handling of swap devise impacts this a lot. If you use SSD instead of HDD for swap devise, situation may be easier. > Excerpt from a historical thread: > On 1998-04-27, Jordan K. Hubbard wrote: > > All the SIGDANGER (Will Robinson) signal is meant to do is give a > > process a little _warning_ before it's chosen as the designated > > sacrifice for the evening and terminated in an untimely fashion. > > > > I don't think the question here is "is this a good idea" - it's a > > perfectly reasonable idea and one which has been proposed before. > > The question here is really "what are the proposed semantics of > > this mechanism?", e.g. how long do you wait from the time you > > SIGDANGER the process and actually shoot it down, and what > > happens if you're also critically short of resources and don't > > have much time to wait? > > [1] > https://docs.freebsd.org/cgi/getmsg.cgi?fetch=209768+0+archive/1998/freebsd-hackers/19980426.freebsd-hackers > > [2] > https://lists.freebsd.org/pipermail/freebsd-current/2008-January/081743.html > > > Ambert I'm also interested in system monitoring and auto-turning based on sysctl and started below. This is for swap space monitoring: https://github.com/uyota/py-prdanlz/blob/main/examples/swap_usage.md Installation: https://github.com/uyota/py-prdanlz#how-to-install You may be interested in. Hiro