From nobody Tue Jun 04 14:18:01 2024 X-Original-To: freebsd-arch@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Vtt3j5KP6z5LV97 for ; Tue, 04 Jun 2024 14:18:09 +0000 (UTC) (envelope-from freebsd-rwg@gndrsh.dnsmgr.net) Received: from gndrsh.dnsmgr.net (pdx.rh.CN85.dnsmgr.net [65.75.216.6]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4Vtt3h6tzqz4lhj; Tue, 4 Jun 2024 14:18:08 +0000 (UTC) (envelope-from freebsd-rwg@gndrsh.dnsmgr.net) Authentication-Results: mx1.freebsd.org; none Received: from gndrsh.dnsmgr.net (localhost [127.0.0.1]) by gndrsh.dnsmgr.net (8.13.3/8.13.3) with ESMTP id 454EI1aV011802; Tue, 4 Jun 2024 07:18:01 -0700 (PDT) (envelope-from freebsd-rwg@gndrsh.dnsmgr.net) Received: (from freebsd-rwg@localhost) by gndrsh.dnsmgr.net (8.13.3/8.13.3/Submit) id 454EI1la011801; Tue, 4 Jun 2024 07:18:01 -0700 (PDT) (envelope-from freebsd-rwg) From: "Rodney W. Grimes" Message-Id: <202406041418.454EI1la011801@gndrsh.dnsmgr.net> Subject: Re: removing support for kernel stack swapping In-Reply-To: To: Mark Johnston Date: Tue, 4 Jun 2024 07:18:01 -0700 (PDT) CC: Konstantin Belousov , freebsd-arch@FreeBSD.org X-Mailer: ELM [version 2.4ME+ PL121h (25)] List-Id: Discussion related to FreeBSD architecture List-Archive: https://lists.freebsd.org/archives/freebsd-arch List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-arch@FreeBSD.org MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=US-ASCII X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:10494, ipnet:65.75.216.0/23, country:US] X-Rspamd-Queue-Id: 4Vtt3h6tzqz4lhj > On Tue, Jun 04, 2024 at 12:11:25AM +0300, Konstantin Belousov wrote: > > On Sun, Jun 02, 2024 at 07:57:04PM -0400, Mark Johnston wrote: > > > FreeBSD will, when free pages are scarce, try to swap out the kernel > > > stacks (typically 16KB per thread) of sleeping user threads. I'm told > > > that this mechanism was first implemented in BSD for the VAX port and > > > that stabilizing it was quite an endeavour. > > > > > > This feature has wide-ranging implications for code in the kernel. For > > > instance, if a thread allocates a structure on its stack, links it into > > > some data structure visible to other threads, and goes to sleep, it must > > > use PHOLD to ensure that the stack doesn't get swapped out while > > > sleeping. A missing PHOLD can thus result in a kernel panic, but this > > > kind of mistake is very easy to make and hard to catch without thorough > > > stress testing. The kernel stack allocator also requires a fair bit of > > > code to implement this feature, and we've had multiple bugs in that > > > area, especially in relation to NUMA support. Moreover, this feature > > > will leave threads swapped out after the system has recovered, resulting > > > in high scheduling latency once they're ready to run again. > > > > > > In a very stressed system, it's possible that we can free up something > > > like 1MB of RAM using this mechanism. I argue that this mechanism is > > > not worth it on modern systems: it isn't going to make the difference > > > between a graceful recovery from memory pressure and a catatonic state > > > which forces a reboot. The complexity and resulting bugs it induces is > > > not worth it. > > On amd64, 1MB of physical memory for stacks is consumed by 64k threads, > > which is not too stressed system. I remember that very long time ago > > Peter ran tests with several hundreds of k threads, which is more realistic > > high load, e.g. from typical java code (at least it was so several years > > ago). > > Those threads are completely idle? > > > For kernel stack to be swapped, normally thread must sleep for at least > > 10 secs. so a latency for next thread running moment should be not too > > important. > > This isn't true in general. A daemon which responds to requests should > do so with low latency even if it's been idle for a long time. If > syslogd sleeps for 10 seconds and then receives a burst of messages, it > should be scheduled as quickly as possible. > > > Having 1MB of essentially free memory is nice for system survival. > > Being able to swap out pcb as well could be useful, IMO. > > There are many things we could do to shrink the kernel when under memory > pressure. There is no pressure to shrink the buffer cache, or vnode or > name caches, for instance. If we wanted to optimize the system in this > direction, there is a lot of lower-hanging fruit to pick. Yes please, better pressure on some much larger memory consumers would be greatly appreciated. > > I'm sure there are special cases where stack swapping might help in > principle, but in practice it is far more common to see a small number > of threads get swapped out, quickly followed by OOM kills. Exactly my experience too. -- Rod Grimes rgrimes@freebsd.org