Re: removing support for kernel stack swapping

> On Tue, Jun 04, 2024 at 12:11:25AM +0300, Konstantin Belousov wrote:
> > On Sun, Jun 02, 2024 at 07:57:04PM -0400, Mark Johnston wrote:
> > > FreeBSD will, when free pages are scarce, try to swap out the kernel
> > > stacks (typically 16KB per thread) of sleeping user threads.  I'm told
> > > that this mechanism was first implemented in BSD for the VAX port and
> > > that stabilizing it was quite an endeavour.
> > > 
> > > This feature has wide-ranging implications for code in the kernel.  For
> > > instance, if a thread allocates a structure on its stack, links it into
> > > some data structure visible to other threads, and goes to sleep, it must
> > > use PHOLD to ensure that the stack doesn't get swapped out while
> > > sleeping.  A missing PHOLD can thus result in a kernel panic, but this
> > > kind of mistake is very easy to make and hard to catch without thorough
> > > stress testing.  The kernel stack allocator also requires a fair bit of
> > > code to implement this feature, and we've had multiple bugs in that
> > > area, especially in relation to NUMA support.  Moreover, this feature
> > > will leave threads swapped out after the system has recovered, resulting
> > > in high scheduling latency once they're ready to run again.
> > > 
> > > In a very stressed system, it's possible that we can free up something
> > > like 1MB of RAM using this mechanism.  I argue that this mechanism is
> > > not worth it on modern systems: it isn't going to make the difference
> > > between a graceful recovery from memory pressure and a catatonic state
> > > which forces a reboot.  The complexity and resulting bugs it induces is
> > > not worth it.
> > On amd64, 1MB of physical memory for stacks is consumed by 64k threads,
> > which is not too stressed system.  I remember that very long time ago
> > Peter ran tests with several hundreds of k threads, which is more realistic
> > high load, e.g. from typical java code (at least it was so several years
> > ago).
> 
> Those threads are completely idle?
> 
> > For kernel stack to be swapped, normally thread must sleep for at least
> > 10 secs. so a latency for next thread running moment should be not too
> > important.
> 
> This isn't true in general.  A daemon which responds to requests should
> do so with low latency even if it's been idle for a long time.  If
> syslogd sleeps for 10 seconds and then receives a burst of messages, it
> should be scheduled as quickly as possible.
> 
> > Having 1MB of essentially free memory is nice for system survival.
> > Being able to swap out pcb as well could be useful, IMO.
> 
> There are many things we could do to shrink the kernel when under memory
> pressure.  There is no pressure to shrink the buffer cache, or vnode or
> name caches, for instance.  If we wanted to optimize the system in this
> direction, there is a lot of lower-hanging fruit to pick.

Yes please, better pressure on some much larger memory consumers
would be greatly appreciated.

> 
> I'm sure there are special cases where stack swapping might help in
> principle, but in practice it is far more common to see a small number
> of threads get swapped out, quickly followed by OOM kills.

Exactly my experience too.

-- 
Rod Grimes                                                 rgrimes@freebsd.org