Re: removing support for kernel stack swapping

On Mon, Jun 03, 2024 at 10:15:15PM +0100, Jessica Clarke wrote:
> On 3 Jun 2024, at 22:11, Konstantin Belousov <kostikbel@gmail.com> wrote:
> > 
> > On Sun, Jun 02, 2024 at 07:57:04PM -0400, Mark Johnston wrote:
> >> FreeBSD will, when free pages are scarce, try to swap out the kernel
> >> stacks (typically 16KB per thread) of sleeping user threads.  I'm told
> >> that this mechanism was first implemented in BSD for the VAX port and
> >> that stabilizing it was quite an endeavour.
> >> 
> >> This feature has wide-ranging implications for code in the kernel.  For
> >> instance, if a thread allocates a structure on its stack, links it into
> >> some data structure visible to other threads, and goes to sleep, it must
> >> use PHOLD to ensure that the stack doesn't get swapped out while
> >> sleeping.  A missing PHOLD can thus result in a kernel panic, but this
> >> kind of mistake is very easy to make and hard to catch without thorough
> >> stress testing.  The kernel stack allocator also requires a fair bit of
> >> code to implement this feature, and we've had multiple bugs in that
> >> area, especially in relation to NUMA support.  Moreover, this feature
> >> will leave threads swapped out after the system has recovered, resulting
> >> in high scheduling latency once they're ready to run again.
> >> 
> >> In a very stressed system, it's possible that we can free up something
> >> like 1MB of RAM using this mechanism.  I argue that this mechanism is
> >> not worth it on modern systems: it isn't going to make the difference
> >> between a graceful recovery from memory pressure and a catatonic state
> >> which forces a reboot.  The complexity and resulting bugs it induces is
> >> not worth it.
> > On amd64, 1MB of physical memory for stacks is consumed by 64k threads,
> 
> To avoid any confusion, you mean 64 kthreads here, right? At least that
> makes sense for the story and the maths.
I mean 65535 threads (each of which must have kernel stack).

> 
> Jess
> 
> > which is not too stressed system.  I remember that very long time ago
> > Peter ran tests with several hundreds of k threads, which is more realistic
> > high load, e.g. from typical java code (at least it was so several years
> > ago).
> > 
> > For kernel stack to be swapped, normally thread must sleep for at least
> > 10 secs. so a latency for next thread running moment should be not too
> > important.
> > 
> > Having 1MB of essentially free memory is nice for system survival.
> > Being able to swap out pcb as well could be useful, IMO.
> > 
> >> 
> >> At the BSDCan devsummit I proposed removing support for kernel stack
> >> swapping and got only positive feedback.  Does anyone here have any
> >> comments or objections?
> >