From nobody Mon Jun 03 09:30:29 2024 X-Original-To: freebsd-arch@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Vt7lF4FBqz5MBkp for ; Mon, 03 Jun 2024 09:31:21 +0000 (UTC) (envelope-from br@bsdpad.com) Received: from mail.bsdpad.com (mail.bsdpad.com [116.202.106.248]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4Vt7lF0y81z4JkD; Mon, 3 Jun 2024 09:31:21 +0000 (UTC) (envelope-from br@bsdpad.com) Authentication-Results: mx1.freebsd.org; none DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=bsdpad.com; s=20201212; h=Subject:To:From; bh=sXBRZ2kLNcxhLNgYVH34h102S7zGIXg9gTODi9JxsAY=; b=ECKQ3NGEgF6UGZ7seoyPCL1ZtQ UUkhzOFru4eKSY1eLdeIFHYjgIBKDMcBKxCyzA5h+eRJ57Xu5uWtc4x3SWxYycPg8xL7h31Yr2J9m P1ICkChttCse2BFryXNG0o9PvoF6V7tviGJWcIU5xykXxUqayC2eogW2+lb00zke5kXfxXr1l3kut T0TyNuQFv5YKkr8yZngDCEouwk3oOYQHqDBZJErWnjQktULR7yHEY8d3XLhmrGzGlqy4hZuIyDyb8 NMLbm1jz9n8tZApd2YFFAtNI2GC6wHHtPfGkmFsV26g6CoEt8EA6kyHYj9jhrwy9+7GxY226JCV95 Gv56edYg==; Received: from localhost ([127.0.0.1] helo=bsdpad.com) by mail.bsdpad.com with smtp (Exim 4.94 (FreeBSD)) (envelope-from ) id 1sE41O-000Axi-0R; Mon, 03 Jun 2024 10:30:30 +0100 Received: by bsdpad.com (nbSMTP-1.00) for uid 1001 br@bsdpad.com; Mon, 3 Jun 2024 10:30:30 +0100 (BST) Date: Mon, 3 Jun 2024 10:30:29 +0100 From: Ruslan Bukin To: Mark Johnston Cc: Warner Losh , "freebsd-arch@freebsd.org" Subject: Re: removing support for kernel stack swapping Message-ID: References: List-Id: Discussion related to FreeBSD architecture List-Archive: https://lists.freebsd.org/archives/freebsd-arch List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-arch@FreeBSD.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:24940, ipnet:116.202.0.0/16, country:DE] X-Rspamd-Queue-Id: 4Vt7lF0y81z4JkD On Sun, Jun 02, 2024 at 08:05:06PM -0400, Warner Losh wrote: > On Sun, Jun 2, 2024, 5:57 PM Mark Johnston wrote: > > > FreeBSD will, when free pages are scarce, try to swap out the kernel > > stacks (typically 16KB per thread) of sleeping user threads. I'm told > > that this mechanism was first implemented in BSD for the VAX port and > > that stabilizing it was quite an endeavour. > > > > This feature has wide-ranging implications for code in the kernel. For > > instance, if a thread allocates a structure on its stack, links it into > > some data structure visible to other threads, and goes to sleep, it must > > use PHOLD to ensure that the stack doesn't get swapped out while > > sleeping. A missing PHOLD can thus result in a kernel panic, but this > > kind of mistake is very easy to make and hard to catch without thorough > > stress testing. The kernel stack allocator also requires a fair bit of > > code to implement this feature, and we've had multiple bugs in that > > area, especially in relation to NUMA support. Moreover, this feature > > will leave threads swapped out after the system has recovered, resulting > > in high scheduling latency once they're ready to run again. > > > > In a very stressed system, it's possible that we can free up something > > like 1MB of RAM using this mechanism. I argue that this mechanism is > > not worth it on modern systems: it isn't going to make the difference > > between a graceful recovery from memory pressure and a catatonic state > > which forces a reboot. The complexity and resulting bugs it induces is > > not worth it. > > > > > +1. > > The smallest bootable system for me is like 256MB, and in a system like > that it might save 256k given the number of threads typical in a system > like that... > > Warner > I managed to boot on 10mb of on-chip static RAM (no DDR at all), including a few mb of mdroot. But now mostly using DDR2/3 which is no way to get less that 32mb, so 1mb is not a problem at all. Ruslan