From nobody Tue Jul 30 15:38:46 2024 X-Original-To: dev-commits-src-main@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4WYKBv31Rqz5RRBF; Tue, 30 Jul 2024 15:38:47 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R11" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4WYKBv0ZVnz42JC; Tue, 30 Jul 2024 15:38:47 +0000 (UTC) (envelope-from git@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1722353927; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=iV6H+1LgzN8uf7PwETLHkQJ2XrsyTA7jmtFpty+1NXc=; b=CMPoD7IfL/8EWzKYMRnkTFTvcLiSgWLU8hp3COBRBMr9tZLmRJ44ZcQOZMoLysczc6Gb9E /TIngMnIHT0qJCJ55B42kdaz4JrZuVmd34P0x/yk3IDeC6KgYsidIm0wQf/cPiIGJOPvOU t2ekvKDKnXjTARcWbXCbLAx6cw0Xx6VUOZkngYFGN4HSiQDrSkNtgEih5kpN3mG3vKAzts sgBzaB+d+Epx45BK64W48YBbQD9k+LAihPf+IVUvcg0WJwlGUnFNrLDwdcm+JGSR+gTGV0 hJNkfBmsniXmrpVOztigr+8iFaaKUcaEKyqwzTgwJ1u7yQDxJD5j3aaENzgrIQ== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1722353927; a=rsa-sha256; cv=none; b=S3VSrbYY3EACbF161NSCfPRZkPbWKqT+MbY+1f4XiTRgy1czqE0YzwK9Vrw0uEeghbbaoZ 1rXbJ+r0fkKvKK3zPV10MT4WNyFpHVcbXpByK6kl7v9/hPO/WgB4ZE3i9iiNjIjGROtAyB kRLmmj5POlHhsXN+WtMaf8RXaW9St8l0/lyobXZP5q/fT4JePrX5htZsyAEUULhIRMSYDC mhqz4S/pIZD67nv91c/+xs3GC0igODAtwigFCpKQxGEkUAYuV+B1QOOdonDlnzA9l5h2GN uInEiWySiOcGXIetCu5z012GGBod8X1DZrUE3pfkJP3UUrnjNt+BOdUlttCljw== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1722353927; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=iV6H+1LgzN8uf7PwETLHkQJ2XrsyTA7jmtFpty+1NXc=; b=THoMMU0YSZHy2PD4vZJizjqxcd2qLfy65pZ+fZWTmurlWxTlvXFo7Qemcwa8aAa3D2wkSn OzbCPeGnh13Lc5b+5G/Xy034VQcmic5CfTzVeqtMwzrhVGj4KpGa7JT3FRiaFUKf4mhmjs XDaWEQLGmPe30RfYQ7sqM8PEUZeIXJDXXvGtdY9IpYFvNH3GhP8lNgk7CVg/Pb0im/EwVX PiR93wIJ3x/kiVeeo1qApNQxelJpaLrEkX2zD9Ob3uqQ1WnfmVFZ/K65jANtTMVIP5R082 Iy2nm05Qhvbom1lvzPlT+z1BCRxF4Jd7+yDx7JUpx7WOQ9bx5GY7EfcaK13qtA== Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4WYKBv0BVCzNWS; Tue, 30 Jul 2024 15:38:47 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org ([127.0.1.44]) by gitrepo.freebsd.org (8.18.1/8.18.1) with ESMTP id 46UFck2W049592; Tue, 30 Jul 2024 15:38:46 GMT (envelope-from git@gitrepo.freebsd.org) Received: (from git@localhost) by gitrepo.freebsd.org (8.18.1/8.18.1/Submit) id 46UFckFQ049589; Tue, 30 Jul 2024 15:38:46 GMT (envelope-from git) Date: Tue, 30 Jul 2024 15:38:46 GMT Message-Id: <202407301538.46UFckFQ049589@gitrepo.freebsd.org> To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-main@FreeBSD.org From: Bojan =?utf-8?Q?Novkovi=C4=87?= Subject: git: a8693e89e3e4 - main - vm: Introduce vm_page_alloc_nofree_domain List-Id: Commit messages for the main branch of the src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-main List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: dev-commits-src-main@freebsd.org Sender: owner-dev-commits-src-main@FreeBSD.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: bnovkov X-Git-Repository: src X-Git-Refname: refs/heads/main X-Git-Reftype: branch X-Git-Commit: a8693e89e3e4a04efd02901cc93bb6148e3e40d6 Auto-Submitted: auto-generated The branch main has been updated by bnovkov: URL: https://cgit.FreeBSD.org/src/commit/?id=a8693e89e3e4a04efd02901cc93bb6148e3e40d6 commit a8693e89e3e4a04efd02901cc93bb6148e3e40d6 Author: Bojan Novković AuthorDate: 2024-07-14 13:14:22 +0000 Commit: Bojan Novković CommitDate: 2024-07-30 15:38:24 +0000 vm: Introduce vm_page_alloc_nofree_domain This patch adds a reservation-aware bump allocator intended for allocating NOFREE pages. The main goal of this change is to reduce the long-term fragmentation issues caused by pages that are never freed during runtime. The `vm_page_alloc_nofree_domain` routine hands out 0-order pages from a preallocated superpage. Once an active NOFREE superpage fills up, the routine will try to allocate a new one and discard the old one. This routine will get invoked whenever VM_ALLOC_NOFREE is passed to vm_page_alloc_noobj or vm_page_alloc. Differential Revision: https://reviews.freebsd.org/D45863 Reviewed by: alc, kib, markj Tested by: alc --- sys/vm/vm_page.c | 62 +++++++++++++++++++++++++++++++++++++++++++++++++++ sys/vm/vm_pagequeue.h | 4 ++++ 2 files changed, 66 insertions(+) diff --git a/sys/vm/vm_page.c b/sys/vm/vm_page.c index 3b6b88e4eb32..ff9df7f4a9fc 100644 --- a/sys/vm/vm_page.c +++ b/sys/vm/vm_page.c @@ -163,6 +163,7 @@ SYSCTL_PROC(_vm, OID_AUTO, page_blacklist, CTLTYPE_STRING | CTLFLAG_RD | static uma_zone_t fakepg_zone; static void vm_page_alloc_check(vm_page_t m); +static vm_page_t vm_page_alloc_nofree_domain(int domain, int req); static bool _vm_page_busy_sleep(vm_object_t obj, vm_page_t m, vm_pindex_t pindex, const char *wmesg, int allocflags, bool locked); static void vm_page_clear_dirty_mask(vm_page_t m, vm_page_bits_t pagebits); @@ -2099,6 +2100,11 @@ vm_page_alloc_domain_after(vm_object_t object, vm_pindex_t pindex, int domain, if (!vm_pager_can_alloc_page(object, pindex)) return (NULL); again: + if (__predict_false((req & VM_ALLOC_NOFREE) != 0)) { + m = vm_page_alloc_nofree_domain(domain, req); + if (m != NULL) + goto found; + } #if VM_NRESERVLEVEL > 0 /* * Can we allocate the page from a reservation? @@ -2430,6 +2436,12 @@ vm_page_alloc_noobj_domain(int domain, int req) ((req & VM_ALLOC_NOFREE) != 0 ? PG_NOFREE : 0); vmd = VM_DOMAIN(domain); again: + if (__predict_false((req & VM_ALLOC_NOFREE) != 0)) { + m = vm_page_alloc_nofree_domain(domain, req); + if (m != NULL) + goto found; + } + if (vmd->vmd_pgcache[VM_FREEPOOL_DIRECT].zone != NULL) { m = uma_zalloc(vmd->vmd_pgcache[VM_FREEPOOL_DIRECT].zone, M_NOWAIT | M_NOVM); @@ -2480,6 +2492,56 @@ found: return (m); } +#if VM_NRESERVLEVEL > 1 +#define VM_NOFREE_IMPORT_ORDER (VM_LEVEL_1_ORDER + VM_LEVEL_0_ORDER) +#elif VM_NRESERVLEVEL > 0 +#define VM_NOFREE_IMPORT_ORDER VM_LEVEL_0_ORDER +#else +#define VM_NOFREE_IMPORT_ORDER 8 +#endif + +/* + * Allocate a single NOFREE page. + * + * This routine hands out NOFREE pages from higher-order + * physical memory blocks in order to reduce memory fragmentation. + * When a NOFREE for a given domain chunk is used up, + * the routine will try to fetch a new one from the freelists + * and discard the old one. + */ +static vm_page_t +vm_page_alloc_nofree_domain(int domain, int req) +{ + vm_page_t m; + struct vm_domain *vmd; + struct vm_nofreeq *nqp; + + KASSERT((req & VM_ALLOC_NOFREE) != 0, ("invalid request %#x", req)); + + vmd = VM_DOMAIN(domain); + nqp = &vmd->vmd_nofreeq; + vm_domain_free_lock(vmd); + if (nqp->offs >= (1 << VM_NOFREE_IMPORT_ORDER) || nqp->ma == NULL) { + if (!vm_domain_allocate(vmd, req, + 1 << VM_NOFREE_IMPORT_ORDER)) { + vm_domain_free_unlock(vmd); + return (NULL); + } + nqp->ma = vm_phys_alloc_pages(domain, VM_FREEPOOL_DEFAULT, + VM_LEVEL_0_ORDER); + if (nqp->ma == NULL) { + vm_domain_freecnt_inc(vmd, 1 << VM_NOFREE_IMPORT_ORDER); + vm_domain_free_unlock(vmd); + return (NULL); + } + nqp->offs = 0; + } + m = &nqp->ma[nqp->offs++]; + vm_domain_free_unlock(vmd); + + return (m); +} + vm_page_t vm_page_alloc_noobj(int req) { diff --git a/sys/vm/vm_pagequeue.h b/sys/vm/vm_pagequeue.h index 7e133ec947b4..86863a0a6400 100644 --- a/sys/vm/vm_pagequeue.h +++ b/sys/vm/vm_pagequeue.h @@ -246,6 +246,10 @@ struct vm_domain { u_int vmd_domain; /* (c) Domain number. */ u_int vmd_page_count; /* (c) Total page count. */ long vmd_segs; /* (c) bitmask of the segments */ + struct vm_nofreeq { + vm_page_t ma; + int offs; + } vmd_nofreeq; /* (f) NOFREE page bump allocator. */ u_int __aligned(CACHE_LINE_SIZE) vmd_free_count; /* (a,f) free page count */ u_int vmd_pageout_deficit; /* (a) Estimated number of pages deficit */ uint8_t vmd_pad[CACHE_LINE_SIZE - (sizeof(u_int) * 2)];