From nobody Tue Oct 15 04:36:55 2024 X-Original-To: dev-commits-src-main@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4XSLsh1GcZz5NsSm; Tue, 15 Oct 2024 04:36:56 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R11" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4XSLsh0kMkz4dSB; Tue, 15 Oct 2024 04:36:56 +0000 (UTC) (envelope-from git@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1728967016; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=/0DUzIR+eipIksUTsIc7CzjLCqKNCV1GV6kXgs/hOA0=; b=mklHitMNTUGTjk3pqREAxMBZIEyQRJe7X5ZSEi6n6z44dYTppbwfc/N44Njhs7e5HoHLcn 0OciQyM7Q1kMTmnt4GPMV9albBexlPbJKLbxif4m6N8kq+PWgxwbH65lr5OXZQhe8qplwJ L/Mwmmqit16tswtB/N2eBL2xkHrXAVvHhmf3BQoAqEroS0chZy6voiHU8ePeL4+81di+xL E/YAeuGSfASTi2FIEWtzohGV9DgZDeqPQYk8KGfEcKTSb696EYLsAqvEYSUIKlhYYJxaR/ fJgIZse1lHP9Q4FMDUIeoN9PAPM9yfjpNauVHSRZaMntCFkRjQ0yEEYA5hG+Og== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1728967016; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=/0DUzIR+eipIksUTsIc7CzjLCqKNCV1GV6kXgs/hOA0=; b=uK4qECXGT/DpYL2NtyCH8tMlCnOzJYCFDlLaG5OwexC17B1LEI/CHOaIUjrYmCjtm6QgOO gO0Evig9ELTnG/zPhUGMgXsFoXwWvLeJ3FKbXi97sW40Ybbwh8eqphk6RWHJWmWjL9Ot/I ZpuwmwQ/31cla9caym2mm1Vh5GqWhEXSX0YCCdy1J2NpTjsHoi1g0QHCQQdVGVyL2brLhK BNvTukJgFjFLYJ9lyw9VH6spifmszg+O3gGECYyVhEivNbrf/ZdDqdobDOW1PehNLjimDk UFTQg/tZ6So6Cab1fUnjzrlno2EtkYnhG5siYnLTNt+gYjx5d/oQLBQ096gHnQ== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1728967016; a=rsa-sha256; cv=none; b=H9ULZp6eGx/88+mrMzs2x/7x21B/vy9+CfWoSlw7zlfBgGJDlFVU8/4ZvIeOom1nWLKwIT EWiiV3EyQC/1wGBIRBvngtwQTgLXsFKHrKsMxgmgF02UIfMQgc2ut6uLpLCdxuqukGUyZz Towu2WgFCjbLuLlZ9i75dtVTQx7O4zomBeTYo0x3CGtczO+f8K+IpUHOfwRNUMvjf32omV Vau0lzIBBrMR9NTP4rJFHAMEc0szcpbwUqW749fJ7WD6Qem4Bol48RG2PymshhRARhN/j7 EhezWfdRFMOmxYJ2MO4bh2q3K4skUpfKUI1EfmLpghTl/7JPtjmWhn7RL+oKZQ== Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4XSLsh0KGVz178x; Tue, 15 Oct 2024 04:36:56 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org ([127.0.1.44]) by gitrepo.freebsd.org (8.18.1/8.18.1) with ESMTP id 49F4atIO018703; Tue, 15 Oct 2024 04:36:55 GMT (envelope-from git@gitrepo.freebsd.org) Received: (from git@localhost) by gitrepo.freebsd.org (8.18.1/8.18.1/Submit) id 49F4atSu018700; Tue, 15 Oct 2024 04:36:55 GMT (envelope-from git) Date: Tue, 15 Oct 2024 04:36:55 GMT Message-Id: <202410150436.49F4atSu018700@gitrepo.freebsd.org> To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-main@FreeBSD.org From: Doug Moore Subject: git: 4a983f05d98b - main - vm_page: move tailq validation to grab_pages List-Id: Commit messages for the main branch of the src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-main List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: dev-commits-src-main@freebsd.org Sender: owner-dev-commits-src-main@FreeBSD.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: dougm X-Git-Repository: src X-Git-Refname: refs/heads/main X-Git-Reftype: branch X-Git-Commit: 4a983f05d98ba1c5eebe65d29dd061dc3eab7594 Auto-Submitted: auto-generated The branch main has been updated by dougm: URL: https://cgit.FreeBSD.org/src/commit/?id=4a983f05d98ba1c5eebe65d29dd061dc3eab7594 commit 4a983f05d98ba1c5eebe65d29dd061dc3eab7594 Author: Doug Moore AuthorDate: 2024-10-15 04:33:43 +0000 Commit: Doug Moore CommitDate: 2024-10-15 04:33:43 +0000 vm_page: move tailq validation to grab_pages Function vm_page_acquire_unlocked both looks up pages and validates them. Much of it serves the needs of only one caller, vm_page_grab_pages_unlocked, by checking the validity of checking tailq links. Extract from that function the parts that serve only vm_page_grab_pages_unlocked, and move them into that function. Reviewed by: kib Differential Revision: https://reviews.freebsd.org/D47001 --- sys/vm/vm_page.c | 111 ++++++++++++++++++++++++++----------------------------- 1 file changed, 53 insertions(+), 58 deletions(-) diff --git a/sys/vm/vm_page.c b/sys/vm/vm_page.c index 1f0b364dbde2..f2b3baf419a0 100644 --- a/sys/vm/vm_page.c +++ b/sys/vm/vm_page.c @@ -1774,7 +1774,7 @@ vm_page_relookup(vm_object_t object, vm_pindex_t pindex) { vm_page_t m; - m = vm_radix_lookup_unlocked(&object->rtree, pindex); + m = vm_page_lookup_unlocked(object, pindex); KASSERT(m != NULL && (vm_page_busied(m) || vm_page_wired(m)) && m->object == object && m->pindex == pindex, ("vm_page_relookup: Invalid page %p", m)); @@ -4795,53 +4795,33 @@ out: } /* - * Locklessly attempt to acquire a page given a (object, pindex) tuple - * and an optional previous page to avoid the radix lookup. The resulting - * page will be validated against the identity tuple and busied or wired - * as requested. A NULL *mp return guarantees that the page was not in - * radix at the time of the call but callers must perform higher level - * synchronization or retry the operation under a lock if they require - * an atomic answer. This is the only lock free validation routine, - * other routines can depend on the resulting page state. - * - * The return value indicates whether the operation failed due to caller - * flags. The return is tri-state with mp: - * - * (true, *mp != NULL) - The operation was successful. - * (true, *mp == NULL) - The page was not found in tree. - * (false, *mp == NULL) - WAITFAIL or NOWAIT prevented acquisition. + * Attempt to validate a page, locklessly acquiring it if necessary, given a + * (object, pindex) tuple and either an invalided page or NULL. The resulting + * page will be validated against the identity tuple, and busied or wired as + * requested. A NULL page returned guarantees that the page was not in radix at + * the time of the call but callers must perform higher level synchronization or + * retry the operation under a lock if they require an atomic answer. This is + * the only lock free validation routine, other routines can depend on the + * resulting page state. + * + * The return value PAGE_NOT_ACQUIRED indicates that the operation failed due to + * caller flags. */ -static bool -vm_page_acquire_unlocked(vm_object_t object, vm_pindex_t pindex, - vm_page_t prev, vm_page_t *mp, int allocflags) +#define PAGE_NOT_ACQUIRED ((vm_page_t)1) +static vm_page_t +vm_page_acquire_unlocked(vm_object_t object, vm_pindex_t pindex, vm_page_t m, + int allocflags) { - vm_page_t m; - - vm_page_grab_check(allocflags); - MPASS(prev == NULL || vm_page_busied(prev) || vm_page_wired(prev)); - - *mp = NULL; - for (;;) { - /* - * We may see a false NULL here because the previous page - * has been removed or just inserted and the list is loaded - * without barriers. Switch to radix to verify. - */ - if (prev == NULL || (m = TAILQ_NEXT(prev, listq)) == NULL || - QMD_IS_TRASHED(m) || m->pindex != pindex || - atomic_load_ptr(&m->object) != object) { - prev = NULL; - /* - * This guarantees the result is instantaneously - * correct. - */ - m = vm_radix_lookup_unlocked(&object->rtree, pindex); - } - if (m == NULL) - return (true); + if (m == NULL) + m = vm_page_lookup_unlocked(object, pindex); + for (; m != NULL; m = vm_page_lookup_unlocked(object, pindex)) { if (vm_page_trybusy(m, allocflags)) { - if (m->object == object && m->pindex == pindex) + if (m->object == object && m->pindex == pindex) { + if ((allocflags & VM_ALLOC_WIRED) != 0) + vm_page_wire(m); + vm_page_grab_release(m, allocflags); break; + } /* relookup. */ vm_page_busy_release(m); cpu_spinwait(); @@ -4849,13 +4829,9 @@ vm_page_acquire_unlocked(vm_object_t object, vm_pindex_t pindex, } if (!vm_page_grab_sleep(object, m, pindex, "pgnslp", allocflags, false)) - return (false); + return (PAGE_NOT_ACQUIRED); } - if ((allocflags & VM_ALLOC_WIRED) != 0) - vm_page_wire(m); - vm_page_grab_release(m, allocflags); - *mp = m; - return (true); + return (m); } /* @@ -4868,8 +4844,8 @@ vm_page_grab_unlocked(vm_object_t object, vm_pindex_t pindex, int allocflags) vm_page_t m; vm_page_grab_check(allocflags); - - if (!vm_page_acquire_unlocked(object, pindex, NULL, &m, allocflags)) + m = vm_page_acquire_unlocked(object, pindex, NULL, allocflags); + if (m == PAGE_NOT_ACQUIRED) return (NULL); if (m != NULL) return (m); @@ -5028,13 +5004,16 @@ vm_page_grab_valid_unlocked(vm_page_t *mp, vm_object_t object, * before we can inspect the valid field and return a wired page. */ flags = allocflags & ~(VM_ALLOC_NOBUSY | VM_ALLOC_WIRED); - if (!vm_page_acquire_unlocked(object, pindex, NULL, mp, flags)) + vm_page_grab_check(flags); + m = vm_page_acquire_unlocked(object, pindex, NULL, flags); + if (m == PAGE_NOT_ACQUIRED) return (VM_PAGER_FAIL); - if ((m = *mp) != NULL) { + if (m != NULL) { if (vm_page_all_valid(m)) { if ((allocflags & VM_ALLOC_WIRED) != 0) vm_page_wire(m); vm_page_grab_release(m, allocflags); + *mp = m; return (VM_PAGER_OK); } vm_page_busy_release(m); @@ -5141,7 +5120,7 @@ int vm_page_grab_pages_unlocked(vm_object_t object, vm_pindex_t pindex, int allocflags, vm_page_t *ma, int count) { - vm_page_t m, pred; + vm_page_t m; int flags; int i; @@ -5154,9 +5133,24 @@ vm_page_grab_pages_unlocked(vm_object_t object, vm_pindex_t pindex, * set it valid if necessary. */ flags = allocflags & ~VM_ALLOC_NOBUSY; - pred = NULL; + vm_page_grab_check(flags); + m = NULL; for (i = 0; i < count; i++, pindex++) { - if (!vm_page_acquire_unlocked(object, pindex, pred, &m, flags)) + /* + * We may see a false NULL here because the previous page has + * been removed or just inserted and the list is loaded without + * barriers. Switch to radix to verify. + */ + if (m == NULL || QMD_IS_TRASHED(m) || m->pindex != pindex || + atomic_load_ptr(&m->object) != object) { + /* + * This guarantees the result is instantaneously + * correct. + */ + m = NULL; + } + m = vm_page_acquire_unlocked(object, pindex, m, flags); + if (m == PAGE_NOT_ACQUIRED) return (i); if (m == NULL) break; @@ -5167,7 +5161,8 @@ vm_page_grab_pages_unlocked(vm_object_t object, vm_pindex_t pindex, } /* m will still be wired or busy according to flags. */ vm_page_grab_release(m, allocflags); - pred = ma[i] = m; + ma[i] = m; + m = TAILQ_NEXT(m, listq); } if (i == count || (allocflags & VM_ALLOC_NOCREAT) != 0) return (i);