From nobody Thu Jun 06 17:35:47 2024 X-Original-To: dev-commits-src-main@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4VwBLq3RRwz5NMN1; Thu, 06 Jun 2024 17:35:47 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4VwBLq2sf3z4P67; Thu, 6 Jun 2024 17:35:47 +0000 (UTC) (envelope-from git@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1717695347; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=JTl61AtsyO/LbRP0eHZb+l2aHongmz0+HZkAn6AaGzc=; b=MNQtQYp2g5Scnss8e2QvbkzOJ8LC58+Lw+j+J8UlqWZlX0gpQScLME2S00tVG6lLvy58e1 JU+o6f+W25CE+Wln+ZtyDqXPG0aXlFlTuPcn2xvqMf07S6F4ygrQLqPtAu69BBjTulcbci ZxVdlWXG/EGZab77IGVG3sqsvEF2K+NhSbqTZl71CvgTMyATxxlWla0hFXmAAr/5kg9amn JtNld3ZosvDzR5cbJFADfknZU9EeAE1DIs5dZ7JcOIyrB+HhM2jBepu1+Lo2T/UWwEAkih NtOttLG5LCgel29VvYy14vGYoPPQ8V/By7PJuhkV62xajfeVKJd41Gj/yfkXGQ== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1717695347; a=rsa-sha256; cv=none; b=W7cREqap6qpINQyM/69QsFKDvGqDnM7/PHYIGTtcfSj0PP5KmOWpoFledxSjDJqUk12sNB 6Opvt+F+x/PmPZzjLd4pbb6hMRRB8Xk2slOGVknlc+DF3LT8y88sYOo5ieXQYDFefoJ+Vb 8kYaRvmlChyRvDGrAEKXsz0D5XIFtC/vyvWUqvdqPTLWPl/pd/O73SCfrG5zNe8/eN773y Ttyb8+IxN/b/mqfbARR2tpTKXVP54DjIneqcEV7UW7gI5lgBhSoCGyhqEGTPwpMKEgdi3P ycLaJ1ICY6C2e/Te44MYmRRoO1K+nTrJhhqHxlR8F/2n493aoInysXwxHKnFVg== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1717695347; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=JTl61AtsyO/LbRP0eHZb+l2aHongmz0+HZkAn6AaGzc=; b=Rw08ZilF7ZHE97kAZ4P0FRFibD+VgH48d32HW3bo4YPnWfMhF1zbe6geo69yIhGAV5CuK7 lN5DMlYCnMMAqJrEBeqrGgH1Z8lkb+bEEv+a0e+VBA7L+x0imSBit4r35Xfjk6/n2TostZ E7DtbJZfZ+1lrGu772q7vvJ4jU2KTORcp1UF7rbfZ1x2xN+xtmUOlnHeL9XbVz/ShZUvEs /MaCt/6w0sdl/z5HK9YKg4XjZPNzjAOtsxkNiOjU8Q9COu2G7wHsl09cqyL71FW4+tBcXB AlYA9PEqIWv14Kew24+gwUrWGu3bxAP9pciHY2ncli2bN1HNUZR2cinoTPgC9w== Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4VwBLq2NtmzvwM; Thu, 6 Jun 2024 17:35:47 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org ([127.0.1.44]) by gitrepo.freebsd.org (8.17.1/8.17.1) with ESMTP id 456HZlJo024196; Thu, 6 Jun 2024 17:35:47 GMT (envelope-from git@gitrepo.freebsd.org) Received: (from git@localhost) by gitrepo.freebsd.org (8.17.1/8.17.1/Submit) id 456HZl0g024193; Thu, 6 Jun 2024 17:35:47 GMT (envelope-from git) Date: Thu, 6 Jun 2024 17:35:47 GMT Message-Id: <202406061735.456HZl0g024193@gitrepo.freebsd.org> To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-main@FreeBSD.org From: Ryan Libby Subject: git: 1b13e36fcc00 - main - vm_page_insert: use pctrie combined insert/lookup List-Id: Commit messages for the main branch of the src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-main List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: dev-commits-src-main@freebsd.org Sender: owner-dev-commits-src-main@FreeBSD.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: rlibby X-Git-Repository: src X-Git-Refname: refs/heads/main X-Git-Reftype: branch X-Git-Commit: 1b13e36fcc000186045186451ab2a2fec5654fac Auto-Submitted: auto-generated The branch main has been updated by rlibby: URL: https://cgit.FreeBSD.org/src/commit/?id=1b13e36fcc000186045186451ab2a2fec5654fac commit 1b13e36fcc000186045186451ab2a2fec5654fac Author: Ryan Libby AuthorDate: 2024-06-06 17:26:50 +0000 Commit: Ryan Libby CommitDate: 2024-06-06 17:26:50 +0000 vm_page_insert: use pctrie combined insert/lookup This reduces work done under vm_page_insert for large objects. Reviewed by: alc, dougm, markj Sponsored by: Dell EMC Isilon Differential Revision: https://reviews.freebsd.org/D45486 --- sys/vm/vm_page.c | 100 ++++++++++++++++++++++++++++++++----------------------- 1 file changed, 59 insertions(+), 41 deletions(-) diff --git a/sys/vm/vm_page.c b/sys/vm/vm_page.c index b41439045205..4b97637668b4 100644 --- a/sys/vm/vm_page.c +++ b/sys/vm/vm_page.c @@ -1416,6 +1416,56 @@ vm_page_dirty_KBI(vm_page_t m) m->dirty = VM_PAGE_BITS_ALL; } +/* + * Insert the given page into the given object at the given pindex. mpred is + * used for memq linkage. From vm_page_insert, lookup is true, mpred is + * initially NULL, and this procedure looks it up. From vm_page_insert_after, + * lookup is false and mpred is known to the caller to be valid, and may be + * NULL if this will be the page with the lowest pindex. + * + * The procedure is marked __always_inline to suggest to the compiler to + * eliminate the lookup parameter and the associated alternate branch. + */ +static __always_inline int +vm_page_insert_lookup(vm_page_t m, vm_object_t object, vm_pindex_t pindex, + vm_page_t mpred, bool lookup) +{ + int error; + + VM_OBJECT_ASSERT_WLOCKED(object); + KASSERT(m->object == NULL, + ("vm_page_insert: page %p already inserted", m)); + + /* + * Record the object/offset pair in this page. + */ + m->object = object; + m->pindex = pindex; + m->ref_count |= VPRC_OBJREF; + + /* + * Add this page to the object's radix tree, and look up mpred if + * needed. + */ + if (lookup) + error = vm_radix_insert_lookup_lt(&object->rtree, m, &mpred); + else + error = vm_radix_insert(&object->rtree, m); + if (__predict_false(error != 0)) { + m->object = NULL; + m->pindex = 0; + m->ref_count &= ~VPRC_OBJREF; + return (1); + } + + /* + * Now link into the object's ordered list of backed pages. + */ + vm_page_insert_radixdone(m, object, mpred); + vm_pager_page_inserted(object, m); + return (0); +} + /* * vm_page_insert: [ internal use only ] * @@ -1426,11 +1476,7 @@ vm_page_dirty_KBI(vm_page_t m) int vm_page_insert(vm_page_t m, vm_object_t object, vm_pindex_t pindex) { - vm_page_t mpred; - - VM_OBJECT_ASSERT_WLOCKED(object); - mpred = vm_radix_lookup_le(&object->rtree, pindex); - return (vm_page_insert_after(m, object, pindex, mpred)); + return (vm_page_insert_lookup(m, object, pindex, NULL, true)); } /* @@ -1447,42 +1493,7 @@ static int vm_page_insert_after(vm_page_t m, vm_object_t object, vm_pindex_t pindex, vm_page_t mpred) { - vm_page_t msucc; - - VM_OBJECT_ASSERT_WLOCKED(object); - KASSERT(m->object == NULL, - ("vm_page_insert_after: page already inserted")); - if (mpred != NULL) { - KASSERT(mpred->object == object, - ("vm_page_insert_after: object doesn't contain mpred")); - KASSERT(mpred->pindex < pindex, - ("vm_page_insert_after: mpred doesn't precede pindex")); - msucc = TAILQ_NEXT(mpred, listq); - } else - msucc = TAILQ_FIRST(&object->memq); - if (msucc != NULL) - KASSERT(msucc->pindex > pindex, - ("vm_page_insert_after: msucc doesn't succeed pindex")); - - /* - * Record the object/offset pair in this page. - */ - m->object = object; - m->pindex = pindex; - m->ref_count |= VPRC_OBJREF; - - /* - * Now link into the object's ordered list of backed pages. - */ - if (vm_radix_insert(&object->rtree, m)) { - m->object = NULL; - m->pindex = 0; - m->ref_count &= ~VPRC_OBJREF; - return (1); - } - vm_page_insert_radixdone(m, object, mpred); - vm_pager_page_inserted(object, m); - return (0); + return (vm_page_insert_lookup(m, object, pindex, mpred, false)); } /* @@ -1510,6 +1521,13 @@ vm_page_insert_radixdone(vm_page_t m, vm_object_t object, vm_page_t mpred) ("vm_page_insert_radixdone: object doesn't contain mpred")); KASSERT(mpred->pindex < m->pindex, ("vm_page_insert_radixdone: mpred doesn't precede pindex")); + KASSERT(TAILQ_NEXT(mpred, listq) == NULL || + m->pindex < TAILQ_NEXT(mpred, listq)->pindex, + ("vm_page_insert_radixdone: pindex doesn't precede msucc")); + } else { + KASSERT(TAILQ_EMPTY(&object->memq) || + m->pindex < TAILQ_FIRST(&object->memq)->pindex, + ("vm_page_insert_radixdone: no mpred but not first page")); } if (mpred != NULL)