From nobody Fri Sep 30 17:15:06 2022 X-Original-To: dev-commits-src-main@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4MfH0q1YFTz4V2Wt; Fri, 30 Sep 2022 17:15:07 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4MfH0q13tSz3yb3; Fri, 30 Sep 2022 17:15:07 +0000 (UTC) (envelope-from git@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1664558107; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=Kg1dUaDIH7rJM1Hpf21T1k5bJGxe91bsTlz35dII5yU=; b=F1ckVx+EYRqiMKOlHTl0vgLNfmMv1woZuvZuqT8cYTbIpAH9/3zyRead278Z+Gdg8apvGH eVlAVrBoxSRQ74vfk7s1RCE0T2aJjCyRm3fAjHQsqVrE5nmI/UmOdpWY4uN64qDxQ12Ns4 Y8PHUlpaWdR0pcKfkKzgmeKeOJELnQkui4adsGxqDSFXkGlbiknRh//eRU2cyc/jFuAAJh sd6eO3QNqr9BU7TDGDSZ65+lc8PDzakPyIUn6ohrIdt9p/3U0Vgv+u2+6ntRHFmpZoHSI1 nvtWpwyMbk5Tc4kFm7NFq/LLWVs5eXu/z+0sYYbhBx8FXtc6cC3T4u4Zu2DOTA== Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4MfH0q06HCzFdV; Fri, 30 Sep 2022 17:15:07 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org ([127.0.1.44]) by gitrepo.freebsd.org (8.16.1/8.16.1) with ESMTP id 28UHF61e031208; Fri, 30 Sep 2022 17:15:06 GMT (envelope-from git@gitrepo.freebsd.org) Received: (from git@localhost) by gitrepo.freebsd.org (8.16.1/8.16.1/Submit) id 28UHF6Jx031207; Fri, 30 Sep 2022 17:15:06 GMT (envelope-from git) Date: Fri, 30 Sep 2022 17:15:06 GMT Message-Id: <202209301715.28UHF6Jx031207@gitrepo.freebsd.org> To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-main@FreeBSD.org From: Alan Cox Subject: git: 1d5ebad06c20 - main - pmap: optimize MADV_WILLNEED on existing superpages List-Id: Commit messages for the main branch of the src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-main List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-dev-commits-src-main@freebsd.org X-BeenThere: dev-commits-src-main@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: alc X-Git-Repository: src X-Git-Refname: refs/heads/main X-Git-Reftype: branch X-Git-Commit: 1d5ebad06c20b1aed3b0c323c4675678afec5e55 Auto-Submitted: auto-generated ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1664558107; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=Kg1dUaDIH7rJM1Hpf21T1k5bJGxe91bsTlz35dII5yU=; b=s03EX06ZM57kp6o5IrJzdRApyrFw0uMCvbRJ53AKzHUddhxLq/d2Ztg5tVXJ4zQWHrK4Um 2fuxbmLxfNE7uF8oHBDSF2cZVrt01tRdekImhAYVjcK2sIMPtECX+j5gjcRqwwm7qwU+KC 6YFfyiS2h7Rl5g6btFoNqHEXKSirwFX1lER6ga5VOLZao2uo5hjVGLUlOnDquiQlXd6QiI 2f6lbjcaqDKTTaWM7nBNNVdsqEmMS9UYF3b8X6K63ACxRndAEhIWD7LF20LnXov9bIOg4L bL4ryodHJON0G/gFxLGe1D3Byo+NgEYdRvbq/VEHPNyxdyDl05fY/qL5GC/RwQ== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1664558107; a=rsa-sha256; cv=none; b=y4HeoYcnImgcIBYGXpzJES/aodPTpqNBpOrPkQZbRR1BHvgX5HjV5e1jp4e/HjJjtBqSWy XIPB1BefTLYN5p/9DgdaJpUD7GIw/nrV19f8K1ol/TEd43ov0osKRXTtkXHKtBUzyu+L7B Jajf4Yba59FzjI/PWNDlE3ZMzzKrroQ44LAeW5chCl5/nlUshIgP5yaWiTVQujJk62nZtx 90mdarBKyi3QlxDyF2A4McObAXUUugaNY09pVcgJTGn+OBeXxXiw3QW5Q/ssmwyBWvIGYv VN2ITJMvCLV5lVzwf+Uyj3D4OJk1Nadmo2KHh/RkPJNbkPIN2Suan5K4S1pn+g== ARC-Authentication-Results: i=1; mx1.freebsd.org; none X-ThisMailContainsUnwantedMimeParts: N The branch main has been updated by alc: URL: https://cgit.FreeBSD.org/src/commit/?id=1d5ebad06c20b1aed3b0c323c4675678afec5e55 commit 1d5ebad06c20b1aed3b0c323c4675678afec5e55 Author: Alan Cox AuthorDate: 2022-09-30 06:54:02 +0000 Commit: Alan Cox CommitDate: 2022-09-30 17:14:05 +0000 pmap: optimize MADV_WILLNEED on existing superpages Specifically, avoid pointless calls to pmap_enter_quick_locked() when madvise(MADV_WILLNEED) is applied to an existing superpage mapping. Reported by: mhorne Reviewed by: kib, markj MFC after: 1 week Differential Revision: https://reviews.freebsd.org/D36801 --- sys/amd64/amd64/pmap.c | 64 +++++++++++++++++++++++++++++++------------------- sys/arm64/arm64/pmap.c | 59 +++++++++++++++++++++++++++------------------- 2 files changed, 75 insertions(+), 48 deletions(-) diff --git a/sys/amd64/amd64/pmap.c b/sys/amd64/amd64/pmap.c index f4df664f0cca..b9b031d55d7d 100644 --- a/sys/amd64/amd64/pmap.c +++ b/sys/amd64/amd64/pmap.c @@ -1258,7 +1258,7 @@ static boolean_t pmap_demote_pde_locked(pmap_t pmap, pd_entry_t *pde, vm_offset_t va, struct rwlock **lockp); static boolean_t pmap_demote_pdpe(pmap_t pmap, pdp_entry_t *pdpe, vm_offset_t va); -static bool pmap_enter_2mpage(pmap_t pmap, vm_offset_t va, vm_page_t m, +static int pmap_enter_2mpage(pmap_t pmap, vm_offset_t va, vm_page_t m, vm_prot_t prot, struct rwlock **lockp); static int pmap_enter_pde(pmap_t pmap, vm_offset_t va, pd_entry_t newpde, u_int flags, vm_page_t m, struct rwlock **lockp); @@ -7271,13 +7271,12 @@ out: } /* - * Tries to create a read- and/or execute-only 2MB page mapping. Returns true - * if successful. Returns false if (1) a page table page cannot be allocated - * without sleeping, (2) a mapping already exists at the specified virtual - * address, or (3) a PV entry cannot be allocated without reclaiming another - * PV entry. + * Tries to create a read- and/or execute-only 2MB page mapping. Returns + * KERN_SUCCESS if the mapping was created. Otherwise, returns an error + * value. See pmap_enter_pde() for the possible error values when "no sleep", + * "no replace", and "no reclaim" are specified. */ -static bool +static int pmap_enter_2mpage(pmap_t pmap, vm_offset_t va, vm_page_t m, vm_prot_t prot, struct rwlock **lockp) { @@ -7295,8 +7294,7 @@ pmap_enter_2mpage(pmap_t pmap, vm_offset_t va, vm_page_t m, vm_prot_t prot, if (va < VM_MAXUSER_ADDRESS) newpde |= PG_U; return (pmap_enter_pde(pmap, va, newpde, PMAP_ENTER_NOSLEEP | - PMAP_ENTER_NOREPLACE | PMAP_ENTER_NORECLAIM, NULL, lockp) == - KERN_SUCCESS); + PMAP_ENTER_NOREPLACE | PMAP_ENTER_NORECLAIM, NULL, lockp)); } /* @@ -7319,12 +7317,19 @@ pmap_every_pte_zero(vm_paddr_t pa) /* * Tries to create the specified 2MB page mapping. Returns KERN_SUCCESS if - * the mapping was created, and either KERN_FAILURE or KERN_RESOURCE_SHORTAGE - * otherwise. Returns KERN_FAILURE if PMAP_ENTER_NOREPLACE was specified and - * a mapping already exists at the specified virtual address. Returns - * KERN_RESOURCE_SHORTAGE if PMAP_ENTER_NOSLEEP was specified and a page table - * page allocation failed. Returns KERN_RESOURCE_SHORTAGE if - * PMAP_ENTER_NORECLAIM was specified and a PV entry allocation failed. + * the mapping was created, and one of KERN_FAILURE, KERN_NO_SPACE, + * KERN_PROTECTION_FAILURE, or KERN_RESOURCE_FAILURE otherwise. Returns + * KERN_FAILURE if either (1) PMAP_ENTER_NOREPLACE was specified and a 4KB + * page mapping already exists within the 2MB virtual address range starting + * at the specified virtual address or (2) the requested 2MB page mapping is + * not supported due to hardware errata. Returns KERN_NO_SPACE if + * PMAP_ENTER_NOREPLACE was specified and a 2MB page mapping already exists at + * the specified virtual address. Returns KERN_PROTECTION_FAILURE if the PKRU + * settings are not the same across the 2MB virtual address range starting at + * the specified virtual address. Returns KERN_RESOURCE_SHORTAGE if either + * (1) PMAP_ENTER_NOSLEEP was specified and a page table page allocation + * failed or (2) PMAP_ENTER_NORECLAIM was specified and a PV entry allocation + * failed. * * The parameter "m" is only used when creating a managed, writeable mapping. */ @@ -7380,14 +7385,23 @@ pmap_enter_pde(pmap_t pmap, vm_offset_t va, pd_entry_t newpde, u_int flags, if ((oldpde & PG_V) != 0) { KASSERT(pdpg == NULL || pdpg->ref_count > 1, ("pmap_enter_pde: pdpg's reference count is too low")); - if ((flags & PMAP_ENTER_NOREPLACE) != 0 && (va < - VM_MAXUSER_ADDRESS || (oldpde & PG_PS) != 0 || - !pmap_every_pte_zero(oldpde & PG_FRAME))) { - if (pdpg != NULL) - pdpg->ref_count--; - CTR2(KTR_PMAP, "pmap_enter_pde: failure for va %#lx" - " in pmap %p", va, pmap); - return (KERN_FAILURE); + if ((flags & PMAP_ENTER_NOREPLACE) != 0) { + if ((oldpde & PG_PS) != 0) { + if (pdpg != NULL) + pdpg->ref_count--; + CTR2(KTR_PMAP, + "pmap_enter_pde: no space for va %#lx" + " in pmap %p", va, pmap); + return (KERN_NO_SPACE); + } else if (va < VM_MAXUSER_ADDRESS || + !pmap_every_pte_zero(oldpde & PG_FRAME)) { + if (pdpg != NULL) + pdpg->ref_count--; + CTR2(KTR_PMAP, + "pmap_enter_pde: failure for va %#lx" + " in pmap %p", va, pmap); + return (KERN_FAILURE); + } } /* Break the existing mapping(s). */ SLIST_INIT(&free); @@ -7482,6 +7496,7 @@ pmap_enter_object(pmap_t pmap, vm_offset_t start, vm_offset_t end, vm_offset_t va; vm_page_t m, mpte; vm_pindex_t diff, psize; + int rv; VM_OBJECT_ASSERT_LOCKED(m_start->object); @@ -7494,7 +7509,8 @@ pmap_enter_object(pmap_t pmap, vm_offset_t start, vm_offset_t end, va = start + ptoa(diff); if ((va & PDRMASK) == 0 && va + NBPDR <= end && m->psind == 1 && pmap_ps_enabled(pmap) && - pmap_enter_2mpage(pmap, va, m, prot, &lock)) + ((rv = pmap_enter_2mpage(pmap, va, m, prot, &lock)) == + KERN_SUCCESS || rv == KERN_NO_SPACE)) m = &m[NBPDR / PAGE_SIZE - 1]; else mpte = pmap_enter_quick_locked(pmap, va, m, prot, diff --git a/sys/arm64/arm64/pmap.c b/sys/arm64/arm64/pmap.c index dcc0c637cc1e..148e325c84ba 100644 --- a/sys/arm64/arm64/pmap.c +++ b/sys/arm64/arm64/pmap.c @@ -4419,13 +4419,12 @@ out: } /* - * Tries to create a read- and/or execute-only 2MB page mapping. Returns true - * if successful. Returns false if (1) a page table page cannot be allocated - * without sleeping, (2) a mapping already exists at the specified virtual - * address, or (3) a PV entry cannot be allocated without reclaiming another - * PV entry. + * Tries to create a read- and/or execute-only 2MB page mapping. Returns + * KERN_SUCCESS if the mapping was created. Otherwise, returns an error + * value. See pmap_enter_l2() for the possible error values when "no sleep", + * "no replace", and "no reclaim" are specified. */ -static bool +static int pmap_enter_2mpage(pmap_t pmap, vm_offset_t va, vm_page_t m, vm_prot_t prot, struct rwlock **lockp) { @@ -4453,8 +4452,7 @@ pmap_enter_2mpage(pmap_t pmap, vm_offset_t va, vm_page_t m, vm_prot_t prot, if (pmap != kernel_pmap) new_l2 |= ATTR_S1_nG; return (pmap_enter_l2(pmap, va, new_l2, PMAP_ENTER_NOSLEEP | - PMAP_ENTER_NOREPLACE | PMAP_ENTER_NORECLAIM, m, lockp) == - KERN_SUCCESS); + PMAP_ENTER_NOREPLACE | PMAP_ENTER_NORECLAIM, m, lockp)); } /* @@ -4477,12 +4475,15 @@ pmap_every_pte_zero(vm_paddr_t pa) /* * Tries to create the specified 2MB page mapping. Returns KERN_SUCCESS if - * the mapping was created, and either KERN_FAILURE or KERN_RESOURCE_SHORTAGE - * otherwise. Returns KERN_FAILURE if PMAP_ENTER_NOREPLACE was specified and - * a mapping already exists at the specified virtual address. Returns - * KERN_RESOURCE_SHORTAGE if PMAP_ENTER_NOSLEEP was specified and a page table - * page allocation failed. Returns KERN_RESOURCE_SHORTAGE if - * PMAP_ENTER_NORECLAIM was specified and a PV entry allocation failed. + * the mapping was created, and one of KERN_FAILURE, KERN_NO_SPACE, or + * KERN_RESOURCE_FAILURE otherwise. Returns KERN_FAILURE if + * PMAP_ENTER_NOREPLACE was specified and a 4KB page mapping already exists + * within the 2MB virtual address range starting at the specified virtual + * address. Returns KERN_NO_SPACE if PMAP_ENTER_NOREPLACE was specified and a + * 2MB page mapping already exists at the specified virtual address. Returns + * KERN_RESOURCE_SHORTAGE if either (1) PMAP_ENTER_NOSLEEP was specified and a + * page table page allocation failed or (2) PMAP_ENTER_NORECLAIM was specified + * and a PV entry allocation failed. */ static int pmap_enter_l2(pmap_t pmap, vm_offset_t va, pd_entry_t new_l2, u_int flags, @@ -4509,15 +4510,23 @@ pmap_enter_l2(pmap_t pmap, vm_offset_t va, pd_entry_t new_l2, u_int flags, if ((old_l2 = pmap_load(l2)) != 0) { KASSERT(l2pg == NULL || l2pg->ref_count > 1, ("pmap_enter_l2: l2pg's ref count is too low")); - if ((flags & PMAP_ENTER_NOREPLACE) != 0 && - (!ADDR_IS_KERNEL(va) || - (old_l2 & ATTR_DESCR_MASK) == L2_BLOCK || - !pmap_every_pte_zero(old_l2 & ~ATTR_MASK))) { - if (l2pg != NULL) - l2pg->ref_count--; - CTR2(KTR_PMAP, "pmap_enter_l2: failure for va %#lx" - " in pmap %p", va, pmap); - return (KERN_FAILURE); + if ((flags & PMAP_ENTER_NOREPLACE) != 0) { + if ((old_l2 & ATTR_DESCR_MASK) == L2_BLOCK) { + if (l2pg != NULL) + l2pg->ref_count--; + CTR2(KTR_PMAP, + "pmap_enter_l2: no space for va %#lx" + " in pmap %p", va, pmap); + return (KERN_NO_SPACE); + } else if (!ADDR_IS_KERNEL(va) || + !pmap_every_pte_zero(old_l2 & ~ATTR_MASK)) { + if (l2pg != NULL) + l2pg->ref_count--; + CTR2(KTR_PMAP, + "pmap_enter_l2: failure for va %#lx" + " in pmap %p", va, pmap); + return (KERN_FAILURE); + } } SLIST_INIT(&free); if ((old_l2 & ATTR_DESCR_MASK) == L2_BLOCK) @@ -4617,6 +4626,7 @@ pmap_enter_object(pmap_t pmap, vm_offset_t start, vm_offset_t end, vm_offset_t va; vm_page_t m, mpte; vm_pindex_t diff, psize; + int rv; VM_OBJECT_ASSERT_LOCKED(m_start->object); @@ -4629,7 +4639,8 @@ pmap_enter_object(pmap_t pmap, vm_offset_t start, vm_offset_t end, va = start + ptoa(diff); if ((va & L2_OFFSET) == 0 && va + L2_SIZE <= end && m->psind == 1 && pmap_ps_enabled(pmap) && - pmap_enter_2mpage(pmap, va, m, prot, &lock)) + ((rv = pmap_enter_2mpage(pmap, va, m, prot, &lock)) == + KERN_SUCCESS || rv == KERN_NO_SPACE)) m = &m[L2_SIZE / PAGE_SIZE - 1]; else mpte = pmap_enter_quick_locked(pmap, va, m, prot, mpte,