From nobody Thu Nov 02 18:47:43 2023 X-Original-To: dev-commits-src-main@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4SLtD01hrCz503l0; Thu, 2 Nov 2023 18:47:44 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4SLtD010jnz3WRL; Thu, 2 Nov 2023 18:47:44 +0000 (UTC) (envelope-from git@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1698950864; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=UaZiA964YdS4JXnqcB2nhxBosPcapNk6hd2CIBLN9UE=; b=LoZj2cmkHUX9H/sNqPKNS2b34O3zJgj35rYrmCrEy+KdAx51qyEjMdDRjrCQb/SQ5bZEkI brNG5aDXjCtcbv+njCgF90RH51Y9gv6KmX8DPf5VLQCkRerkt9thGwRCywR3PE4bTiUEmY OpZgiXxtdURx6lrWger+XQ0r1Dx7RP5vrdJxjHiJE5KeYgK5Bz05wFtdI8BrUEU02DTuyB iKChgUH24eqHJmxgXhan0CKCB64etZqmeyikE9tlRJ4etzhnPK++haHj/PpgpC8pdFqUts OyPSMe1THhtcJmciqb6oBaa2B7572LsYzmrZYckp7b9HlvraKGduGW55E2jtLA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1698950864; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=UaZiA964YdS4JXnqcB2nhxBosPcapNk6hd2CIBLN9UE=; b=EKRT1ENHVt0MY1ESiKW3vfxubU+KkexWjhakTYMFlrpxQO9zEAVMF0T8FdbNbGzDUF0hR4 rku1nXWSrMW/KMhUcOTupLxiQUx3yX1NkwyvtGY0ZdzAseVZOT4TezjpJQEjqB427v9VHo h/+rI6J1UxlY2SNXBmcXPxyym9osmb0YnvP0JiyrMNQK14o2rVWWkUYa74Ky4tsBeYYYdm wFQ20slLffDzTfNULxbPk6gQDTI4dPsKT8/zUpSfmm9CaUYiZLo/Y9I1u+86ZdyezcoW55 OMpz0LRcjFO5PlmUUBt7SobduL4SsTBwDruMqcAgL60c7JqwzPOxkiy4Sg3EHg== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1698950864; a=rsa-sha256; cv=none; b=ie6zkoB+YOEf3Uz7yte4/sQV5eWO2rOx9TZysy575TJv+C/96ZEVf509WJR4apjb7O+yQB JPY/4Fl4JRnwPkDumbXAPeBu0kXsaA1A/RuYG62ee9UcqY7HPjIIVTFY7/4B6OLl77mWxC T16q374mb/ogIh1B0cWqnjw746zjD2s4yadLG2JuSyIEsM4Ca85Eq3KE+7BomAO23sPfo2 sh1iZwbcECkYB+w7Y2umw39dmHCVlBYoCvTJvF35QsO6zw6qauC7m4D5zADOPJgVvIJJMw cHypjf0N8KGaQC11C3rDTyph3JF11RFdmSHumnqaabeNZmU6aLkxtJmEHyqrCg== Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4SLtCz72lJzh56; Thu, 2 Nov 2023 18:47:43 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org ([127.0.1.44]) by gitrepo.freebsd.org (8.17.1/8.17.1) with ESMTP id 3A2IlhWS044652; Thu, 2 Nov 2023 18:47:43 GMT (envelope-from git@gitrepo.freebsd.org) Received: (from git@localhost) by gitrepo.freebsd.org (8.17.1/8.17.1/Submit) id 3A2Ilhks044649; Thu, 2 Nov 2023 18:47:43 GMT (envelope-from git) Date: Thu, 2 Nov 2023 18:47:43 GMT Message-Id: <202311021847.3A2Ilhks044649@gitrepo.freebsd.org> To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-main@FreeBSD.org From: Mark Johnston Subject: git: 7703ac2e983b - main - riscv: Port improvements from arm64/amd64 pmaps, part 1 List-Id: Commit messages for the main branch of the src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-main List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-dev-commits-src-main@freebsd.org X-BeenThere: dev-commits-src-main@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: markj X-Git-Repository: src X-Git-Refname: refs/heads/main X-Git-Reftype: branch X-Git-Commit: 7703ac2e983bcd44ba93878f52eb30355cbc4821 Auto-Submitted: auto-generated The branch main has been updated by markj: URL: https://cgit.FreeBSD.org/src/commit/?id=7703ac2e983bcd44ba93878f52eb30355cbc4821 commit 7703ac2e983bcd44ba93878f52eb30355cbc4821 Author: Mark Johnston AuthorDate: 2023-11-02 18:33:37 +0000 Commit: Mark Johnston CommitDate: 2023-11-02 18:33:37 +0000 riscv: Port improvements from arm64/amd64 pmaps, part 1 - When promoting, do not require that all PTEs all have PTE_A set. Instead, record whether they did and store this information in the PTP's valid bits. - Synchronize some comments in pmap_promote_l2(). - Make pmap_promote_l2() scan starting from the end of the 2MB range instead of the beginning. See the commit log for 9d1b7fa31f510 for justification of this, which I believe applies here as well. Reviewed by: kib MFC after: 1 month Differential Revision: https://reviews.freebsd.org/D42288 --- sys/riscv/include/pte.h | 2 +- sys/riscv/riscv/pmap.c | 66 ++++++++++++++++++++++++++++++++++++++----------- 2 files changed, 52 insertions(+), 16 deletions(-) diff --git a/sys/riscv/include/pte.h b/sys/riscv/include/pte.h index 2cede135ebae..da7bd051e122 100644 --- a/sys/riscv/include/pte.h +++ b/sys/riscv/include/pte.h @@ -80,7 +80,7 @@ typedef uint64_t pn_t; /* page number */ #define PTE_RWX (PTE_R | PTE_W | PTE_X) #define PTE_RX (PTE_R | PTE_X) #define PTE_KERN (PTE_V | PTE_R | PTE_W | PTE_A | PTE_D) -#define PTE_PROMOTE (PTE_V | PTE_RWX | PTE_D | PTE_A | PTE_G | PTE_U | \ +#define PTE_PROMOTE (PTE_V | PTE_RWX | PTE_D | PTE_G | PTE_U | \ PTE_SW_MANAGED | PTE_SW_WIRED) /* Bits 63 - 54 are reserved for future use. */ diff --git a/sys/riscv/riscv/pmap.c b/sys/riscv/riscv/pmap.c index 5747f04b7e0b..30bcf113628c 100644 --- a/sys/riscv/riscv/pmap.c +++ b/sys/riscv/riscv/pmap.c @@ -1194,14 +1194,26 @@ pmap_add_delayed_free_list(vm_page_t m, struct spglist *free, * for mapping a distinct range of virtual addresses. The pmap's collection is * ordered by this virtual address range. * - * If "promoted" is false, then the page table page "ml3" must be zero filled. + * If "promoted" is false, then the page table page "mpte" must be zero filled; + * "mpte"'s valid field will be set to 0. + * + * If "promoted" is true and "all_l3e_PTE_A_set" is false, then "mpte" must + * contain valid mappings with identical attributes except for PTE_A; + * "mpte"'s valid field will be set to 1. + * + * If "promoted" and "all_l3e_PTE_A_set" are both true, then "mpte" must contain + * valid mappings with identical attributes including PTE_A; "mpte"'s valid + * field will be set to VM_PAGE_BITS_ALL. */ static __inline int -pmap_insert_pt_page(pmap_t pmap, vm_page_t ml3, bool promoted) +pmap_insert_pt_page(pmap_t pmap, vm_page_t ml3, bool promoted, + bool all_l3e_PTE_A_set) { PMAP_LOCK_ASSERT(pmap, MA_OWNED); - ml3->valid = promoted ? VM_PAGE_BITS_ALL : 0; + KASSERT(promoted || !all_l3e_PTE_A_set, + ("a zero-filled PTP can't have PTE_A set in every PTE")); + ml3->valid = promoted ? (all_l3e_PTE_A_set ? VM_PAGE_BITS_ALL : 1) : 0; return (vm_radix_insert(&pmap->pm_root, ml3)); } @@ -2169,7 +2181,7 @@ pmap_remove_kernel_l2(pmap_t pmap, pt_entry_t *l2, vm_offset_t va) * If this page table page was unmapped by a promotion, then it * contains valid mappings. Zero it to invalidate those mappings. */ - if (ml3->valid != 0) + if (vm_page_any_valid(ml3)) pagezero((void *)PHYS_TO_DMAP(ml3pa)); /* @@ -2230,7 +2242,7 @@ pmap_remove_l2(pmap_t pmap, pt_entry_t *l2, vm_offset_t sva, } else { ml3 = pmap_remove_pt_page(pmap, sva); if (ml3 != NULL) { - KASSERT(ml3->valid == VM_PAGE_BITS_ALL, + KASSERT(vm_page_any_valid(ml3), ("pmap_remove_l2: l3 page not promoted")); pmap_resident_count_dec(pmap, 1); KASSERT(ml3->ref_count == Ln_ENTRIES, @@ -2703,7 +2715,7 @@ pmap_demote_l2_locked(pmap_t pmap, pd_entry_t *l2, vm_offset_t va, * If the page table page is not leftover from an earlier promotion, * initialize it. */ - if (mpte->valid == 0) { + if (!vm_page_all_valid(mpte)) { for (i = 0; i < Ln_ENTRIES; i++) pmap_store(firstl3 + i, newl3 + (i << PTE_PPN0_S)); } @@ -2712,8 +2724,7 @@ pmap_demote_l2_locked(pmap_t pmap, pd_entry_t *l2, vm_offset_t va, "addresses")); /* - * If the mapping has changed attributes, update the page table - * entries. + * If the mapping has changed attributes, update the PTEs. */ if ((pmap_load(firstl3) & PTE_PROMOTE) != (newl3 & PTE_PROMOTE)) for (i = 0; i < Ln_ENTRIES; i++) @@ -2752,7 +2763,7 @@ static void pmap_promote_l2(pmap_t pmap, pd_entry_t *l2, vm_offset_t va, vm_page_t ml3, struct rwlock **lockp) { - pt_entry_t *firstl3, firstl3e, *l3, l3e; + pt_entry_t all_l3e_PTE_A, *firstl3, firstl3e, *l3, l3e; vm_paddr_t pa; PMAP_LOCK_ASSERT(pmap, MA_OWNED); @@ -2760,6 +2771,11 @@ pmap_promote_l2(pmap_t pmap, pd_entry_t *l2, vm_offset_t va, vm_page_t ml3, KASSERT((pmap_load(l2) & PTE_RWX) == 0, ("pmap_promote_l2: invalid l2 entry %p", l2)); + /* + * Examine the first L3E in the specified PTP. Abort if this L3E is + * ineligible for promotion or does not map the first 4KB physical page + * within a 2MB page. + */ firstl3 = (pt_entry_t *)PHYS_TO_DMAP(PTE_TO_PHYS(pmap_load(l2))); firstl3e = pmap_load(firstl3); pa = PTE_TO_PHYS(firstl3e); @@ -2787,8 +2803,14 @@ pmap_promote_l2(pmap_t pmap, pd_entry_t *l2, vm_offset_t va, vm_page_t ml3, } } - pa += PAGE_SIZE; - for (l3 = firstl3 + 1; l3 < firstl3 + Ln_ENTRIES; l3++) { + /* + * Examine each of the other PTEs in the specified PTP. Abort if this + * PTE maps an unexpected 4KB physical page or does not have identical + * characteristics to the first PTE. + */ + all_l3e_PTE_A = firstl3e & PTE_A; + pa += L2_SIZE - PAGE_SIZE; + for (l3 = firstl3 + Ln_ENTRIES - 1; l3 > firstl3; l3--) { l3e = pmap_load(l3); if (PTE_TO_PHYS(l3e) != pa) { CTR2(KTR_PMAP, @@ -2810,14 +2832,28 @@ pmap_promote_l2(pmap_t pmap, pd_entry_t *l2, vm_offset_t va, vm_page_t ml3, atomic_add_long(&pmap_l2_p_failures, 1); return; } - pa += PAGE_SIZE; + all_l3e_PTE_A &= l3e; + pa -= PAGE_SIZE; } + /* + * Unless all PTEs have PTE_A set, clear it from the superpage + * mapping, so that promotions triggered by speculative mappings, + * such as pmap_enter_quick(), don't automatically mark the + * underlying pages as referenced. + */ + firstl3e &= ~PTE_A | all_l3e_PTE_A; + + /* + * Save the page table page in its current state until the L2 + * mapping the superpage is demoted by pmap_demote_l2() or + * destroyed by pmap_remove_l3(). + */ if (ml3 == NULL) ml3 = PHYS_TO_VM_PAGE(PTE_TO_PHYS(pmap_load(l2))); KASSERT(ml3->pindex == pmap_l2_pindex(va), ("pmap_promote_l2: page table page's pindex is wrong")); - if (pmap_insert_pt_page(pmap, ml3, true)) { + if (pmap_insert_pt_page(pmap, ml3, true, all_l3e_PTE_A != 0)) { CTR2(KTR_PMAP, "pmap_promote_l2: failure for va %#lx pmap %p", va, pmap); atomic_add_long(&pmap_l2_p_failures, 1); @@ -3229,7 +3265,7 @@ pmap_enter_l2(pmap_t pmap, vm_offset_t va, pd_entry_t new_l2, u_int flags, * leave the kernel page table page zero filled. */ mt = PHYS_TO_VM_PAGE(PTE_TO_PHYS(pmap_load(l2))); - if (pmap_insert_pt_page(pmap, mt, false)) + if (pmap_insert_pt_page(pmap, mt, false, false)) panic("pmap_enter_l2: trie insert failed"); } else KASSERT(pmap_load(l2) == 0, @@ -3847,7 +3883,7 @@ pmap_remove_pages_pv(pmap_t pmap, vm_page_t m, pv_entry_t pv, } mpte = pmap_remove_pt_page(pmap, pv->pv_va); if (mpte != NULL) { - KASSERT(mpte->valid == VM_PAGE_BITS_ALL, + KASSERT(vm_page_any_valid(mpte), ("pmap_remove_pages: pte page not promoted")); pmap_resident_count_dec(pmap, 1); KASSERT(mpte->ref_count == Ln_ENTRIES,