From nobody Wed Mar 15 12:01:17 2023 X-Original-To: dev-commits-src-main@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Pc8B60rH5z3yRDf; Wed, 15 Mar 2023 12:01:18 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Pc8B603Bkz4F7R; Wed, 15 Mar 2023 12:01:18 +0000 (UTC) (envelope-from git@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1678881678; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=aRYf28S4DotnGRb8f0rB4+d4UYxeRJaeepAwF0O72Xg=; b=wjgiYz+QTFxnV+W5I89uWEC3I2SM42uQa9gGultYkxu7yiJPDAPrRnjUnx+l9okEUhmz4B yfR3U5v+Pn8OV5JL/EKY7leTJNwONJ8VtSwehXVW0An8bzOpsdqA+c0uqFtRGt//WN0Zcf RW9ufyq01f2WuYvGyuQo3BaskvYBn2K5AJilNstRPtHALSFtm4MU2XnB6LM35vKpxIpomM Vf4l5b+QOnaRKkGeZH3HQKHDWpzQ5gaLX3JH57ldjzkuv5qQwK7NUHR/XGwMH1FIHL0WXL fUYKgcR+ux3wrzqU3g47jlpctH+XxgKjfI24mQLZtPc8rc59A4C/dceCIbyBmg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1678881678; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=aRYf28S4DotnGRb8f0rB4+d4UYxeRJaeepAwF0O72Xg=; b=dy1BUQtEGwj6EbKN17KaISCwWP1fQOTal+Yy33Z3QzOVZXR38DMZMek9NpaACBjolONskk yM3Wunvh1rxRhrUb+W9oDW8MMwkJxAy2HWji3ZSf4dJMX9izKV05vKGomhrVXdIp99+TwK zgiI7EtFtjrI2GmNjWfmv8p8GH6WVFWSsbcP3tylFvZ+x//g3fZm930Wr6pb1sEBljRfGk Njd8ZjcYP+VgK2qPa9lQBgw/BoGccKQ7jYzQJ/MGAWuf4ryYOZCaYwp5xZSRYk+a5xPB0j 9E+e+KS9ybzftW5P9yJUN5EeSdO5zlz695USSe7e2R287zs0kY9xpcFo2JMZIQ== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1678881678; a=rsa-sha256; cv=none; b=ZhXryQBUolHyA+yQDbrgMFPceh5GLsOfcHzeLIO8vRz6la4K1Q04K2ub66R/iQbGenRquk wn5vyxPTpjDsHBmQ0z8v0J4s+ZCdysqQiqvupL9L55qVNwa08ctaT4tpV5DfFoNOxjyzGo Jc8gLAMUWS+MNsrvlN2dGZonVvZJYrmU+zU/xbbkRgHVxt9gx+pom6G4iqiLAioJe8V3Mv pn6lyEU2GyfKfbFgfLr2FRX5iiEdnwthAxarfhP5DiOil6N8lt5Xrkr9MrC/DatZfzGP5p RXbZvdRT1wABccG350MIJgMwj75jQQ6ajhDh1bqdHORQdQQHbHL9igDYaPFbqw== Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4Pc8B565Sqz1Fp5; Wed, 15 Mar 2023 12:01:17 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org ([127.0.1.44]) by gitrepo.freebsd.org (8.16.1/8.16.1) with ESMTP id 32FC1Hu0096776; Wed, 15 Mar 2023 12:01:17 GMT (envelope-from git@gitrepo.freebsd.org) Received: (from git@localhost) by gitrepo.freebsd.org (8.16.1/8.16.1/Submit) id 32FC1H31096775; Wed, 15 Mar 2023 12:01:17 GMT (envelope-from git) Date: Wed, 15 Mar 2023 12:01:17 GMT Message-Id: <202303151201.32FC1H31096775@gitrepo.freebsd.org> To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-main@FreeBSD.org From: Andrew Turner Subject: git: 6419b48f7d7f - main - Support arm64 stage2 TLB invalidation List-Id: Commit messages for the main branch of the src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-main List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-dev-commits-src-main@freebsd.org X-BeenThere: dev-commits-src-main@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: andrew X-Git-Repository: src X-Git-Refname: refs/heads/main X-Git-Reftype: branch X-Git-Commit: 6419b48f7d7f3d9048c59945d4ba9ed1b59bb87c Auto-Submitted: auto-generated X-ThisMailContainsUnwantedMimeParts: N The branch main has been updated by andrew: URL: https://cgit.FreeBSD.org/src/commit/?id=6419b48f7d7f3d9048c59945d4ba9ed1b59bb87c commit 6419b48f7d7f3d9048c59945d4ba9ed1b59bb87c Author: Andrew Turner AuthorDate: 2022-11-03 16:01:37 +0000 Commit: Andrew Turner CommitDate: 2023-03-15 11:34:32 +0000 Support arm64 stage2 TLB invalidation To invalidate stage 2 mappings on arm64 we may need to call into the hypervisor so add a function pointer that bhyve can use to implement this. Sponsored by: The FreeBSD Foundation Differential Revision: https://reviews.freebsd.org/D37254 --- sys/arm64/arm64/pmap.c | 83 ++++++++++++++++++++++++++++++++++++++---------- sys/arm64/include/pmap.h | 3 ++ 2 files changed, 69 insertions(+), 17 deletions(-) diff --git a/sys/arm64/arm64/pmap.c b/sys/arm64/arm64/pmap.c index 357584e22875..a272bf5756ae 100644 --- a/sys/arm64/arm64/pmap.c +++ b/sys/arm64/arm64/pmap.c @@ -402,6 +402,8 @@ SYSCTL_INT(_vm_pmap_vmid, OID_AUTO, epoch, CTLFLAG_RD, &vmids.asid_epoch, 0, void (*pmap_clean_stage2_tlbi)(void); void (*pmap_invalidate_vpipt_icache)(void); +void (*pmap_stage2_invalidate_range)(uint64_t, vm_offset_t, vm_offset_t, bool); +void (*pmap_stage2_invalidate_all)(uint64_t); /* * A pmap's cookie encodes an ASID and epoch number. Cookies for reserved @@ -1549,6 +1551,24 @@ pmap_s1_invalidate_page(pmap_t pmap, vm_offset_t va, bool final_only) isb(); } +static __inline void +pmap_s2_invalidate_page(pmap_t pmap, vm_offset_t va, bool final_only) +{ + PMAP_ASSERT_STAGE2(pmap); + MPASS(pmap_stage2_invalidate_range != NULL); + pmap_stage2_invalidate_range(pmap_to_ttbr0(pmap), va, va + PAGE_SIZE, + final_only); +} + +static __inline void +pmap_invalidate_page(pmap_t pmap, vm_offset_t va, bool final_only) +{ + if (pmap->pm_stage == PM_STAGE1) + pmap_s1_invalidate_page(pmap, va, final_only); + else + pmap_s2_invalidate_page(pmap, va, final_only); +} + /* * Invalidates any cached final- and optionally intermediate-level TLB entries * for the specified virtual address range in the given virtual address space. @@ -1578,6 +1598,25 @@ pmap_s1_invalidate_range(pmap_t pmap, vm_offset_t sva, vm_offset_t eva, isb(); } +static __inline void +pmap_s2_invalidate_range(pmap_t pmap, vm_offset_t sva, vm_offset_t eva, + bool final_only) +{ + PMAP_ASSERT_STAGE2(pmap); + MPASS(pmap_stage2_invalidate_range != NULL); + pmap_stage2_invalidate_range(pmap_to_ttbr0(pmap), sva, eva, final_only); +} + +static __inline void +pmap_invalidate_range(pmap_t pmap, vm_offset_t sva, vm_offset_t eva, + bool final_only) +{ + if (pmap->pm_stage == PM_STAGE1) + pmap_s1_invalidate_range(pmap, sva, eva, final_only); + else + pmap_s2_invalidate_range(pmap, sva, eva, final_only); +} + /* * Invalidates all cached intermediate- and final-level TLB entries for the * given virtual address space. @@ -1600,6 +1639,23 @@ pmap_s1_invalidate_all(pmap_t pmap) isb(); } +static __inline void +pmap_s2_invalidate_all(pmap_t pmap) +{ + PMAP_ASSERT_STAGE2(pmap); + MPASS(pmap_stage2_invalidate_all != NULL); + pmap_stage2_invalidate_all(pmap_to_ttbr0(pmap)); +} + +static __inline void +pmap_invalidate_all(pmap_t pmap) +{ + if (pmap->pm_stage == PM_STAGE1) + pmap_s1_invalidate_all(pmap); + else + pmap_s2_invalidate_all(pmap); +} + /* * Routine: pmap_extract * Function: @@ -2046,7 +2102,7 @@ _pmap_unwire_l3(pmap_t pmap, vm_offset_t va, vm_page_t m, struct spglist *free) l1pg = PHYS_TO_VM_PAGE(tl0 & ~ATTR_MASK); pmap_unwire_l3(pmap, va, l1pg, free); } - pmap_s1_invalidate_page(pmap, va, false); + pmap_invalidate_page(pmap, va, false); /* * Put page on a list so that it is released after @@ -3347,7 +3403,7 @@ pmap_remove_l3_range(pmap_t pmap, pd_entry_t l2e, vm_offset_t sva, for (l3 = pmap_l2_to_l3(&l2e, sva); sva != eva; l3++, sva += L3_SIZE) { if (!pmap_l3_valid(pmap_load(l3))) { if (va != eva) { - pmap_s1_invalidate_range(pmap, va, sva, true); + pmap_invalidate_range(pmap, va, sva, true); va = eva; } continue; @@ -3374,7 +3430,7 @@ pmap_remove_l3_range(pmap_t pmap, pd_entry_t l2e, vm_offset_t sva, * still provides access to that page. */ if (va != eva) { - pmap_s1_invalidate_range(pmap, va, + pmap_invalidate_range(pmap, va, sva, true); va = eva; } @@ -3405,7 +3461,7 @@ pmap_remove_l3_range(pmap_t pmap, pd_entry_t l2e, vm_offset_t sva, va = sva; } if (va != eva) - pmap_s1_invalidate_range(pmap, va, sva, true); + pmap_invalidate_range(pmap, va, sva, true); } /* @@ -4311,12 +4367,6 @@ havel3: * Is the specified virtual address already mapped? */ if (pmap_l3_valid(orig_l3)) { - /* - * Only allow adding new entries on stage 2 tables for now. - * This simplifies cache invalidation as we may need to call - * into EL2 to perform such actions. - */ - PMAP_ASSERT_STAGE1(pmap); /* * Wiring change, just update stats. We don't worry about * wiring PT pages as they remain resident as long as there @@ -4371,7 +4421,7 @@ havel3: if (pmap_pte_dirty(pmap, orig_l3)) vm_page_dirty(om); if ((orig_l3 & ATTR_AF) != 0) { - pmap_s1_invalidate_page(pmap, va, true); + pmap_invalidate_page(pmap, va, true); vm_page_aflag_set(om, PGA_REFERENCED); } CHANGE_PV_LIST_LOCK_TO_PHYS(&lock, opa); @@ -4386,7 +4436,7 @@ havel3: } else { KASSERT((orig_l3 & ATTR_AF) != 0, ("pmap_enter: unmanaged mapping lacks ATTR_AF")); - pmap_s1_invalidate_page(pmap, va, true); + pmap_invalidate_page(pmap, va, true); } orig_l3 = 0; } else { @@ -4439,12 +4489,11 @@ validate: * Update the L3 entry */ if (pmap_l3_valid(orig_l3)) { - PMAP_ASSERT_STAGE1(pmap); KASSERT(opa == pa, ("pmap_enter: invalid update")); if ((orig_l3 & ~ATTR_AF) != (new_l3 & ~ATTR_AF)) { /* same PA, different attributes */ orig_l3 = pmap_load_store(l3, new_l3); - pmap_s1_invalidate_page(pmap, va, true); + pmap_invalidate_page(pmap, va, true); if ((orig_l3 & ATTR_SW_MANAGED) != 0 && pmap_pte_dirty(pmap, orig_l3)) vm_page_dirty(m); @@ -5588,7 +5637,7 @@ pmap_remove_pages(pmap_t pmap) } if (lock != NULL) rw_wunlock(lock); - pmap_s1_invalidate_all(pmap); + pmap_invalidate_all(pmap); free_pv_chunk_batch(free_chunks); PMAP_UNLOCK(pmap); vm_page_free_pages_toq(&free, true); @@ -5913,7 +5962,7 @@ retry: (uintptr_t)pmap) & (Ln_ENTRIES - 1)) == 0 && (tpte & ATTR_SW_WIRED) == 0) { pmap_clear_bits(pte, ATTR_AF); - pmap_s1_invalidate_page(pmap, va, true); + pmap_invalidate_page(pmap, va, true); cleared++; } else not_cleared++; @@ -5954,7 +6003,7 @@ small_mappings: if ((tpte & ATTR_AF) != 0) { if ((tpte & ATTR_SW_WIRED) == 0) { pmap_clear_bits(pte, ATTR_AF); - pmap_s1_invalidate_page(pmap, pv->pv_va, true); + pmap_invalidate_page(pmap, pv->pv_va, true); cleared++; } else not_cleared++; diff --git a/sys/arm64/include/pmap.h b/sys/arm64/include/pmap.h index f61725e21b7c..eb72f46668c3 100644 --- a/sys/arm64/include/pmap.h +++ b/sys/arm64/include/pmap.h @@ -172,6 +172,9 @@ struct pcb *pmap_switch(struct thread *); extern void (*pmap_clean_stage2_tlbi)(void); extern void (*pmap_invalidate_vpipt_icache)(void); +extern void (*pmap_stage2_invalidate_range)(uint64_t, vm_offset_t, vm_offset_t, + bool); +extern void (*pmap_stage2_invalidate_all)(uint64_t); static inline int pmap_vmspace_copy(pmap_t dst_pmap __unused, pmap_t src_pmap __unused)