From nobody Mon Nov 25 21:08:45 2024 X-Original-To: dev-commits-src-main@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Xxyx936jhz5fCVj; Mon, 25 Nov 2024 21:08:45 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R10" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Xxyx92fHHz4typ; Mon, 25 Nov 2024 21:08:45 +0000 (UTC) (envelope-from git@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1732568925; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=joFD2TKePZcc5Qlfd1GhPvsonTdu0J5i6yXzENeZCX0=; b=HWLJbZvVvQBxty7/eNrg/54jhHcZxhrD1KXenKC6iT4QqkxeL8HaOlrxTlUVvy/EIEaplU RUPNOmbczoJRl4a6WTH1bcatyhN9fr2m0v8b0lYyWuYHMnUY/tsTED8wXQ4HeLIJpIH6Lm ZFHEvEIQOTmTTlO7u25hPKWzeFMQ3ziLxjuwKDkJrlG8R7jcZyeISG/5Wxp/9gIRyjzc7b Rsw8jGWYlL7cQdBBH3pTxuSJqQz0jt9/bICsmzOrNzz39YU04A/A6Enm5jBXfByHUZlMOa KYN1fCiSbyZ7LV43YjN187Wx4M+ojsTIDOHZaldldH/wLVtKCSa4vFcXdKvH0A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1732568925; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=joFD2TKePZcc5Qlfd1GhPvsonTdu0J5i6yXzENeZCX0=; b=T1GeK9wUO7qmoo/R426k+Y2ILYX57BVvDMp4YL+d/Y1UvBIAXZneLbvnbelU36PdYUAxpQ YhmP3v7qkd6tQzsRfA6hTnP0mYhQ7axcTkkoaxymgE8xQCpXLn7dNnmi/sUyi56OQ+3OMR 2BR8sFUDX00flw+v8C1KZG4dzSV8KwhaS6jG510jED7Sd/MrPFVP+qlNkNzgw5A1RMMcYT D/u4wCYxYW7J2Pqyrxb1RCcwKtLA3qrsBbsV4uZL4NHX/38TgQZQNlm5UcGcabbnFmwSqS XK9mzIJGtUM+EanSHknU7I0odyPz0/f+orLYvkO7GiJP2cYFbpirxnsZbHPQPQ== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1732568925; a=rsa-sha256; cv=none; b=NzXTD5BC3+DzGau2GCFQegEF9PXGZq/yMOAYo5Eisg3x3xz0ZOf7tcZLAeb6f4grIbNa53 4vVgl5L6xel5mQeWhqL/HRGNolMHSdSkjbvWZQ0xUkTVyn0N0sYFuPcs9Zl/Rj6eRYk5a2 EQ/VRLRNwjSjh0l5RCryWt3PEfryz0UdY1L2f3qTNrJOa8zA4yR7Ps/GkDhnjfF8AcxumJ v0czH1KzShfZW676hioSq1Y6y2icSLz9MLbMbjdzsi2WFT8srkUP56zR/bmugnOKD1izol UIWcJQwguUxBHKx0YndiTxy6H2eYB0oX6ZNukC3lLRyH2oC+qWwdYuuWamM1VQ== Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4Xxyx926wKzwKG; Mon, 25 Nov 2024 21:08:45 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org ([127.0.1.44]) by gitrepo.freebsd.org (8.18.1/8.18.1) with ESMTP id 4APL8jF9011349; Mon, 25 Nov 2024 21:08:45 GMT (envelope-from git@gitrepo.freebsd.org) Received: (from git@localhost) by gitrepo.freebsd.org (8.18.1/8.18.1/Submit) id 4APL8j2T011346; Mon, 25 Nov 2024 21:08:45 GMT (envelope-from git) Date: Mon, 25 Nov 2024 21:08:45 GMT Message-Id: <202411252108.4APL8j2T011346@gitrepo.freebsd.org> To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-main@FreeBSD.org From: Mitchell Horne Subject: git: 4ab2a84e0924 - main - riscv: dcache flush hooks List-Id: Commit messages for the main branch of the src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-main List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: dev-commits-src-main@freebsd.org Sender: owner-dev-commits-src-main@FreeBSD.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: mhorne X-Git-Repository: src X-Git-Refname: refs/heads/main X-Git-Reftype: branch X-Git-Commit: 4ab2a84e092467be04d3c85ab2ceebc1e722004c Auto-Submitted: auto-generated The branch main has been updated by mhorne: URL: https://cgit.FreeBSD.org/src/commit/?id=4ab2a84e092467be04d3c85ab2ceebc1e722004c commit 4ab2a84e092467be04d3c85ab2ceebc1e722004c Author: Mitchell Horne AuthorDate: 2024-11-21 18:11:30 +0000 Commit: Mitchell Horne CommitDate: 2024-11-25 21:08:03 +0000 riscv: dcache flush hooks Cache management operations were, for a long time, unspecified by the RISC-V ISA, and thus these functions have been no-ops. To cope, hardware with non-coherent I/O has implemented custom cache flush mechanisms, either in the form of custom instructions or special device registers. Additionally, the RISC-V CMO extension is ratified and these official instructions will start to show up in hardware eventually. Therefore, a method is needed to select the dcache management routines at runtime. Add a simple set of function hooks, as well as a routine to install them and specify the minimum dcache line size. The first consumer will be the non-standard cache management instructions for T-HEAD CPUs. The unused I-cache variables and macros are removed. Sponsored by: The FreeBSD Foundation Differential Revision: https://reviews.freebsd.org/D47454 --- sys/conf/files.riscv | 1 + sys/riscv/include/cpufunc.h | 42 +++++++++++++++++++++++++++++------ sys/riscv/riscv/cache.c | 53 +++++++++++++++++++++++++++++++++++++++++++++ sys/riscv/riscv/machdep.c | 17 --------------- sys/riscv/riscv/pmap.c | 2 +- 5 files changed, 90 insertions(+), 25 deletions(-) diff --git a/sys/conf/files.riscv b/sys/conf/files.riscv index d325502e03ee..915bce34603d 100644 --- a/sys/conf/files.riscv +++ b/sys/conf/files.riscv @@ -36,6 +36,7 @@ riscv/riscv/bus_machdep.c standard riscv/riscv/bus_space_asm.S standard riscv/riscv/busdma_bounce.c standard riscv/riscv/busdma_machdep.c standard +riscv/riscv/cache.c standard riscv/riscv/clock.c standard riscv/riscv/copyinout.S standard riscv/riscv/cpufunc_asm.S standard diff --git a/sys/riscv/include/cpufunc.h b/sys/riscv/include/cpufunc.h index 4521e1605d8f..8f5b87d24ce3 100644 --- a/sys/riscv/include/cpufunc.h +++ b/sys/riscv/include/cpufunc.h @@ -44,6 +44,8 @@ breakpoint(void) #ifdef _KERNEL +#include + #include static __inline register_t @@ -107,16 +109,42 @@ sfence_vma_page(uintptr_t addr) #define rdinstret() csr_read64(instret) #define rdhpmcounter(n) csr_read64(hpmcounter##n) +/* Cache hooks. */ + extern int64_t dcache_line_size; -extern int64_t icache_line_size; -#define cpu_dcache_wbinv_range(a, s) -#define cpu_dcache_inv_range(a, s) -#define cpu_dcache_wb_range(a, s) +typedef void (*cache_op_t)(vm_offset_t start, vm_size_t size); + +struct riscv_cache_ops { + cache_op_t dcache_wbinv_range; + cache_op_t dcache_inv_range; + cache_op_t dcache_wb_range; +}; + +extern struct riscv_cache_ops cache_ops; + +static __inline void +cpu_dcache_wbinv_range(vm_offset_t addr, vm_size_t size) +{ + if (cache_ops.dcache_wbinv_range != NULL) + cache_ops.dcache_wbinv_range(addr, size); +} + +static __inline void +cpu_dcache_inv_range(vm_offset_t addr, vm_size_t size) +{ + if (cache_ops.dcache_inv_range != NULL) + cache_ops.dcache_inv_range(addr, size); +} + +static __inline void +cpu_dcache_wb_range(vm_offset_t addr, vm_size_t size) +{ + if (cache_ops.dcache_wb_range != NULL) + cache_ops.dcache_wb_range(addr, size); +} -#define cpu_idcache_wbinv_range(a, s) -#define cpu_icache_sync_range(a, s) -#define cpu_icache_sync_range_checked(a, s) +void riscv_cache_install_hooks(struct riscv_cache_ops *, u_int); #define cpufunc_nullop() riscv_nullop() diff --git a/sys/riscv/riscv/cache.c b/sys/riscv/riscv/cache.c new file mode 100644 index 000000000000..edbaebc9abd6 --- /dev/null +++ b/sys/riscv/riscv/cache.c @@ -0,0 +1,53 @@ +/*- + * SPDX-License-Identifier: BSD-2-Clause + * + * Copyright (c) 2024 The FreeBSD Foundation + * + * This software was developed by Mitchell Horne under + * sponsorship from the FreeBSD Foundation. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + */ + +#include +#include +#include + +#include + +struct riscv_cache_ops __read_frequently cache_ops; + +int64_t __read_frequently dcache_line_size; /* The minimum D cache line size */ + +static bool cache_initialized; + +void +riscv_cache_install_hooks(struct riscv_cache_ops *newops, u_int line_size) +{ + if (cache_initialized) + panic("cache hooks already installed!"); + + bcopy(newops, &cache_ops, sizeof(cache_ops)); + dcache_line_size = line_size; + + cache_initialized = true; +} diff --git a/sys/riscv/riscv/machdep.c b/sys/riscv/riscv/machdep.c index eda7ebc32bbf..c5da4832dd36 100644 --- a/sys/riscv/riscv/machdep.c +++ b/sys/riscv/riscv/machdep.c @@ -113,10 +113,6 @@ int cold = 1; struct kva_md_info kmi; -int64_t dcache_line_size; /* The minimum D cache line size */ -int64_t icache_line_size; /* The minimum I cache line size */ -int64_t idcache_line_size; /* The minimum cache line size */ - #define BOOT_HART_INVALID 0xffffffff uint32_t boot_hart = BOOT_HART_INVALID; /* The hart we booted on. */ @@ -329,17 +325,6 @@ try_load_dtb(caddr_t kmdp) } #endif -static void -cache_setup(void) -{ - - /* TODO */ - - dcache_line_size = 0; - icache_line_size = 0; - idcache_line_size = 0; -} - /* * Fake up a boot descriptor table. */ @@ -550,8 +535,6 @@ initriscv(struct riscv_bootparams *rvbp) /* Do basic tuning, hz etc */ init_param1(); - cache_setup(); - #ifdef FDT /* * XXX: Unconditionally exclude the lowest 2MB of physical memory, as diff --git a/sys/riscv/riscv/pmap.c b/sys/riscv/riscv/pmap.c index 69eb36c2cd4c..e11adba0d832 100644 --- a/sys/riscv/riscv/pmap.c +++ b/sys/riscv/riscv/pmap.c @@ -5013,7 +5013,7 @@ pmap_change_attr_locked(vm_offset_t va, vm_size_t size, int mode) if (anychanged) { pmap_invalidate_range(kernel_pmap, base, tmpva); if (mode == VM_MEMATTR_UNCACHEABLE) - cpu_dcache_wbinv_range((void *)base, size); + cpu_dcache_wbinv_range(base, size); } return (error);