From nobody Wed Jan 24 23:50:24 2024 X-Original-To: dev-commits-src-all@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TL10w4w7vz58xfJ; Wed, 24 Jan 2024 23:50:24 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4TL10w41Grz4qqX; Wed, 24 Jan 2024 23:50:24 +0000 (UTC) (envelope-from git@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1706140224; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=zp896zkDlwLh3r7wV+C6ZWnKZXVig1/IhOr5LZh+q7k=; b=whyrFHAefaZcsU6Xrxu4/Z1vOZkRttt0zlUEjo/WMnbi4rQzew0c2eL3/Y1mt11l8abbvD xNXu6pgsxX5Sugs79u+DxAfvVatKuFITGu2xlLLyuznuDGWFVyMj6afZSK9qx/BAztup0a 17whr2up/2TkUNxrviye173gszWE7U2+1zIqpLIzhe50eIb7OullGaendo8cMBQV9/zaik GRuJwWZiwSde1svVioig3NENiRqFrBMiQrNM296ayjkPdtsZlRRUzy+MBEePoLzm1zSztw x/gllnHrs8IPeNZ5EO7YUyxq2T6ZDVZzROYCEQDJKrWDApLqEL8YiNXPpjMJkw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1706140224; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=zp896zkDlwLh3r7wV+C6ZWnKZXVig1/IhOr5LZh+q7k=; b=RUg5NecZgll4b/KDjwapmiS68do5jbC9MGV4DRShoE4iM9SFbLYyiovL8PUPmLwbhQwDfP uFkxVSSx2nwKJLp73pJ0zyRPAh8fYGentloVuxKhJMwVbr2y/6Vfm+AHL8UvBW/8hut9mC sFQ9ulxEnC7+n9AIMzQ1T/0p32pHDcfxa2CcJm9okCkePp2gj2FZcmxU+zrywQGJZ/vcsa Yu+X0ixwhQuXYrfn9ciD76vIi58g3bf7RZMtH92ZBy/rhalRSUuch2z35MQqgQkThw0nFP /sy91oLGazF8+KH1CUn/4KKTCPiNt8GGMT8yA5JxKDPQuz2n5eXmvZpNpIZVTw== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1706140224; a=rsa-sha256; cv=none; b=wRUsERVj6sPxMv5z7pwON/lChj3IYx1Wqo5hu9NkAQLYGKEWMsqmOF2O/VHerWlIeMX8YN fNAeVNWSfPIhDVr+bQltlA1RLjX6os4s1MzLv+YHbmHvSK4Shb1L5ivZIzjjYkVEkV/d/z qI+QVT3fUTjnvIHRo6qvgUe/NqoKa3mfwB/slYI0vOvyXJU/A4KkrE/CA77fObDAc8kwXD JB3G/emgQoA7qDex8zQO1O2AydV952D0IxSjTbuB1YSBbjgTQaArFqvBZHdLZTB7le/0HD EsgwuUWgyj72GbY7OiRMOPJO8rqw8LvgvwKxlQ7UG85zy0vrTnT/3CfnV09M/w== Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4TL10w372NzZft; Wed, 24 Jan 2024 23:50:24 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org ([127.0.1.44]) by gitrepo.freebsd.org (8.17.1/8.17.1) with ESMTP id 40ONoOrH018690; Wed, 24 Jan 2024 23:50:24 GMT (envelope-from git@gitrepo.freebsd.org) Received: (from git@localhost) by gitrepo.freebsd.org (8.17.1/8.17.1/Submit) id 40ONoOti018687; Wed, 24 Jan 2024 23:50:24 GMT (envelope-from git) Date: Wed, 24 Jan 2024 23:50:24 GMT Message-Id: <202401242350.40ONoOti018687@gitrepo.freebsd.org> To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-main@FreeBSD.org From: Jessica Clarke Subject: git: 6ec8bf9f3d17 - main - riscv: Convert local interrupt controller to a newbus PIC List-Id: Commit messages for all branches of the src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-all List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-dev-commits-src-all@freebsd.org X-BeenThere: dev-commits-src-all@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: jrtc27 X-Git-Repository: src X-Git-Refname: refs/heads/main X-Git-Reftype: branch X-Git-Commit: 6ec8bf9f3d17b712d657c42f74fba936dd284d04 Auto-Submitted: auto-generated The branch main has been updated by jrtc27: URL: https://cgit.FreeBSD.org/src/commit/?id=6ec8bf9f3d17b712d657c42f74fba936dd284d04 commit 6ec8bf9f3d17b712d657c42f74fba936dd284d04 Author: Jessica Clarke AuthorDate: 2024-01-24 23:49:54 +0000 Commit: Jessica Clarke CommitDate: 2024-01-24 23:49:54 +0000 riscv: Convert local interrupt controller to a newbus PIC Currently the local interrupt controller implementation is based on pre-INTRNG arm/arm64 code, using hand-rolled event code rather than INTRNG. This then interacts weirdly with the PLIC, and other future interrupt controllers like the APLIC and IMSICs in the upcoming AIA specification, since they become the root PIC despite not being the logical root. Instead, use a real newbus device for it and register it as the root PIC. This also adapts the IPI code to make use of the newly-added INTRNG generic IPI handling framework, adding a new sbi_ipi as the PIC. In future there will be alternative devices for sending IPIs that will register with higher priorities, such as the proposed AIA IMSIC and ACLINT SSWI. Reviewed by: mhorne MFC after: 1 month Differential Revision: https://reviews.freebsd.org/D35901 --- sys/conf/files.riscv | 3 +- sys/riscv/include/intr.h | 24 +--- sys/riscv/include/smp.h | 17 +-- sys/riscv/riscv/intc.c | 311 ++++++++++++++++++++++++++++++++++++++++ sys/riscv/riscv/intr_machdep.c | 274 ----------------------------------- sys/riscv/riscv/mp_machdep.c | 162 ++++++++++++--------- sys/riscv/riscv/plic.c | 49 +++++-- sys/riscv/riscv/riscv_console.c | 2 - sys/riscv/riscv/sbi.c | 43 ++++++ sys/riscv/riscv/sbi_ipi.c | 208 +++++++++++++++++++++++++++ sys/riscv/riscv/timer.c | 33 ++++- sys/riscv/riscv/trap.c | 6 +- 12 files changed, 735 insertions(+), 397 deletions(-) diff --git a/sys/conf/files.riscv b/sys/conf/files.riscv index 975f9d3f576e..b6a87c63422b 100644 --- a/sys/conf/files.riscv +++ b/sys/conf/files.riscv @@ -43,7 +43,7 @@ riscv/riscv/elf_machdep.c standard riscv/riscv/exception.S standard riscv/riscv/exec_machdep.c standard riscv/riscv/gdb_machdep.c optional gdb -riscv/riscv/intr_machdep.c standard +riscv/riscv/intc.c standard riscv/riscv/identcpu.c standard riscv/riscv/locore.S standard no-obj riscv/riscv/machdep.c standard @@ -58,6 +58,7 @@ riscv/riscv/ptrace_machdep.c standard riscv/riscv/riscv_console.c optional rcons riscv/riscv/riscv_syscon.c optional syscon riscv_syscon fdt riscv/riscv/sbi.c standard +riscv/riscv/sbi_ipi.c optional smp riscv/riscv/stack_machdep.c optional ddb | stack riscv/riscv/support.S standard riscv/riscv/swtch.S standard diff --git a/sys/riscv/include/intr.h b/sys/riscv/include/intr.h index a4e3a7093492..ad811dcbc449 100644 --- a/sys/riscv/include/intr.h +++ b/sys/riscv/include/intr.h @@ -35,32 +35,11 @@ #ifndef _MACHINE_INTR_MACHDEP_H_ #define _MACHINE_INTR_MACHDEP_H_ -#define RISCV_NIRQ 1024 - #ifndef NIRQ -#define NIRQ RISCV_NIRQ +#define NIRQ 1024 #endif -#ifdef INTRNG #include -#endif - -struct trapframe; - -int riscv_teardown_intr(void *); -int riscv_setup_intr(const char *, driver_filter_t *, driver_intr_t *, - void *, int, int, void **); -void riscv_cpu_intr(struct trapframe *); - -typedef unsigned long * riscv_intrcnt_t; - -riscv_intrcnt_t riscv_intrcnt_create(const char *); -void riscv_intrcnt_setname(riscv_intrcnt_t, const char *); - -#ifdef SMP -void riscv_setup_ipihandler(driver_filter_t *); -void riscv_unmask_ipi(void); -#endif enum { IRQ_SOFTWARE_USER, @@ -75,7 +54,6 @@ enum { IRQ_EXTERNAL_SUPERVISOR, IRQ_EXTERNAL_HYPERVISOR, IRQ_EXTERNAL_MACHINE, - INTC_NIRQS }; #endif /* !_MACHINE_INTR_MACHDEP_H_ */ diff --git a/sys/riscv/include/smp.h b/sys/riscv/include/smp.h index 8bafc1ac6a3a..863f0bbd6d21 100644 --- a/sys/riscv/include/smp.h +++ b/sys/riscv/include/smp.h @@ -37,14 +37,15 @@ #include -#define IPI_AST (1 << 0) -#define IPI_PREEMPT (1 << 1) -#define IPI_RENDEZVOUS (1 << 2) -#define IPI_STOP (1 << 3) -#define IPI_STOP_HARD (1 << 4) -#define IPI_HARDCLOCK (1 << 5) - -#define INTR_IPI_COUNT 1 +enum { + IPI_AST, + IPI_PREEMPT, + IPI_RENDEZVOUS, + IPI_STOP, + IPI_STOP_HARD, + IPI_HARDCLOCK, + INTR_IPI_COUNT +}; void ipi_all_but_self(u_int ipi); void ipi_cpu(int cpu, u_int ipi); diff --git a/sys/riscv/riscv/intc.c b/sys/riscv/riscv/intc.c new file mode 100644 index 000000000000..399bb05bbcfe --- /dev/null +++ b/sys/riscv/riscv/intc.c @@ -0,0 +1,311 @@ +/*- + * Copyright (c) 2015-2017 Ruslan Bukin + * All rights reserved. + * Copyright (c) 2021 Jessica Clarke + * + * Portions of this software were developed by SRI International and the + * University of Cambridge Computer Laboratory under DARPA/AFRL contract + * FA8750-10-C-0237 ("CTSRD"), as part of the DARPA CRASH research programme. + * + * Portions of this software were developed by the University of Cambridge + * Computer Laboratory as part of the CTSRD Project, with support from the + * UK Higher Education Innovation Fund (HEIF). + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + */ + +#include + +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include +#include +#include +#include +#include + +#include + +#include +#include +#include + +#include "pic_if.h" + +#define INTC_NIRQS 16 + +struct intc_irqsrc { + struct intr_irqsrc isrc; + u_int irq; +}; + +struct intc_softc { + device_t dev; + struct intc_irqsrc isrcs[INTC_NIRQS]; +}; + +static int intc_intr(void *arg); + +static phandle_t +intc_ofw_find(device_t dev, uint32_t hartid) +{ + phandle_t node; + pcell_t reg; + + node = OF_finddevice("/cpus"); + if (node == -1) { + device_printf(dev, "Can't find cpus node\n"); + return ((phandle_t)-1); + } + + for (node = OF_child(node); node != 0; node = OF_peer(node)) { + if (!ofw_bus_node_status_okay(node)) + continue; + + if (!ofw_bus_node_is_compatible(node, "riscv")) + continue; + + if (OF_searchencprop(node, "reg", ®, sizeof(reg)) == -1) + continue; + + if (reg == hartid) + break; + } + + if (node == 0) { + device_printf(dev, "Can't find boot cpu node\n"); + return ((phandle_t)-1); + } + + for (node = OF_child(node); node != 0; node = OF_peer(node)) { + if (!ofw_bus_node_status_okay(node)) + continue; + + if (ofw_bus_node_is_compatible(node, "riscv,cpu-intc")) + break; + } + + if (node == 0) { + device_printf(dev, + "Can't find boot cpu local interrupt controller\n"); + return ((phandle_t)-1); + } + + return (node); +} + +static void +intc_identify(driver_t *driver, device_t parent) +{ + device_t dev; + phandle_t node; + + if (device_find_child(parent, "intc", -1) != NULL) + return; + + node = intc_ofw_find(parent, PCPU_GET(hart)); + if (node == -1) + return; + + dev = simplebus_add_device(parent, node, 0, "intc", -1, NULL); + if (dev == NULL) + device_printf(parent, "Can't add intc child\n"); +} + +static int +intc_probe(device_t dev) +{ + device_set_desc(dev, "RISC-V Local Interrupt Controller"); + + return (BUS_PROBE_NOWILDCARD); +} + +static int +intc_attach(device_t dev) +{ + struct intc_irqsrc *isrcs; + struct intc_softc *sc; + struct intr_pic *pic; + const char *name; + phandle_t xref; + u_int flags; + int i, error; + + sc = device_get_softc(dev); + sc->dev = dev; + + name = device_get_nameunit(dev); + xref = OF_xref_from_node(ofw_bus_get_node(dev)); + + isrcs = sc->isrcs; + for (i = 0; i < INTC_NIRQS; i++) { + isrcs[i].irq = i; + flags = i == IRQ_SOFTWARE_SUPERVISOR + ? INTR_ISRCF_IPI : INTR_ISRCF_PPI; + error = intr_isrc_register(&isrcs[i].isrc, sc->dev, flags, + "%s,%u", name, i); + if (error != 0) { + device_printf(dev, "Can't register interrupt %d\n", i); + return (error); + } + } + + pic = intr_pic_register(sc->dev, xref); + if (pic == NULL) + return (ENXIO); + + return (intr_pic_claim_root(sc->dev, xref, intc_intr, sc)); +} + +static void +intc_disable_intr(device_t dev, struct intr_irqsrc *isrc) +{ + u_int irq; + + irq = ((struct intc_irqsrc *)isrc)->irq; + if (irq >= INTC_NIRQS) + panic("%s: Unsupported IRQ %u", __func__, irq); + + csr_clear(sie, 1ul << irq); +} + +static void +intc_enable_intr(device_t dev, struct intr_irqsrc *isrc) +{ + u_int irq; + + irq = ((struct intc_irqsrc *)isrc)->irq; + if (irq >= INTC_NIRQS) + panic("%s: Unsupported IRQ %u", __func__, irq); + + csr_set(sie, 1ul << irq); +} + +static int +intc_map_intr(device_t dev, struct intr_map_data *data, + struct intr_irqsrc **isrcp) +{ + struct intr_map_data_fdt *daf; + struct intc_softc *sc; + + sc = device_get_softc(dev); + + if (data->type != INTR_MAP_DATA_FDT) + return (ENOTSUP); + + daf = (struct intr_map_data_fdt *)data; + if (daf->ncells != 1 || daf->cells[0] >= INTC_NIRQS) + return (EINVAL); + + *isrcp = &sc->isrcs[daf->cells[0]].isrc; + + return (0); +} + +static int +intc_setup_intr(device_t dev, struct intr_irqsrc *isrc, + struct resource *res, struct intr_map_data *data) +{ + if (isrc->isrc_flags & INTR_ISRCF_PPI) + CPU_SET(PCPU_GET(cpuid), &isrc->isrc_cpu); + + return (0); +} + +#ifdef SMP +static void +intc_init_secondary(device_t dev) +{ + struct intc_softc *sc; + struct intr_irqsrc *isrc; + u_int cpu, irq; + + sc = device_get_softc(dev); + cpu = PCPU_GET(cpuid); + + /* Unmask attached interrupts */ + for (irq = 0; irq < INTC_NIRQS; irq++) { + isrc = &sc->isrcs[irq].isrc; + if (intr_isrc_init_on_cpu(isrc, cpu)) + intc_enable_intr(dev, isrc); + } +} +#endif + +static int +intc_intr(void *arg) +{ + struct trapframe *frame; + struct intc_softc *sc; + uint64_t active_irq; + struct intc_irqsrc *src; + + sc = arg; + frame = curthread->td_intr_frame; + + KASSERT((frame->tf_scause & SCAUSE_INTR) != 0, + ("%s: not an interrupt frame", __func__)); + + active_irq = frame->tf_scause & SCAUSE_CODE; + + if (active_irq >= INTC_NIRQS) + return (FILTER_HANDLED); + + src = &sc->isrcs[active_irq]; + if (intr_isrc_dispatch(&src->isrc, frame) != 0) { + intc_disable_intr(sc->dev, &src->isrc); + device_printf(sc->dev, "Stray irq %lu disabled\n", + active_irq); + } + + return (FILTER_HANDLED); +} + +static device_method_t intc_methods[] = { + /* Device interface */ + DEVMETHOD(device_identify, intc_identify), + DEVMETHOD(device_probe, intc_probe), + DEVMETHOD(device_attach, intc_attach), + + /* Interrupt controller interface */ + DEVMETHOD(pic_disable_intr, intc_disable_intr), + DEVMETHOD(pic_enable_intr, intc_enable_intr), + DEVMETHOD(pic_map_intr, intc_map_intr), + DEVMETHOD(pic_setup_intr, intc_setup_intr), +#ifdef SMP + DEVMETHOD(pic_init_secondary, intc_init_secondary), +#endif + + DEVMETHOD_END +}; + +DEFINE_CLASS_0(intc, intc_driver, intc_methods, sizeof(struct intc_softc)); +EARLY_DRIVER_MODULE(intc, ofwbus, intc_driver, 0, 0, + BUS_PASS_INTERRUPT + BUS_PASS_ORDER_FIRST); diff --git a/sys/riscv/riscv/intr_machdep.c b/sys/riscv/riscv/intr_machdep.c deleted file mode 100644 index 6c00b29c120d..000000000000 --- a/sys/riscv/riscv/intr_machdep.c +++ /dev/null @@ -1,274 +0,0 @@ -/*- - * Copyright (c) 2015-2017 Ruslan Bukin - * All rights reserved. - * - * Portions of this software were developed by SRI International and the - * University of Cambridge Computer Laboratory under DARPA/AFRL contract - * FA8750-10-C-0237 ("CTSRD"), as part of the DARPA CRASH research programme. - * - * Portions of this software were developed by the University of Cambridge - * Computer Laboratory as part of the CTSRD Project, with support from the - * UK Higher Education Innovation Fund (HEIF). - * - * Redistribution and use in source and binary forms, with or without - * modification, are permitted provided that the following conditions - * are met: - * 1. Redistributions of source code must retain the above copyright - * notice, this list of conditions and the following disclaimer. - * 2. Redistributions in binary form must reproduce the above copyright - * notice, this list of conditions and the following disclaimer in the - * documentation and/or other materials provided with the distribution. - * - * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND - * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE - * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE - * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE - * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL - * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS - * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) - * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT - * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY - * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF - * SUCH DAMAGE. - */ - -#include -#include -#include -#include -#include -#include -#include -#include -#include - -#include -#include -#include -#include -#include -#include -#include - -#include -#include -#include - -#ifdef SMP -#include -#endif - -void intr_irq_handler(struct trapframe *tf); - -struct intc_irqsrc { - struct intr_irqsrc isrc; - u_int irq; -}; - -struct intc_irqsrc isrcs[INTC_NIRQS]; - -static void -riscv_mask_irq(void *source) -{ - int irq; - - irq = (int)(uintptr_t)source; - - switch (irq) { - case IRQ_TIMER_SUPERVISOR: - csr_clear(sie, SIE_STIE); - break; - case IRQ_SOFTWARE_USER: - csr_clear(sie, SIE_USIE); - break; - case IRQ_SOFTWARE_SUPERVISOR: - csr_clear(sie, SIE_SSIE); - break; - default: - panic("Unknown irq %d\n", irq); - } -} - -static void -riscv_unmask_irq(void *source) -{ - int irq; - - irq = (int)(uintptr_t)source; - - switch (irq) { - case IRQ_TIMER_SUPERVISOR: - csr_set(sie, SIE_STIE); - break; - case IRQ_SOFTWARE_USER: - csr_set(sie, SIE_USIE); - break; - case IRQ_SOFTWARE_SUPERVISOR: - csr_set(sie, SIE_SSIE); - break; - default: - panic("Unknown irq %d\n", irq); - } -} - -int -riscv_setup_intr(const char *name, driver_filter_t *filt, - void (*handler)(void*), void *arg, int irq, int flags, void **cookiep) -{ - struct intr_irqsrc *isrc; - int error; - - if (irq < 0 || irq >= INTC_NIRQS) - panic("%s: unknown intr %d", __func__, irq); - - isrc = &isrcs[irq].isrc; - if (isrc->isrc_event == NULL) { - error = intr_event_create(&isrc->isrc_event, isrc, 0, irq, - riscv_mask_irq, riscv_unmask_irq, NULL, NULL, "int%d", irq); - if (error) - return (error); - riscv_unmask_irq((void*)(uintptr_t)irq); - } - - error = intr_event_add_handler(isrc->isrc_event, name, - filt, handler, arg, intr_priority(flags), flags, cookiep); - if (error) { - printf("Failed to setup intr: %d\n", irq); - return (error); - } - - return (0); -} - -int -riscv_teardown_intr(void *ih) -{ - - /* TODO */ - - return (0); -} - -void -riscv_cpu_intr(struct trapframe *frame) -{ - struct intr_irqsrc *isrc; - int active_irq; - - KASSERT((frame->tf_scause & SCAUSE_INTR) != 0, - ("riscv_cpu_intr: wrong frame passed")); - - active_irq = frame->tf_scause & SCAUSE_CODE; - - CTR3(KTR_TRAP, "%s: irq=%d, umode=%d", __func__, active_irq, - TRAPF_USERMODE(frame)); - - switch (active_irq) { - case IRQ_SOFTWARE_USER: - case IRQ_SOFTWARE_SUPERVISOR: - case IRQ_TIMER_SUPERVISOR: - critical_enter(); - isrc = &isrcs[active_irq].isrc; - if (intr_isrc_dispatch(isrc, frame) != 0) - printf("stray interrupt %d\n", active_irq); - critical_exit(); - break; - case IRQ_EXTERNAL_SUPERVISOR: - intr_irq_handler(frame); - break; - } -} - -#ifdef SMP -void -riscv_setup_ipihandler(driver_filter_t *filt) -{ - - riscv_setup_intr("ipi", filt, NULL, NULL, IRQ_SOFTWARE_SUPERVISOR, - INTR_TYPE_MISC, NULL); -} - -void -riscv_unmask_ipi(void) -{ - - csr_set(sie, SIE_SSIE); -} - -/* Sending IPI */ -static void -ipi_send(struct pcpu *pc, int ipi) -{ - u_long mask; - - CTR3(KTR_SMP, "%s: cpu: %d, ipi: %x", __func__, pc->pc_cpuid, ipi); - - atomic_set_32(&pc->pc_pending_ipis, ipi); - mask = (1 << pc->pc_hart); - - sbi_send_ipi(&mask); - - CTR1(KTR_SMP, "%s: sent", __func__); -} - -void -ipi_all_but_self(u_int ipi) -{ - cpuset_t other_cpus; - - other_cpus = all_cpus; - CPU_CLR(PCPU_GET(cpuid), &other_cpus); - - CTR2(KTR_SMP, "%s: ipi: %x", __func__, ipi); - ipi_selected(other_cpus, ipi); -} - -void -ipi_cpu(int cpu, u_int ipi) -{ - cpuset_t cpus; - - CPU_ZERO(&cpus); - CPU_SET(cpu, &cpus); - - ipi_send(cpuid_to_pcpu[cpu], ipi); -} - -void -ipi_selected(cpuset_t cpus, u_int ipi) -{ - struct pcpu *pc; - u_long mask; - - CTR1(KTR_SMP, "ipi_selected: ipi: %x", ipi); - - mask = 0; - STAILQ_FOREACH(pc, &cpuhead, pc_allcpu) { - if (CPU_ISSET(pc->pc_cpuid, &cpus)) { - CTR3(KTR_SMP, "%s: pc: %p, ipi: %x\n", __func__, pc, - ipi); - atomic_set_32(&pc->pc_pending_ipis, ipi); - mask |= (1 << pc->pc_hart); - } - } - sbi_send_ipi(&mask); -} -#endif - -/* Interrupt machdep initialization routine. */ -static void -intc_init(void *dummy __unused) -{ - int error; - int i; - - for (i = 0; i < INTC_NIRQS; i++) { - isrcs[i].irq = i; - error = intr_isrc_register(&isrcs[i].isrc, NULL, - 0, "intc,%u", i); - if (error != 0) - printf("Can't register interrupt %d\n", i); - } -} - -SYSINIT(intc_init, SI_SUB_INTR, SI_ORDER_MIDDLE, intc_init, NULL); diff --git a/sys/riscv/riscv/mp_machdep.c b/sys/riscv/riscv/mp_machdep.c index 2c144e457b4e..be5c50485bf4 100644 --- a/sys/riscv/riscv/mp_machdep.c +++ b/sys/riscv/riscv/mp_machdep.c @@ -83,7 +83,11 @@ static device_identify_t riscv64_cpu_identify; static device_probe_t riscv64_cpu_probe; static device_attach_t riscv64_cpu_attach; -static int ipi_handler(void *); +static void ipi_ast(void *); +static void ipi_hardclock(void *); +static void ipi_preempt(void *); +static void ipi_rendezvous(void *); +static void ipi_stop(void *); extern uint32_t boot_hart; extern cpuset_t all_harts; @@ -191,8 +195,13 @@ release_aps(void *dummy __unused) if (mp_ncpus == 1) return; - /* Setup the IPI handler */ - riscv_setup_ipihandler(ipi_handler); + /* Setup the IPI handlers */ + intr_ipi_setup(IPI_AST, "ast", ipi_ast, NULL); + intr_ipi_setup(IPI_PREEMPT, "preempt", ipi_preempt, NULL); + intr_ipi_setup(IPI_RENDEZVOUS, "rendezvous", ipi_rendezvous, NULL); + intr_ipi_setup(IPI_STOP, "stop", ipi_stop, NULL); + intr_ipi_setup(IPI_STOP_HARD, "stop hard", ipi_stop, NULL); + intr_ipi_setup(IPI_HARDCLOCK, "hardclock", ipi_hardclock, NULL); atomic_store_rel_int(&aps_ready, 1); @@ -244,17 +253,14 @@ init_secondary(uint64_t hart) pcpup->pc_curthread = pcpup->pc_idlethread; schedinit_ap(); - /* Enable software interrupts */ - riscv_unmask_ipi(); + /* Setup and enable interrupts */ + intr_pic_init_secondary(); #ifndef EARLY_AP_STARTUP /* Start per-CPU event timers. */ cpu_initclocks_ap(); #endif - /* Enable external (PLIC) interrupts */ - csr_set(sie, SIE_SEIE); - /* Activate this hart in the kernel pmap. */ CPU_SET_ATOMIC(hart, &kernel_pmap->pm_active); @@ -308,74 +314,59 @@ smp_after_idle_runnable(void *arg __unused) SYSINIT(smp_after_idle_runnable, SI_SUB_SMP, SI_ORDER_ANY, smp_after_idle_runnable, NULL); -static int -ipi_handler(void *arg) +static void +ipi_ast(void *dummy __unused) +{ + CTR0(KTR_SMP, "IPI_AST"); +} + +static void +ipi_preempt(void *dummy __unused) +{ + CTR1(KTR_SMP, "%s: IPI_PREEMPT", __func__); + sched_preempt(curthread); +} + +static void +ipi_rendezvous(void *dummy __unused) { - u_int ipi_bitmap; - u_int cpu, ipi; - int bit; + CTR0(KTR_SMP, "IPI_RENDEZVOUS"); + smp_rendezvous_action(); +} + +static void +ipi_stop(void *dummy __unused) +{ + u_int cpu; - csr_clear(sip, SIP_SSIP); + CTR0(KTR_SMP, "IPI_STOP"); cpu = PCPU_GET(cpuid); + savectx(&stoppcbs[cpu]); - mb(); - - ipi_bitmap = atomic_readandclear_int(PCPU_PTR(pending_ipis)); - if (ipi_bitmap == 0) - return (FILTER_HANDLED); - - while ((bit = ffs(ipi_bitmap))) { - bit = (bit - 1); - ipi = (1 << bit); - ipi_bitmap &= ~ipi; - - mb(); - - switch (ipi) { - case IPI_AST: - CTR0(KTR_SMP, "IPI_AST"); - break; - case IPI_PREEMPT: - CTR1(KTR_SMP, "%s: IPI_PREEMPT", __func__); - sched_preempt(curthread); - break; - case IPI_RENDEZVOUS: - CTR0(KTR_SMP, "IPI_RENDEZVOUS"); - smp_rendezvous_action(); - break; - case IPI_STOP: - case IPI_STOP_HARD: - CTR0(KTR_SMP, (ipi == IPI_STOP) ? "IPI_STOP" : "IPI_STOP_HARD"); - savectx(&stoppcbs[cpu]); - - /* Indicate we are stopped */ - CPU_SET_ATOMIC(cpu, &stopped_cpus); - - /* Wait for restart */ - while (!CPU_ISSET(cpu, &started_cpus)) - cpu_spinwait(); - - CPU_CLR_ATOMIC(cpu, &started_cpus); - CPU_CLR_ATOMIC(cpu, &stopped_cpus); - CTR0(KTR_SMP, "IPI_STOP (restart)"); - - /* - * The kernel debugger might have set a breakpoint, - * so flush the instruction cache. - */ - fence_i(); - break; - case IPI_HARDCLOCK: - CTR1(KTR_SMP, "%s: IPI_HARDCLOCK", __func__); - hardclockintr(); - break; - default: - panic("Unknown IPI %#0x on cpu %d", ipi, curcpu); - } - } + /* Indicate we are stopped */ + CPU_SET_ATOMIC(cpu, &stopped_cpus); + + /* Wait for restart */ + while (!CPU_ISSET(cpu, &started_cpus)) + cpu_spinwait(); - return (FILTER_HANDLED); + CPU_CLR_ATOMIC(cpu, &started_cpus); + CPU_CLR_ATOMIC(cpu, &stopped_cpus); + CTR0(KTR_SMP, "IPI_STOP (restart)"); + + /* + * The kernel debugger might have set a breakpoint, + * so flush the instruction cache. + */ + fence_i(); +} + +static void +ipi_hardclock(void *dummy __unused) +{ + CTR1(KTR_SMP, "%s: IPI_HARDCLOCK", __func__); + hardclockintr(); } struct cpu_group * @@ -574,3 +565,34 @@ cpu_mp_setmaxid(void) } } } + +void +ipi_all_but_self(u_int ipi) +{ + cpuset_t other_cpus; + + other_cpus = all_cpus; + CPU_CLR(PCPU_GET(cpuid), &other_cpus); + + CTR2(KTR_SMP, "%s: ipi: %x", __func__, ipi); + intr_ipi_send(other_cpus, ipi); +} + +void +ipi_cpu(int cpu, u_int ipi) +{ + cpuset_t cpus; + + CPU_ZERO(&cpus); + CPU_SET(cpu, &cpus); + + CTR3(KTR_SMP, "%s: cpu: %d, ipi: %x", __func__, cpu, ipi); + intr_ipi_send(cpus, ipi); +} + +void +ipi_selected(cpuset_t cpus, u_int ipi) +{ + CTR1(KTR_SMP, "ipi_selected: ipi: %x", ipi); + intr_ipi_send(cpus, ipi); +} diff --git a/sys/riscv/riscv/plic.c b/sys/riscv/riscv/plic.c index 1e8a03127038..406aff5bbff0 100644 --- a/sys/riscv/riscv/plic.c +++ b/sys/riscv/riscv/plic.c @@ -91,16 +91,18 @@ struct plic_context { struct plic_softc { device_t dev; - struct resource * intc_res; + struct resource *mem_res; + struct resource *irq_res; + void *ih; struct plic_irqsrc isrcs[PLIC_MAX_IRQS]; struct plic_context contexts[MAXCPU]; int ndev; }; #define RD4(sc, reg) \ - bus_read_4(sc->intc_res, (reg)) + bus_read_4(sc->mem_res, (reg)) #define WR4(sc, reg, val) \ - bus_write_4(sc->intc_res, (reg), (val)) + bus_write_4(sc->mem_res, (reg), (val)) static u_int plic_irq_cpu; @@ -276,9 +278,9 @@ plic_attach(device_t dev) /* Request memory resources */ rid = 0; - sc->intc_res = bus_alloc_resource_any(dev, SYS_RES_MEMORY, &rid, + sc->mem_res = bus_alloc_resource_any(dev, SYS_RES_MEMORY, &rid, RF_ACTIVE); - if (sc->intc_res == NULL) { + if (sc->mem_res == NULL) { device_printf(dev, "Error: could not allocate memory resources\n"); return (ENXIO); @@ -302,7 +304,7 @@ plic_attach(device_t dev) * *** 503 LINES SKIPPED ***