From nobody Fri Jul 07 20:07:31 2023 X-Original-To: dev-commits-src-main@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4QyPZW58wjz4lhpD; Fri, 7 Jul 2023 20:07:31 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4QyPZW4dW2z40PG; Fri, 7 Jul 2023 20:07:31 +0000 (UTC) (envelope-from git@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1688760451; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=0wq7v3O3WK3wq6+TcAqe/E7p07QyrOIlqE3zNBmu+MU=; b=bfuNAFgorAc0rI6l+tek6JmGse6KCY7G9gLtNpedY5g/VLYE13/Poij1cIvSF+tytGVxQp eFM/D8aQSs80hf8l1MU/s0as1/A8xPaR5y9Vx2z7fm6nio1+6rkKuYDe0+tN4yyGxH0L4f nQ518iNuXDm5vXXz4XsNfrn9Mtof8YR3UpMJlzWbdPLmBMTWLwchufhvgF/5/7yYv9yaHd ELOGjgiidCqfW+33vfK/rcbqqA2pbs9bW7xtWxudLfAWiuzF5UA3c26t2drYw7wAFxES+R MFESACxmshCtkEVw5FQRGPzK8V8Yr4Yp+nAQ+ww+nkoJ8wjI1YHJUDSHEmLdew== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1688760451; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=0wq7v3O3WK3wq6+TcAqe/E7p07QyrOIlqE3zNBmu+MU=; b=inNB0IfLf18Z41jiaWRweqTZiN2HcRePbHabk9zF/fB7RliXsMyUtUcLX+VlLNOLyn+HPp Vj3HnJgtMIqRw8lX3uPR+ZMBuXYbWKJ0SzqGkT8/PAc2meK6dvtUxjKwTwm70Yr42cUnYt NVC/4kuzYyCN005EohBqFyDgUXjCyNctcqG6yLkEKSUf72uIYHRSiKKjfYg8IS1mY/x6M5 Va5adDEqsRk+q4rw+2R13KznNTLiwHl5uH/jiuY2xoWQkyCMy02Fhq6EFW6f/i7lpFQQtC plPBtiyEjoXTEcy4j/LmKczPtqfiIgKGEA38BbZXSBz4QEMaRuHatRpcyGcQ3A== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1688760451; a=rsa-sha256; cv=none; b=rqYm6OnbQ2CRqV+JM5B4h8334H8YaC+7LO+pHFQeyR0jLSrKR0FLBfb2gYFrkTLpjO98ul jbTvuHlvrOxF1pPomtzMO9dwYZMjgUvT+0924/4EZtwwmlCaSLWGD6jRS34vYUfMuV4qAI TbUEi7RjfPeLvVLpIEsDiclYkN6fs7l7e7KR+aArVFdv0Ff/z1IQ5v7EssY5nzjikzLiuA ORwHFDYsIiZKQlycdUiieaECIiYPftIEeeiSI489s18bTtcr2RrodWHJQcKkBBQf5snq1w 3HgD4O9Bccg+Wbfv3dcmgzYmCMBMKD5EtqhRihEoLckSO7D8RVBZ6/5fYkyLKA== Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4QyPZW3hpZzrfG; Fri, 7 Jul 2023 20:07:31 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org ([127.0.1.44]) by gitrepo.freebsd.org (8.17.1/8.17.1) with ESMTP id 367K7Vxg030356; Fri, 7 Jul 2023 20:07:31 GMT (envelope-from git@gitrepo.freebsd.org) Received: (from git@localhost) by gitrepo.freebsd.org (8.17.1/8.17.1/Submit) id 367K7VR0030355; Fri, 7 Jul 2023 20:07:31 GMT (envelope-from git) Date: Fri, 7 Jul 2023 20:07:31 GMT Message-Id: <202307072007.367K7VR0030355@gitrepo.freebsd.org> To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-main@FreeBSD.org From: John Baldwin Subject: git: 2329393c615c - main - amd64: Use __seg_gs to implement per-CPU data accesses. List-Id: Commit messages for the main branch of the src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-main List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-dev-commits-src-main@freebsd.org X-BeenThere: dev-commits-src-main@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: jhb X-Git-Repository: src X-Git-Refname: refs/heads/main X-Git-Reftype: branch X-Git-Commit: 2329393c615cf61c19b0d7bd1ca18663e02789ce Auto-Submitted: auto-generated X-ThisMailContainsUnwantedMimeParts: N The branch main has been updated by jhb: URL: https://cgit.FreeBSD.org/src/commit/?id=2329393c615cf61c19b0d7bd1ca18663e02789ce commit 2329393c615cf61c19b0d7bd1ca18663e02789ce Author: John Baldwin AuthorDate: 2023-07-07 20:06:55 +0000 Commit: John Baldwin CommitDate: 2023-07-07 20:06:55 +0000 amd64: Use __seg_gs to implement per-CPU data accesses. This makes use of the alternate address space support in both GCC and clang to access per-CPU data as accesses relative to GS:. The original motivation for this is that it quiets verbose warnings from GCC 12. However, this version is also much easier to read and allows the compiler to generate better code (e.g. the compiler can use a GS: memory operand directly in other instructions such as IMUL and CMP rather than always MOVing to a temporary register). The one caveat is that the current approach is very inefficient at -O0 since the compiler expects to load the 0 base offset from a global variable instead of assuming it is 0 (even with the const). Reviewed by: kib Differential Revision: https://reviews.freebsd.org/D40647 --- sys/amd64/include/pcpu.h | 69 +++++++++++++++++++++++++++++++++++++++++++----- 1 file changed, 63 insertions(+), 6 deletions(-) diff --git a/sys/amd64/include/pcpu.h b/sys/amd64/include/pcpu.h index 523cf105e1c7..dc78a07b83e9 100644 --- a/sys/amd64/include/pcpu.h +++ b/sys/amd64/include/pcpu.h @@ -113,18 +113,74 @@ _Static_assert(sizeof(struct monitorbuf) == 128, "2x cache line"); #define MONITOR_STOPSTATE_RUNNING 0 #define MONITOR_STOPSTATE_STOPPED 1 -/* - * Evaluates to the byte offset of the per-cpu variable name. - */ -#define __pcpu_offset(name) \ - __offsetof(struct pcpu, name) - /* * Evaluates to the type of the per-cpu variable name. */ #define __pcpu_type(name) \ __typeof(((struct pcpu *)0)->name) +#ifdef __SEG_GS +#define get_pcpu() __extension__ ({ \ + static struct pcpu __seg_gs *__pc = 0; \ + \ + __pc->pc_prvspace; \ +}) + +/* + * Evaluates to the address of the per-cpu variable name. + */ +#define __PCPU_PTR(name) __extension__ ({ \ + struct pcpu *__pc = get_pcpu(); \ + \ + &__pc->name; \ +}) + +/* + * Evaluates to the value of the per-cpu variable name. + */ +#define __PCPU_GET(name) __extension__ ({ \ + static struct pcpu __seg_gs *__pc = 0; \ + \ + __pc->name; \ +}) + +/* + * Adds the value to the per-cpu counter name. The implementation + * must be atomic with respect to interrupts. + */ +#define __PCPU_ADD(name, val) do { \ + static struct pcpu __seg_gs *__pc = 0; \ + __pcpu_type(name) __val; \ + \ + __val = (val); \ + if (sizeof(__val) == 1 || sizeof(__val) == 2 || \ + sizeof(__val) == 4 || sizeof(__val) == 8) { \ + __pc->name += __val; \ + } else \ + *__PCPU_PTR(name) += __val; \ +} while (0) + +/* + * Sets the value of the per-cpu variable name to value val. + */ +#define __PCPU_SET(name, val) { \ + static struct pcpu __seg_gs *__pc = 0; \ + __pcpu_type(name) __val; \ + \ + __val = (val); \ + if (sizeof(__val) == 1 || sizeof(__val) == 2 || \ + sizeof(__val) == 4 || sizeof(__val) == 8) { \ + __pc->name = __val; \ + } else \ + *__PCPU_PTR(name) = __val; \ +} while (0) +#else /* !__SEG_GS */ +/* + * Evaluates to the byte offset of the per-cpu variable name. + */ +#define __pcpu_offset(name) \ + __offsetof(struct pcpu, name) + /* * Evaluates to the address of the per-cpu variable name. */ @@ -210,6 +266,7 @@ _Static_assert(sizeof(struct monitorbuf) == 128, "2x cache line"); : "m" (*(struct pcpu *)(__pcpu_offset(pc_prvspace)))); \ __pc; \ }) +#endif /* !__SEG_GS */ #define PCPU_GET(member) __PCPU_GET(pc_ ## member) #define PCPU_ADD(member, val) __PCPU_ADD(pc_ ## member, val)