From nobody Sat Feb 01 09:02:27 2025 X-Original-To: dev-commits-src-main@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4YlRbm58vTz5mbWH; Sat, 01 Feb 2025 09:02:28 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R11" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4YlRbm135bz41Zg; Sat, 01 Feb 2025 09:02:28 +0000 (UTC) (envelope-from git@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1738400548; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=uGQR9EHXSIJazg5cMTbsao2YL0DmmHRfiSK+e25c0N4=; b=XOsx9Zmbj+TPHSeDDqeOaGMolUJ+LYnrdMEW+YSCjm4/q2Yg1ki7rqUWmIHqmgTS1LolnY x6888RLefFi2boTsFokVjRWGI+0c+n28Is4hFZLvAOi/TuqUBlWxTq2HHq9Uu9Eg3HIBMz XqInFFocX6zVZ06ynxgXCdyV7+AYwHRa/erO0SAPJTU7jUITvh2XN4k9b5Bf7DQnw2EwKG BwAy5F0blSH257RmCbpXgUD6I/BBarW0SlqL8QpIMj2RghHnoRJzTL2ZroNBgJKLtnW/d2 cf0Uyvp9pzTkkYvYzygtXmEzIRGmABg8M4Sdex8aOU9ipDx7y7e2I5xlwddLAQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1738400548; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=uGQR9EHXSIJazg5cMTbsao2YL0DmmHRfiSK+e25c0N4=; b=ygVmFgWqXImlUsVSUQ1RetXuQkvhRDQ7FIqxA5EmkM6pV8ss7y04uz+Co4DK1QZtyUrqAj EOtqu4OZ5alY9ZNTUq5JlgJtDQgtNxm4RaZ8tzKHK17l0Ca9FX2kPFB8eqv9Jo9tp1T/PR Laxns2D+KUPQitT2taPetTPjxQZb2gMwRatcNCs9EHvCADBMW3hFyHGNC1/DAlH06l7oKH uOHi7U29CXgb2F9Oo/mTvqGXitQzNZFebZELdvetwT+uxyVirIyhz7VnIe7nz9fniaUD5z Diimv9xNfiyiaFv6/ptbGNquLngZ0JxAlZxJBqnJJZCb9BRva7bnBSw6vOrEuw== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1738400548; a=rsa-sha256; cv=none; b=ErqwThIH2kN+fSSceET3GsV0bclg8wD1lIvWzF7CzdkeIl/acQ5NQhPT+29ksBHxHKusTS WaNktKL/Z93+i1LDhfgGbjd8yiJUuox63ijuUtp0Wk+HafUFMRiOnLcs3cr7lCGljrLNk/ F7AUonC025t/UDrVqgJUHDoeuQj2kFvuEOUvTyNCZndDcpk8s80ltfYXLftOS8/nbUvolO qSkh5r69w1msrGI+CFVSOEmcEAxC2iAysdWKn4iKkR5t5GACXKwcVKGH8RyfxnlRGuf390 Wdb65TICP7IMUEQNg1AnqCusOVNCpMnHOtyBs3xwMaTJ5SGkIl5Th7Nuu1uMIA== ARC-Authentication-Results: i=1; mx1.freebsd.org; none Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4YlRbm0by8zxnQ; Sat, 01 Feb 2025 09:02:28 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org ([127.0.1.44]) by gitrepo.freebsd.org (8.18.1/8.18.1) with ESMTP id 51192RSD048485; Sat, 1 Feb 2025 09:02:27 GMT (envelope-from git@gitrepo.freebsd.org) Received: (from git@localhost) by gitrepo.freebsd.org (8.18.1/8.18.1/Submit) id 51192RCx048482; Sat, 1 Feb 2025 09:02:27 GMT (envelope-from git) Date: Sat, 1 Feb 2025 09:02:27 GMT Message-Id: <202502010902.51192RCx048482@gitrepo.freebsd.org> To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-main@FreeBSD.org From: Gleb Smirnoff Subject: git: c62ae124cc78 - main - rpc: limited multithread support for svc_nl List-Id: Commit messages for the main branch of the src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-main List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: dev-commits-src-main@freebsd.org Sender: owner-dev-commits-src-main@FreeBSD.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: glebius X-Git-Repository: src X-Git-Refname: refs/heads/main X-Git-Reftype: branch X-Git-Commit: c62ae124cc783bc43ae9fa539e58683c46cc35c1 Auto-Submitted: auto-generated The branch main has been updated by glebius: URL: https://cgit.FreeBSD.org/src/commit/?id=c62ae124cc783bc43ae9fa539e58683c46cc35c1 commit c62ae124cc783bc43ae9fa539e58683c46cc35c1 Author: Gleb Smirnoff AuthorDate: 2025-02-01 01:03:18 +0000 Commit: Gleb Smirnoff CommitDate: 2025-02-01 09:00:28 +0000 rpc: limited multithread support for svc_nl The rpc(3) itself was not designed with multithreading in mind, but we can actually achieve some parallelism without modifying the library and the framework. This transport will allow to process RPCs in threads, with some hacks on the application side (documented in code). We make reentrable only one method - SVC_REPLY(). Reading and parsing of incoming calls is still done synchronously. But the actual processing of the calls can be offloaded to a thread, and once finished the thread can safely execute svc_sendreply() and the reply would be sent with the correct xid. Differential Revision: https://reviews.freebsd.org/D48569 --- include/rpc/svc.h | 7 ++++ lib/libc/rpc/svc_nl.c | 100 +++++++++++++++++++++++++++++++++++++++++++------- 2 files changed, 93 insertions(+), 14 deletions(-) diff --git a/include/rpc/svc.h b/include/rpc/svc.h index deb626ad7c1d..99e2fcee4121 100644 --- a/include/rpc/svc.h +++ b/include/rpc/svc.h @@ -461,6 +461,13 @@ extern SVCXPRT *svcunixfd_create(int, u_int, u_int); */ extern SVCXPRT *svc_nl_create(const char *); +/* + * Arguments to SVC_CONTROL(svc_nl) + */ +enum { + SVCNL_GET_XIDKEY = 1, /* obtain pthread specific key for xid */ +}; + /* * Memory based rpc (for speed check and testing) */ diff --git a/lib/libc/rpc/svc_nl.c b/lib/libc/rpc/svc_nl.c index f866acaf3015..5cce44d9d98f 100644 --- a/lib/libc/rpc/svc_nl.c +++ b/lib/libc/rpc/svc_nl.c @@ -28,6 +28,7 @@ #include #include #include +#include #include #include @@ -37,6 +38,7 @@ #include #include "rpc_com.h" +#include "libc_private.h" /* * RPC server to serve a kernel RPC client(s) over netlink(4). See clnt_nl.c @@ -54,6 +56,7 @@ static bool_t svc_nl_reply(SVCXPRT *, struct rpc_msg *); static enum xprt_stat svc_nl_stat(SVCXPRT *); static bool_t svc_nl_getargs(SVCXPRT *, xdrproc_t, void *); static bool_t svc_nl_freeargs(SVCXPRT *, xdrproc_t, void *); +static bool_t svc_nl_control(SVCXPRT *, const u_int, void *); static struct xp_ops nl_ops = { .xp_recv = svc_nl_recv, @@ -63,11 +66,15 @@ static struct xp_ops nl_ops = { .xp_freeargs = svc_nl_freeargs, .xp_destroy = svc_nl_destroy, }; +static struct xp_ops2 nl_ops2 = { + .xp_control = svc_nl_control, +}; struct nl_softc { struct snl_state snl; XDR xdrs; struct nlmsghdr *hdr; + pthread_key_t xidkey; size_t mlen; enum xprt_stat stat; uint32_t xid; @@ -108,9 +115,15 @@ svc_nl_create(const char *service) sc->stat = XPRT_IDLE, sc->family = family; + if (__isthreaded && + (pthread_key_create(&sc->xidkey, NULL) != 0 || + pthread_setspecific(sc->xidkey, &sc->xid) != 0)) + goto fail; + xprt->xp_fd = sc->snl.fd, xprt->xp_p1 = sc, xprt->xp_ops = &nl_ops, + xprt->xp_ops2 = &nl_ops2, xprt->xp_rtaddr = (struct netbuf){ .maxlen = sizeof(struct sockaddr_nl), .len = sizeof(struct sockaddr_nl), @@ -208,26 +221,62 @@ svc_nl_recv(SVCXPRT *xprt, struct rpc_msg *msg) return (FALSE); } +/* + * Reenterable reply method. Note that both the softc and xprt are declared + * const. The qualifier for xprt is commented out to match the library + * prototype. If doing any substantial changes to the function please + * temporarily uncomment the const for xprt and check your changes. + * + * Applications that want to use svc_nl_reply in a spawned thread context + * should do the following hacks in self: + * 1) - Create xprt with svc_nl_create() with libc in threaded mode, e.g. + * at least one pthread_create() shall happen before svc_nl_create(). + * - After xprt creation query it for the pthread_key_t with the + * SVCNL_GET_XIDKEY control and save this key. + * 2) In the RPC function body that wants to become multithreaded: + * - Make a copy of the arguments and of the xid with help of + * pthread_getspecific() using the key. + * - pthread_create() the worker function, pointing it at the copy of + * arguments and xid. + * - return FALSE, so that RPC generated code doesn't do anything. + * 3) In the spawned thread: + * - Use arguments provided in the copy by the parent. + * - Allocate appropriately typed result on stack. + * - *** do the actual work *** + * - Populate the on-stack result same way as pointed result is populated + * in a regular RPC function. + * - Point the thread specific storage to the copy of xid provided by the + * parent with help of pthread_setspecific(). + * - Call svc_sendreply() just like the rpcgen(1) generated code does. + * + * If all done correctly svc_nl_reply() will use thread specific xid for + * a call that was processed asynchronously and most recent xid when entered + * synchronously. So you can make only some methods of your application + * reentrable, keeping others as is. + */ + static bool_t -svc_nl_reply(SVCXPRT *xprt, struct rpc_msg *msg) +svc_nl_reply(/* const */ SVCXPRT *xprt, struct rpc_msg *msg) { - struct nl_softc *sc = xprt->xp_p1; + const struct nl_softc *sc = xprt->xp_p1; struct snl_state snl; struct snl_writer nw; + XDR xdrs; struct nlattr *body; bool_t rv; - msg->rm_xid = sc->xid; + msg->rm_xid = __isthreaded ? + *(uint32_t *)pthread_getspecific(sc->xidkey) : + sc->xid; if (__predict_false(!snl_clone(&snl, &sc->snl))) return (FALSE); - snl_init_writer(&sc->snl, &nw); + snl_init_writer(&snl, &nw); snl_create_genl_msg_request(&nw, sc->family, RPCNL_REPLY); snl_add_msg_attr_u32(&nw, RPCNL_REPLY_GROUP, sc->group); body = snl_reserve_msg_attr_raw(&nw, RPCNL_REPLY_BODY, RPC_MAXDATASIZE); - xdrmem_create(&sc->xdrs, (char *)(body + 1), RPC_MAXDATASIZE, - XDR_ENCODE); + xdrmem_create(&xdrs, (char *)(body + 1), RPC_MAXDATASIZE, XDR_ENCODE); if (msg->rm_reply.rp_stat == MSG_ACCEPTED && msg->rm_reply.rp_acpt.ar_stat == SUCCESS) { @@ -240,28 +289,28 @@ svc_nl_reply(SVCXPRT *xprt, struct rpc_msg *msg) msg->acpted_rply.ar_results.proc = (xdrproc_t) xdr_void; msg->acpted_rply.ar_results.where = NULL; - pos = xdr_getpos(&sc->xdrs); - if (!xdr_replymsg(&sc->xdrs, msg) || - !SVCAUTH_WRAP(&SVC_AUTH(xprt), &sc->xdrs, xdr_proc, + pos = xdr_getpos(&xdrs); + if (!xdr_replymsg(&xdrs, msg) || + !SVCAUTH_WRAP(&SVC_AUTH(xprt), &xdrs, xdr_proc, xdr_where)) { - xdr_setpos(&sc->xdrs, pos); + xdr_setpos(&xdrs, pos); rv = FALSE; } else rv = TRUE; } else - rv = xdr_replymsg(&sc->xdrs, msg); + rv = xdr_replymsg(&xdrs, msg); if (rv) { /* snl_finalize_msg() really doesn't work for us */ - body->nla_len = sizeof(struct nlattr) + xdr_getpos(&sc->xdrs); + body->nla_len = sizeof(struct nlattr) + xdr_getpos(&xdrs); nw.hdr->nlmsg_len = ((char *)body - (char *)nw.hdr) + body->nla_len; nw.hdr->nlmsg_type = sc->family; nw.hdr->nlmsg_flags = NLM_F_REQUEST; - nw.hdr->nlmsg_seq = sc->xid; + nw.hdr->nlmsg_seq = msg->rm_xid; if (write(xprt->xp_fd, nw.hdr, nw.hdr->nlmsg_len) != nw.hdr->nlmsg_len) - DIE(sc); + DIE(__DECONST(struct nl_softc *, sc)); } snl_free(&snl); @@ -269,6 +318,29 @@ svc_nl_reply(SVCXPRT *xprt, struct rpc_msg *msg) return (rv); } +static bool_t +svc_nl_control(SVCXPRT *xprt, const u_int req, void *v) +{ + struct nl_softc *sc = xprt->xp_p1; + + switch (req) { + case SVCNL_GET_XIDKEY: + if (!__isthreaded) { + /* + * Report to application that it had created xprt not + * in threaded mode, but definitly plans to use it with + * threads. If it tries so, it would very likely crash. + */ + errno = EDOOFUS; + DIE(sc); + }; + *(pthread_key_t *)v = sc->xidkey; + return (TRUE); + default: + return (FALSE); + } +} + static enum xprt_stat svc_nl_stat(SVCXPRT *xprt) {