From nobody Tue Sep 24 20:16:31 2024 X-Original-To: dev-commits-src-all@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4XCrjW69XQz5YFSN; Tue, 24 Sep 2024 20:16:31 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R11" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4XCrjW5fFhz3xDn; Tue, 24 Sep 2024 20:16:31 +0000 (UTC) (envelope-from git@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1727208991; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=RWNpCs818cHnEyENJOI594KBiDwio9B5/aek2ApYAvE=; b=sOeUNDYmvRB8COBu0rw1MbVX5dS1KqqcLNH0qxfkxZymY7Q4fZtRkTOxqJTotvAium8P44 nvxTl7ejU9h07QTjydXAOoEDwIDIWalPGb2oBQt8QFIpjkisPXwl3qwrwFkiAlxibqkcEa ndoYQsMzMwca+9gjFOHklq2m8T73MeZo1fGM4M4F3sIW78gS1e5yCFyhHyi15Hm9jFiRmw VkwF4rJ4Q5dvDIb5GHbWV5IzVV2dE8xdL7kezmqfZMYqvFnGdryUWDYmUVAuqPNkm8ysN8 WBfI27yjG+wf83ZCPa1anKOcsXSeIQMlmjnUtnlgFVx0iBAJHtXR3l2oNVrsFg== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1727208991; a=rsa-sha256; cv=none; b=xr/2yzgj1hU4sg/QZAQXiGSWjRW+yLZ8Z8UvmfgjBudIr4FgBbdgTrso5PkVdofHt1irBA 1gMxCLtvTMVaYb1HzBqU3diJIz8y2kRjuFTSZWPFAzQd0AzLStPrIxjAlAVYKKgYmy9SFC RghvkH9TdSipI5eP+t1baiKshCWjYoYeSKbTWTvVQrKcDI+A2OkxCGWvZ82gQ52XaQPoCh RGJ4QP6Gxct60m9llWL4qqk9zqiINQ7yD5gCVCtieFhkhnbgbEL/oyxs7BdLsedZ/6KP88 poFZBXJF6O4aM3QOHt/SBZGdWVNTjNbnoQAQDQK4E5iCecbHI+i5R6elow7ozQ== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1727208991; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=RWNpCs818cHnEyENJOI594KBiDwio9B5/aek2ApYAvE=; b=IdAb27fxd4OWJWRHKUeO2AYeYDPT7X8NVgBQNoC8qEM1zBw6CJTJnBDm0IQ7MpE2cybfUN wB0uOieSa9vw7497DZxu7gsfWc4/z4EXnVx4e4w2HYWBGWZACl1ejZy/6tWAvEEx1fry0n aAirqXddYMxXsDE7DtCjM7TqxVHgZ94qEplrKe74MZfB+CF54/3HedzWCsXlHpE3LUT6ro HEMVW8gX0sPWSwcuF+4BqREriV1cth7cUg/mAZWKxPEQa46lxHWPp83R7AM6lTrdzFY5a1 McrIfIX4ruxWLwa1NhjYax4b9RAYUSlS9oR4mLAOJq1qmpBT3XlzqD+iLl4PrQ== Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4XCrjW5FCwzjW4; Tue, 24 Sep 2024 20:16:31 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org ([127.0.1.44]) by gitrepo.freebsd.org (8.18.1/8.18.1) with ESMTP id 48OKGVxU000213; Tue, 24 Sep 2024 20:16:31 GMT (envelope-from git@gitrepo.freebsd.org) Received: (from git@localhost) by gitrepo.freebsd.org (8.18.1/8.18.1/Submit) id 48OKGVGE000210; Tue, 24 Sep 2024 20:16:31 GMT (envelope-from git) Date: Tue, 24 Sep 2024 20:16:31 GMT Message-Id: <202409242016.48OKGVGE000210@gitrepo.freebsd.org> To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-main@FreeBSD.org From: John Baldwin Subject: git: 1b3fa1ac36d1 - main - nvmft: Defer datamove operations to a pool of taskqueue threads List-Id: Commit messages for all branches of the src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-all List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: dev-commits-src-all@freebsd.org Sender: owner-dev-commits-src-all@FreeBSD.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: jhb X-Git-Repository: src X-Git-Refname: refs/heads/main X-Git-Reftype: branch X-Git-Commit: 1b3fa1ac36d1b48a906dc98d9d9f22447a213d47 Auto-Submitted: auto-generated The branch main has been updated by jhb: URL: https://cgit.FreeBSD.org/src/commit/?id=1b3fa1ac36d1b48a906dc98d9d9f22447a213d47 commit 1b3fa1ac36d1b48a906dc98d9d9f22447a213d47 Author: John Baldwin AuthorDate: 2024-09-24 20:16:11 +0000 Commit: John Baldwin CommitDate: 2024-09-24 20:16:11 +0000 nvmft: Defer datamove operations to a pool of taskqueue threads Some block devices may request datamove operations from an ithread context while holding locks. Queue datamove operations to a taskqueue backed by a thread pool to safely permit blocking allocations, etc. in datamove handling. Reviewed by: asomers Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D46551 --- sys/dev/nvmf/controller/ctl_frontend_nvmf.c | 52 +++++++++++++++++++++++++- sys/dev/nvmf/controller/nvmft_qpair.c | 57 +++++++++++++++++++++++++++++ sys/dev/nvmf/controller/nvmft_var.h | 5 +++ 3 files changed, 112 insertions(+), 2 deletions(-) diff --git a/sys/dev/nvmf/controller/ctl_frontend_nvmf.c b/sys/dev/nvmf/controller/ctl_frontend_nvmf.c index a203bb1c90a6..bc061947a9a0 100644 --- a/sys/dev/nvmf/controller/ctl_frontend_nvmf.c +++ b/sys/dev/nvmf/controller/ctl_frontend_nvmf.c @@ -19,7 +19,9 @@ #include #include #include +#include #include +#include #include #include @@ -31,8 +33,10 @@ #include #include +#include #include #include +#include /* * Store pointers to the capsule and qpair in the two pointer members @@ -47,6 +51,9 @@ static int nvmft_ioctl(struct cdev *cdev, u_long cmd, caddr_t data, int flag, struct thread *td); static int nvmft_shutdown(void); +extern struct ctl_softc *control_softc; + +static struct taskqueue *nvmft_taskq; static TAILQ_HEAD(, nvmft_port) nvmft_ports; static struct sx nvmft_ports_lock; @@ -458,8 +465,8 @@ nvmft_datamove_in(struct ctl_nvmeio *ctnio, struct nvmft_qpair *qp, ctl_datamove_done((union ctl_io *)ctnio, true); } -static void -nvmft_datamove(union ctl_io *io) +void +nvmft_handle_datamove(union ctl_io *io) { struct nvmf_capsule *nc; struct nvmft_qpair *qp; @@ -478,6 +485,35 @@ nvmft_datamove(union ctl_io *io) nvmft_datamove_out(&io->nvmeio, qp, nc); } +void +nvmft_abort_datamove(union ctl_io *io) +{ + io->io_hdr.port_status = 1; + io->io_hdr.flags |= CTL_FLAG_ABORT; + ctl_datamove_done(io, true); +} + +static void +nvmft_datamove(union ctl_io *io) +{ + struct nvmft_qpair *qp; + + qp = NVMFT_QP(io); + nvmft_qpair_datamove(qp, io); +} + +void +nvmft_enqueue_task(struct task *task) +{ + taskqueue_enqueue(nvmft_taskq, task); +} + +void +nvmft_drain_task(struct task *task) +{ + taskqueue_drain(nvmft_taskq, task); +} + static void hip_add(uint64_t pair[2], uint64_t addend) { @@ -561,6 +597,17 @@ end: static int nvmft_init(void) { + int error; + + nvmft_taskq = taskqueue_create("nvmft", M_WAITOK, + taskqueue_thread_enqueue, &nvmft_taskq); + error = taskqueue_start_threads_in_proc(&nvmft_taskq, mp_ncpus, PWAIT, + control_softc->ctl_proc, "nvmft"); + if (error != 0) { + taskqueue_free(nvmft_taskq); + return (error); + } + TAILQ_INIT(&nvmft_ports); sx_init(&nvmft_ports_lock, "nvmft ports"); return (0); @@ -1115,6 +1162,7 @@ nvmft_shutdown(void) if (!TAILQ_EMPTY(&nvmft_ports)) return (EBUSY); + taskqueue_free(nvmft_taskq); sx_destroy(&nvmft_ports_lock); return (0); } diff --git a/sys/dev/nvmf/controller/nvmft_qpair.c b/sys/dev/nvmf/controller/nvmft_qpair.c index 6cb3ebd76884..e66d98f38225 100644 --- a/sys/dev/nvmf/controller/nvmft_qpair.c +++ b/sys/dev/nvmf/controller/nvmft_qpair.c @@ -34,6 +34,9 @@ struct nvmft_qpair { uint16_t sqtail; volatile u_int qp_refs; /* Internal references on 'qp'. */ + struct task datamove_task; + STAILQ_HEAD(, ctl_io_hdr) datamove_queue; + struct mtx lock; char name[16]; @@ -41,6 +44,7 @@ struct nvmft_qpair { static int _nvmft_send_generic_error(struct nvmft_qpair *qp, struct nvmf_capsule *nc, uint8_t sc_status); +static void nvmft_datamove_task(void *context, int pending); static void nvmft_qpair_error(void *arg, int error) @@ -114,6 +118,8 @@ nvmft_qpair_init(enum nvmf_trtype trtype, strlcpy(qp->name, name, sizeof(qp->name)); mtx_init(&qp->lock, "nvmft qp", NULL, MTX_DEF); qp->cids = BITSET_ALLOC(NUM_CIDS, M_NVMFT, M_WAITOK | M_ZERO); + STAILQ_INIT(&qp->datamove_queue); + TASK_INIT(&qp->datamove_task, 0, nvmft_datamove_task, qp); qp->qp = nvmf_allocate_qpair(trtype, true, handoff, nvmft_qpair_error, qp, nvmft_receive_capsule, qp); @@ -131,14 +137,25 @@ nvmft_qpair_init(enum nvmf_trtype trtype, void nvmft_qpair_shutdown(struct nvmft_qpair *qp) { + STAILQ_HEAD(, ctl_io_hdr) datamove_queue; struct nvmf_qpair *nq; + union ctl_io *io; + STAILQ_INIT(&datamove_queue); mtx_lock(&qp->lock); nq = qp->qp; qp->qp = NULL; + STAILQ_CONCAT(&datamove_queue, &qp->datamove_queue); mtx_unlock(&qp->lock); if (nq != NULL && refcount_release(&qp->qp_refs)) nvmf_free_qpair(nq); + + while (!STAILQ_EMPTY(&datamove_queue)) { + io = (union ctl_io *)STAILQ_FIRST(&datamove_queue); + STAILQ_REMOVE_HEAD(&datamove_queue, links); + nvmft_abort_datamove(io); + } + nvmft_drain_task(&qp->datamove_task); } void @@ -359,3 +376,43 @@ nvmft_finish_accept(struct nvmft_qpair *qp, rsp.status_code_specific.success.cntlid = htole16(ctrlr->cntlid); return (nvmft_send_connect_response(qp, &rsp)); } + +void +nvmft_qpair_datamove(struct nvmft_qpair *qp, union ctl_io *io) +{ + bool enqueue_task; + + mtx_lock(&qp->lock); + if (qp->qp == NULL) { + mtx_unlock(&qp->lock); + nvmft_abort_datamove(io); + return; + } + enqueue_task = STAILQ_EMPTY(&qp->datamove_queue); + STAILQ_INSERT_TAIL(&qp->datamove_queue, &io->io_hdr, links); + mtx_unlock(&qp->lock); + if (enqueue_task) + nvmft_enqueue_task(&qp->datamove_task); +} + +static void +nvmft_datamove_task(void *context, int pending __unused) +{ + struct nvmft_qpair *qp = context; + union ctl_io *io; + bool abort; + + mtx_lock(&qp->lock); + while (!STAILQ_EMPTY(&qp->datamove_queue)) { + io = (union ctl_io *)STAILQ_FIRST(&qp->datamove_queue); + STAILQ_REMOVE_HEAD(&qp->datamove_queue, links); + abort = (qp->qp == NULL); + mtx_unlock(&qp->lock); + if (abort) + nvmft_abort_datamove(io); + else + nvmft_handle_datamove(io); + mtx_lock(&qp->lock); + } + mtx_unlock(&qp->lock); +} diff --git a/sys/dev/nvmf/controller/nvmft_var.h b/sys/dev/nvmf/controller/nvmft_var.h index fc1f86754382..4fda297c8a85 100644 --- a/sys/dev/nvmf/controller/nvmft_var.h +++ b/sys/dev/nvmf/controller/nvmft_var.h @@ -110,6 +110,10 @@ void nvmft_populate_active_nslist(struct nvmft_port *np, uint32_t nsid, void nvmft_dispatch_command(struct nvmft_qpair *qp, struct nvmf_capsule *nc, bool admin); void nvmft_terminate_commands(struct nvmft_controller *ctrlr); +void nvmft_abort_datamove(union ctl_io *io); +void nvmft_handle_datamove(union ctl_io *io); +void nvmft_drain_task(struct task *task); +void nvmft_enqueue_task(struct task *task); /* nvmft_controller.c */ void nvmft_controller_error(struct nvmft_controller *ctrlr, @@ -138,6 +142,7 @@ struct nvmft_qpair *nvmft_qpair_init(enum nvmf_trtype trtype, void nvmft_qpair_shutdown(struct nvmft_qpair *qp); void nvmft_qpair_destroy(struct nvmft_qpair *qp); struct nvmft_controller *nvmft_qpair_ctrlr(struct nvmft_qpair *qp); +void nvmft_qpair_datamove(struct nvmft_qpair *qp, union ctl_io *io); uint16_t nvmft_qpair_id(struct nvmft_qpair *qp); const char *nvmft_qpair_name(struct nvmft_qpair *qp); void nvmft_command_completed(struct nvmft_qpair *qp,