From nobody Wed Jun 05 20:04:03 2024 X-Original-To: dev-commits-src-main@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4VvdhN0Jrjz5M3Mg; Wed, 05 Jun 2024 20:04:04 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4VvdhM5zmWz4jHb; Wed, 5 Jun 2024 20:04:03 +0000 (UTC) (envelope-from git@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1717617843; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=4o++ehul8RtFicrGyxP+rNauncusG1yCJkw85m+vLV0=; b=q3/OYQ1412nqkPctkGPgUOW3HQNa2CY7JynFjtpURnkL43uSLUhqczKgZvnHpKEWDxQIVO 4lKa0iMqAsID/0BQIE5xKe0JfGVpyUwDxxqL6hSI5ZV/w6ode1dE83vpXtMqL5+IFOKof0 omfVAmApWZZc1w9pXbgEHIj1SfN6NXpp8+HbyxS8RLxjmY/dJPRblxNQEAUtiA8KPyOjuG M6u4huRLZ4PJ8tARKvhzY/h55izaKxCpBf1OULPYCGm3I9NnZXPIK7Z5pK/HwraEfubtcu Ho3Ve3UtgmZsutoPq3HBXLWL0rGfOFqj3tOspQCqcVc8uSzA9WR9KX5ublbHOg== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1717617843; a=rsa-sha256; cv=none; b=h9UiqwVQYDYNk5kvhhzK/r+eXaCbVVfnVX7OpRC/evHoRwggBN5mmFWtaL50BcvYGl1uvJ Pzyc1vCMyP50mXslos15wwsh3OhhZylLedpkLPSmJ55l5kGn8y0lt8oNE8Isxi7Ww8pynI XsIo6nkLsK/+iUxD7+5/xhFl77sZ7btukUSD5RpC+Ig5C4eV0EFb7KtHzc1jzhcc5kZzoz lAi3vXInWuX04fXkIVFlTAK1WBwTV6+Fg1I/37JNhU6KvqxEeEmflZcMWUr8ZvzNxWm1Hw eBlZ16SgSbzE4iAxU/mVaGGtgit65gu/F5bkLFxqzKCkKhX+yJBC6N0gw0MnLw== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1717617843; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=4o++ehul8RtFicrGyxP+rNauncusG1yCJkw85m+vLV0=; b=mjMYFARAgHGwLcECe8LtIBfgTeNy6KTSfOWgeqtYAhIiPREzWWIYepvr4QolL28MVabUGR ram88dBvjv0kstUG8d1F1UWTkC12+NOaYoVo6pHzGlSSbfL7e0T8GrW8foC8FsZU+Kh8qY V1Pb5wI+8tckx0uscexJ0FwUNt5AlA6EWP16wEe6RhTX+2Zwe1jCdE36tkvn5OBDtjrxN9 yBajCM+Huz2X5sBO+kcnzod934uevAOtAxn0OJUXUCE8YU5cscurhuf03i1Jf5IOY58D0x XofQVTy6dYx++FN6ptURPCK4uPTP05dhlqmSDKI5mlCZ/y220UveD58Fu6agig== Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4VvdhM5YfyzG9n; Wed, 5 Jun 2024 20:04:03 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org ([127.0.1.44]) by gitrepo.freebsd.org (8.17.1/8.17.1) with ESMTP id 455K43Yo011833; Wed, 5 Jun 2024 20:04:03 GMT (envelope-from git@gitrepo.freebsd.org) Received: (from git@localhost) by gitrepo.freebsd.org (8.17.1/8.17.1/Submit) id 455K43i6011830; Wed, 5 Jun 2024 20:04:03 GMT (envelope-from git) Date: Wed, 5 Jun 2024 20:04:03 GMT Message-Id: <202406052004.455K43i6011830@gitrepo.freebsd.org> To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-main@FreeBSD.org From: John Baldwin Subject: git: aacaeeee8ecd - main - nvmf: Permit failing I/O requests while disconnected List-Id: Commit messages for the main branch of the src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-main List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: dev-commits-src-main@freebsd.org Sender: owner-dev-commits-src-main@FreeBSD.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: jhb X-Git-Repository: src X-Git-Refname: refs/heads/main X-Git-Reftype: branch X-Git-Commit: aacaeeee8ecd7084a33b2d8e140aad37b3b0eddc Auto-Submitted: auto-generated The branch main has been updated by jhb: URL: https://cgit.FreeBSD.org/src/commit/?id=aacaeeee8ecd7084a33b2d8e140aad37b3b0eddc commit aacaeeee8ecd7084a33b2d8e140aad37b3b0eddc Author: John Baldwin AuthorDate: 2024-06-05 19:54:15 +0000 Commit: John Baldwin CommitDate: 2024-06-05 19:59:07 +0000 nvmf: Permit failing I/O requests while disconnected Add a kern.nvmf.fail_on_disconnection sysctl similar to the kern.iscsi.fail_on_disconnection sysctl. This causes pending I/O requests to fail with an error if an association is disconnected instead of requeueing to be retried once the association is reconnected. As with iSCSI, the default is to queue and retry operations. Reviewed by: imp Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D45308 --- share/man/man4/nvmf.4 | 24 +++++++++++++++++++++++- sys/dev/nvmf/host/nvmf.c | 5 +++++ sys/dev/nvmf/host/nvmf_ns.c | 25 ++++++++++++++++++++----- sys/dev/nvmf/host/nvmf_sim.c | 10 ++++++++-- sys/dev/nvmf/host/nvmf_var.h | 3 +++ 5 files changed, 59 insertions(+), 8 deletions(-) diff --git a/share/man/man4/nvmf.4 b/share/man/man4/nvmf.4 index 8afbb4d9daaf..298365acefa9 100644 --- a/share/man/man4/nvmf.4 +++ b/share/man/man4/nvmf.4 @@ -3,7 +3,7 @@ .\" .\" Copyright (c) 2024 Chelsio Communications, Inc. .\" -.Dd May 2, 2024 +.Dd June 5, 2024 .Dt NVMF 4 .Os .Sh NAME @@ -65,6 +65,28 @@ disk driver. Associations require a supported transport such as .Xr nvmf_tcp 4 for associations using TCP/IP. +.Sh SYSCTL VARIABLES +The following variables are available as both +.Xr sysctl 8 +variables and +.Xr loader 8 +tunables: +.Bl -tag -width indent +.It Va kern.nvmf.fail_on_disconnection +Determines the behavior when an association's connection is interrupted. +By default, input/output operations are suspended while a host is disconnected. +This includes operations pending at the time the association's connection was +interrupted as well as new requests submitted while the host is disconnected. +Once a new association is established, suspended I/O requests are retried. +When set to 1, input/output operations fail with +.Er EIO +while a host is disconnected and +.Xr nda 4 +peripherals are destroyed after the first failed I/O request. +Note that any destroyed +.Xr nda 4 +peripherals will be recreated after a new association is established. +.El .Sh SEE ALSO .Xr nda 4 , .Xr nvme 4 , diff --git a/sys/dev/nvmf/host/nvmf.c b/sys/dev/nvmf/host/nvmf.c index 9684170c1de9..c309836ed8a8 100644 --- a/sys/dev/nvmf/host/nvmf.c +++ b/sys/dev/nvmf/host/nvmf.c @@ -15,6 +15,7 @@ #include #include #include +#include #include #include #include @@ -23,6 +24,10 @@ static struct cdevsw nvmf_cdevsw; +bool nvmf_fail_disconnect = false; +SYSCTL_BOOL(_kern_nvmf, OID_AUTO, fail_on_disconnection, CTLFLAG_RWTUN, + &nvmf_fail_disconnect, 0, "Fail I/O requests on connection failure"); + MALLOC_DEFINE(M_NVMF, "nvmf", "NVMe over Fabrics host"); static void nvmf_disconnect_task(void *arg, int pending); diff --git a/sys/dev/nvmf/host/nvmf_ns.c b/sys/dev/nvmf/host/nvmf_ns.c index 0727ca960a57..8381cc4aec54 100644 --- a/sys/dev/nvmf/host/nvmf_ns.c +++ b/sys/dev/nvmf/host/nvmf_ns.c @@ -84,13 +84,22 @@ nvmf_ns_biodone(struct bio *bio) ns = bio->bio_dev->si_drv1; /* If a request is aborted, resubmit or queue it for resubmission. */ - if (bio->bio_error == ECONNABORTED) { + if (bio->bio_error == ECONNABORTED && !nvmf_fail_disconnect) { bio->bio_error = 0; bio->bio_driver2 = 0; mtx_lock(&ns->lock); if (ns->disconnected) { - TAILQ_INSERT_TAIL(&ns->pending_bios, bio, bio_queue); - mtx_unlock(&ns->lock); + if (nvmf_fail_disconnect) { + mtx_unlock(&ns->lock); + bio->bio_error = ECONNABORTED; + bio->bio_flags |= BIO_ERROR; + bio->bio_resid = bio->bio_bcount; + biodone(bio); + } else { + TAILQ_INSERT_TAIL(&ns->pending_bios, bio, + bio_queue); + mtx_unlock(&ns->lock); + } } else { mtx_unlock(&ns->lock); nvmf_ns_strategy(bio); @@ -163,6 +172,7 @@ nvmf_ns_submit_bio(struct nvmf_namespace *ns, struct bio *bio) struct nvme_dsm_range *dsm_range; struct memdesc mem; uint64_t lba, lba_count; + int error; dsm_range = NULL; memset(&cmd, 0, sizeof(cmd)); @@ -201,10 +211,15 @@ nvmf_ns_submit_bio(struct nvmf_namespace *ns, struct bio *bio) mtx_lock(&ns->lock); if (ns->disconnected) { - TAILQ_INSERT_TAIL(&ns->pending_bios, bio, bio_queue); + if (nvmf_fail_disconnect) { + error = ECONNABORTED; + } else { + TAILQ_INSERT_TAIL(&ns->pending_bios, bio, bio_queue); + error = 0; + } mtx_unlock(&ns->lock); free(dsm_range, M_NVMF); - return (0); + return (error); } req = nvmf_allocate_request(nvmf_select_io_queue(ns->sc), &cmd, diff --git a/sys/dev/nvmf/host/nvmf_sim.c b/sys/dev/nvmf/host/nvmf_sim.c index 00dad07889d1..71bb71dd4063 100644 --- a/sys/dev/nvmf/host/nvmf_sim.c +++ b/sys/dev/nvmf/host/nvmf_sim.c @@ -40,7 +40,10 @@ nvmf_ccb_done(union ccb *ccb) return; if (nvmf_cqe_aborted(&ccb->nvmeio.cpl)) { - ccb->ccb_h.status = CAM_REQUEUE_REQ; + if (nvmf_fail_disconnect) + ccb->ccb_h.status = CAM_DEV_NOT_THERE; + else + ccb->ccb_h.status = CAM_REQUEUE_REQ; xpt_done(ccb); } else if (ccb->nvmeio.cpl.status != 0) { ccb->ccb_h.status = CAM_NVME_STATUS_ERROR; @@ -106,7 +109,10 @@ nvmf_sim_io(struct nvmf_softc *sc, union ccb *ccb) mtx_lock(&sc->sim_mtx); if (sc->sim_disconnected) { mtx_unlock(&sc->sim_mtx); - nvmeio->ccb_h.status = CAM_REQUEUE_REQ; + if (nvmf_fail_disconnect) + nvmeio->ccb_h.status = CAM_DEV_NOT_THERE; + else + nvmeio->ccb_h.status = CAM_REQUEUE_REQ; xpt_done(ccb); return; } diff --git a/sys/dev/nvmf/host/nvmf_var.h b/sys/dev/nvmf/host/nvmf_var.h index 2fa0216baab8..cf88d2f7b01e 100644 --- a/sys/dev/nvmf/host/nvmf_var.h +++ b/sys/dev/nvmf/host/nvmf_var.h @@ -140,6 +140,9 @@ extern driver_t nvme_nvmf_driver; MALLOC_DECLARE(M_NVMF); #endif +/* If true, I/O requests will fail while the host is disconnected. */ +extern bool nvmf_fail_disconnect; + /* nvmf.c */ void nvmf_complete(void *arg, const struct nvme_completion *cqe); void nvmf_io_complete(void *arg, size_t xfered, int error);