From nobody Mon Nov 06 16:11:52 2023 X-Original-To: dev-commits-src-all@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4SPGZJ2Vc8z507h8; Mon, 6 Nov 2023 16:11:52 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4SPGZJ22kpz3gGl; Mon, 6 Nov 2023 16:11:52 +0000 (UTC) (envelope-from git@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1699287112; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=q/BYY+Z3k1RyBZthksHNFQn+BAGASg0d3E7dddj7bdE=; b=IOV9SF1NclVfKVugImTT675zeh5bs+YhiOXz9K5amfeYuLQgNYgBDr/nCVUsoAxE84Y0sa AjXgXgT7lCf4WWv0zkJyro1Mf2rQb+hMoLr+dNGYpy4g17AZCwxtlorOplwzuMWyVFK8rA N9ezshg6XFCzFT2krQo2M1u6maYcL/09Lu8H/tWsUPuYO0bw8y53F5C5pH/+JZ5xtDr0kG RElK6silWwrJ9m7J2u7se+2nAel4pxB8IPy978EmXih0+U2+WeNdMNRYmvy1mMBhUZRbTa sPyr2uqvXcqEdBY4svFDMKaa+GhgQS1rq3DQil7ay9Nt/KR+8qNwjR1Cz2vrGA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1699287112; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=q/BYY+Z3k1RyBZthksHNFQn+BAGASg0d3E7dddj7bdE=; b=Pq47xiEXBJIw89G1vZy2JF9G0UzfHSjbeuR2DSnybeoHDyK6wB338QhPlr2VfSYQcF5JHx +6Jy/Xv7vx8Is7TV6RnKgvd8fgpQqZfU5Z5hjVnHfFLHs8W0x/SOJME8M6/OIItCekyitD X/qnEyeZ0Yy+uaBcsS1qRUkbuMWaaO1bHojykyfElHkBVmoEDIV10mI7HOHx/fQqVVuvKY oMwC97kzLmEKgW3sBeRr4jJHX7QL4il/Z1TD/ly6VWKvPmtZ3NfOai4cpdSH27sXLKGCb3 x3gMAy4uPpnOPRcDIW6/gHdRokuWkGRuV7xk1C7Kg4qzJLxiYxGvZzhVtAm/4w== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1699287112; a=rsa-sha256; cv=none; b=rwWT/AvIeGk+hmFXzVPUWKKbvyuLdTBfGINYsGVqTtdZEhwwVsu78H51BS/zkITNBB2M4t kUAp2A/brQtL68IN68jdrVKQCYaIPThh4NvI60/BSuarctr9U2KV3ir4pWCo/D+6IEprbA uzuLZGgek3MBPSZRhTkveMONwEKhbmbldYYqNIZlFfWr8D9kvBoMwGzoWBRUDnz6gXckB3 giPhIm/yXZuBvvxr+UQsP//BiEEX7bxmTNgEHEaAYQRydqWv3EkV6ETSBqR383O0WCT1BM 98zIrZgmoWFH3PeVgl9ldBl0A1xj5rsFygj5dU1KZhboLdrTSzQdHX2ya+AAKg== Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4SPGZJ166Mz1Cnk; Mon, 6 Nov 2023 16:11:52 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org ([127.0.1.44]) by gitrepo.freebsd.org (8.17.1/8.17.1) with ESMTP id 3A6GBqFE005139; Mon, 6 Nov 2023 16:11:52 GMT (envelope-from git@gitrepo.freebsd.org) Received: (from git@localhost) by gitrepo.freebsd.org (8.17.1/8.17.1/Submit) id 3A6GBqwP005136; Mon, 6 Nov 2023 16:11:52 GMT (envelope-from git) Date: Mon, 6 Nov 2023 16:11:52 GMT Message-Id: <202311061611.3A6GBqwP005136@gitrepo.freebsd.org> To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-main@FreeBSD.org From: Alexander Motin Subject: git: 8d6c0743e36e - main - nvme: Introduce longer timeouts for admin queue List-Id: Commit messages for all branches of the src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-all List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-dev-commits-src-all@freebsd.org X-BeenThere: dev-commits-src-all@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: mav X-Git-Repository: src X-Git-Refname: refs/heads/main X-Git-Reftype: branch X-Git-Commit: 8d6c0743e36e3cff9279c40468711a82db98df23 Auto-Submitted: auto-generated The branch main has been updated by mav: URL: https://cgit.FreeBSD.org/src/commit/?id=8d6c0743e36e3cff9279c40468711a82db98df23 commit 8d6c0743e36e3cff9279c40468711a82db98df23 Author: Alexander Motin AuthorDate: 2023-11-06 16:05:48 +0000 Commit: Alexander Motin CommitDate: 2023-11-06 16:05:48 +0000 nvme: Introduce longer timeouts for admin queue KIOXIA CD8 SSDs routinely take ~25 seconds to delete non-empty namespace. In some cases like hot-plug it takes longer, triggering timeout and controller resets after just 30 seconds. Linux for many years has separate 60 seconds timeout for admin queue. This patch does the same. And it is good to be consistent. Sponsored by: iXsystems, Inc. Reviewed by: imp MFC after: 1 week Differential Revision: https://reviews.freebsd.org/D42454 --- sys/dev/nvme/nvme_ctrlr.c | 6 ++++++ sys/dev/nvme/nvme_private.h | 2 ++ sys/dev/nvme/nvme_qpair.c | 2 ++ sys/dev/nvme/nvme_sysctl.c | 15 ++++++++++----- 4 files changed, 20 insertions(+), 5 deletions(-) diff --git a/sys/dev/nvme/nvme_ctrlr.c b/sys/dev/nvme/nvme_ctrlr.c index 90cd0b64a980..b72359a061da 100644 --- a/sys/dev/nvme/nvme_ctrlr.c +++ b/sys/dev/nvme/nvme_ctrlr.c @@ -1405,6 +1405,12 @@ nvme_ctrlr_construct(struct nvme_controller *ctrlr, device_t dev) to = NVME_CAP_LO_TO(cap_lo) + 1; ctrlr->ready_timeout_in_ms = to * 500; + timeout_period = NVME_ADMIN_TIMEOUT_PERIOD; + TUNABLE_INT_FETCH("hw.nvme.admin_timeout_period", &timeout_period); + timeout_period = min(timeout_period, NVME_MAX_TIMEOUT_PERIOD); + timeout_period = max(timeout_period, NVME_MIN_TIMEOUT_PERIOD); + ctrlr->admin_timeout_period = timeout_period; + timeout_period = NVME_DEFAULT_TIMEOUT_PERIOD; TUNABLE_INT_FETCH("hw.nvme.timeout_period", &timeout_period); timeout_period = min(timeout_period, NVME_MAX_TIMEOUT_PERIOD); diff --git a/sys/dev/nvme/nvme_private.h b/sys/dev/nvme/nvme_private.h index f6ad51795adb..1b6ea03c909a 100644 --- a/sys/dev/nvme/nvme_private.h +++ b/sys/dev/nvme/nvme_private.h @@ -86,6 +86,7 @@ MALLOC_DECLARE(M_NVME); #define NVME_MAX_CONSUMERS (2) #define NVME_MAX_ASYNC_EVENTS (8) +#define NVME_ADMIN_TIMEOUT_PERIOD (60) /* in seconds */ #define NVME_DEFAULT_TIMEOUT_PERIOD (30) /* in seconds */ #define NVME_MIN_TIMEOUT_PERIOD (5) #define NVME_MAX_TIMEOUT_PERIOD (120) @@ -279,6 +280,7 @@ struct nvme_controller { uint32_t int_coal_threshold; /** timeout period in seconds */ + uint32_t admin_timeout_period; uint32_t timeout_period; /** doorbell stride */ diff --git a/sys/dev/nvme/nvme_qpair.c b/sys/dev/nvme/nvme_qpair.c index 569d54ab6aef..7345b1b59be5 100644 --- a/sys/dev/nvme/nvme_qpair.c +++ b/sys/dev/nvme/nvme_qpair.c @@ -1181,6 +1181,8 @@ nvme_qpair_submit_tracker(struct nvme_qpair *qpair, struct nvme_tracker *tr) if (req->timeout) { if (req->cb_fn == nvme_completion_poll_cb) timeout = 1; + else if (qpair->id == 0) + timeout = ctrlr->admin_timeout_period; else timeout = ctrlr->timeout_period; tr->deadline = getsbinuptime() + timeout * SBT_1S; diff --git a/sys/dev/nvme/nvme_sysctl.c b/sys/dev/nvme/nvme_sysctl.c index ac0d507e2337..d6452a2e5492 100644 --- a/sys/dev/nvme/nvme_sysctl.c +++ b/sys/dev/nvme/nvme_sysctl.c @@ -132,8 +132,8 @@ nvme_sysctl_int_coal_threshold(SYSCTL_HANDLER_ARGS) static int nvme_sysctl_timeout_period(SYSCTL_HANDLER_ARGS) { - struct nvme_controller *ctrlr = arg1; - uint32_t newval = ctrlr->timeout_period; + uint32_t *ptr = arg1; + uint32_t newval = *ptr; int error = sysctl_handle_int(oidp, &newval, 0, req); if (error || (req->newptr == NULL)) @@ -143,7 +143,7 @@ nvme_sysctl_timeout_period(SYSCTL_HANDLER_ARGS) newval < NVME_MIN_TIMEOUT_PERIOD) { return (EINVAL); } else { - ctrlr->timeout_period = newval; + *ptr = newval; } return (0); @@ -352,10 +352,15 @@ nvme_sysctl_initialize_ctrlr(struct nvme_controller *ctrlr) nvme_sysctl_int_coal_threshold, "IU", "Interrupt coalescing threshold"); + SYSCTL_ADD_PROC(ctrlr_ctx, ctrlr_list, OID_AUTO, + "admin_timeout_period", CTLTYPE_UINT | CTLFLAG_RW | CTLFLAG_MPSAFE, + &ctrlr->admin_timeout_period, 0, nvme_sysctl_timeout_period, "IU", + "Timeout period for Admin queue (in seconds)"); + SYSCTL_ADD_PROC(ctrlr_ctx, ctrlr_list, OID_AUTO, "timeout_period", CTLTYPE_UINT | CTLFLAG_RW | CTLFLAG_MPSAFE, - ctrlr, 0, nvme_sysctl_timeout_period, "IU", - "Timeout period (in seconds)"); + &ctrlr->timeout_period, 0, nvme_sysctl_timeout_period, "IU", + "Timeout period for I/O queues (in seconds)"); SYSCTL_ADD_PROC(ctrlr_ctx, ctrlr_list, OID_AUTO, "num_cmds", CTLTYPE_S64 | CTLFLAG_RD | CTLFLAG_MPSAFE,