From nobody Tue Nov 14 12:57:14 2023 X-Original-To: dev-commits-src-branches@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4SV5t26J2xz50sZV; Tue, 14 Nov 2023 12:57:14 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4SV5t25skPz3GGR; Tue, 14 Nov 2023 12:57:14 +0000 (UTC) (envelope-from git@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1699966634; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=2yFVPDC40YH/ervxs50WVtIjLdYHGeaPSFcR1Wya9t4=; b=kjfIulAKZr6+3WL2ey8y/0QrxRMdceb1dgZ6B6TuI6XkL+b0YPJ2siCWm0t9MQ4DkA2m9m K1HtKPgJLZxzulCQ0QaTTcrQ0OZz6GLWws05ojEXh/yqBnXFC9Zp+ojiKurhKIrk/pZdf1 wsMA9eGSlJGKvBQNRhkmJCkPzpNar4RMrO3AtL5/4gHtBgrZYcpqJgN2FclkXMLbkOgpiY qYLidGK9GOy+0Tg2kh3g06I/DIG2MnRL8T9Oqg8iUsPJ2EeFryJ/W8iqwbfWAMQPL13sFI J8zDPzbiEGrcxV41PE8eXtJ/nQRV1jccYbr+OPGsYr3QyEzPSYaM8EYlY59gMA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1699966634; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=2yFVPDC40YH/ervxs50WVtIjLdYHGeaPSFcR1Wya9t4=; b=kRk3oAOMBahxyxrglsJWF2luvSAF1FPRcSnT2GKng3opw76kIJvBDgFUSRDf/XIIphyaYB mIBVqb0G5T7fFk/5rTJY5uq2gOooqIEZE0tddytgavQDrBlomG0ZQV2704tfRmrEOFobsa r98ISQVNCIZbw+jBPB6HfYPKoyvc+y5BB7GjCw3JPkzaxAUVg3AJrJsUsuFIAAvY3LtuQ9 ifb5hMJ0WmZzpQEI7+f46ZFZPDOCfzz/Lc0hnqZ/qGhQU/WMHVzEC2sNjajDfcz5uDFbod XpeTCAsd/+S/3Ocd4h2hIeA0rFdC35vsRzJFtdeyGENqrgG5rSXNGJwkr4HcXA== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1699966634; a=rsa-sha256; cv=none; b=ID5cpdAUfKKbFr0TceeBwpvLQkQ4Jns5oSguhRsj6+Wox+zP/fX/d5IDrZ71lgWGU0GT/T j6t0k+E6ifZf3rj4tE/Pkd04wP2fTlh5rXFhUr9bQnSKjchKGVWO/ntcMIVDhzNhmufl/g Hi9NW1ug/RRVOntESI4oGRYy3ZMV+Jq2cE+VDDw3/xT+WFR8y8aX5reMPFGPLQbf/8AHyX gfchJCBEu6XPgu+x3XCDPirJCf6ZFgqnL2O+hA9hVqIXWHiJFFfjUgagub4CHdO3jF9Enw F1CbvMXv9qkskXA0i4GLLPo+KB5yBvssxWBm3zgqFHLh6NeTb6+mSg385G5RAg== Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4SV5t24wRmzq8G; Tue, 14 Nov 2023 12:57:14 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org ([127.0.1.44]) by gitrepo.freebsd.org (8.17.1/8.17.1) with ESMTP id 3AECvEZj017030; Tue, 14 Nov 2023 12:57:14 GMT (envelope-from git@gitrepo.freebsd.org) Received: (from git@localhost) by gitrepo.freebsd.org (8.17.1/8.17.1/Submit) id 3AECvEkZ017027; Tue, 14 Nov 2023 12:57:14 GMT (envelope-from git) Date: Tue, 14 Nov 2023 12:57:14 GMT Message-Id: <202311141257.3AECvEkZ017027@gitrepo.freebsd.org> To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-branches@FreeBSD.org From: Alexander Motin Subject: git: 7ab9e36fc544 - stable/14 - nvme: Introduce longer timeouts for admin queue List-Id: Commits to the stable branches of the FreeBSD src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-branches List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-dev-commits-src-branches@freebsd.org X-BeenThere: dev-commits-src-branches@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: mav X-Git-Repository: src X-Git-Refname: refs/heads/stable/14 X-Git-Reftype: branch X-Git-Commit: 7ab9e36fc54445fd72741fb0c85d84694da046f4 Auto-Submitted: auto-generated The branch stable/14 has been updated by mav: URL: https://cgit.FreeBSD.org/src/commit/?id=7ab9e36fc54445fd72741fb0c85d84694da046f4 commit 7ab9e36fc54445fd72741fb0c85d84694da046f4 Author: Alexander Motin AuthorDate: 2023-11-06 16:05:48 +0000 Commit: Alexander Motin CommitDate: 2023-11-14 12:54:18 +0000 nvme: Introduce longer timeouts for admin queue KIOXIA CD8 SSDs routinely take ~25 seconds to delete non-empty namespace. In some cases like hot-plug it takes longer, triggering timeout and controller resets after just 30 seconds. Linux for many years has separate 60 seconds timeout for admin queue. This patch does the same. And it is good to be consistent. Sponsored by: iXsystems, Inc. Reviewed by: imp MFC after: 1 week Differential Revision: https://reviews.freebsd.org/D42454 (cherry picked from commit 8d6c0743e36e3cff9279c40468711a82db98df23) --- sys/dev/nvme/nvme_ctrlr.c | 6 ++++++ sys/dev/nvme/nvme_private.h | 2 ++ sys/dev/nvme/nvme_qpair.c | 2 ++ sys/dev/nvme/nvme_sysctl.c | 15 ++++++++++----- 4 files changed, 20 insertions(+), 5 deletions(-) diff --git a/sys/dev/nvme/nvme_ctrlr.c b/sys/dev/nvme/nvme_ctrlr.c index ef4d7daa6efa..3eca24f1f6a5 100644 --- a/sys/dev/nvme/nvme_ctrlr.c +++ b/sys/dev/nvme/nvme_ctrlr.c @@ -1451,6 +1451,12 @@ nvme_ctrlr_construct(struct nvme_controller *ctrlr, device_t dev) to = NVME_CAP_LO_TO(cap_lo) + 1; ctrlr->ready_timeout_in_ms = to * 500; + timeout_period = NVME_ADMIN_TIMEOUT_PERIOD; + TUNABLE_INT_FETCH("hw.nvme.admin_timeout_period", &timeout_period); + timeout_period = min(timeout_period, NVME_MAX_TIMEOUT_PERIOD); + timeout_period = max(timeout_period, NVME_MIN_TIMEOUT_PERIOD); + ctrlr->admin_timeout_period = timeout_period; + timeout_period = NVME_DEFAULT_TIMEOUT_PERIOD; TUNABLE_INT_FETCH("hw.nvme.timeout_period", &timeout_period); timeout_period = min(timeout_period, NVME_MAX_TIMEOUT_PERIOD); diff --git a/sys/dev/nvme/nvme_private.h b/sys/dev/nvme/nvme_private.h index 496bd8229e0a..72797d30a3bf 100644 --- a/sys/dev/nvme/nvme_private.h +++ b/sys/dev/nvme/nvme_private.h @@ -86,6 +86,7 @@ MALLOC_DECLARE(M_NVME); #define NVME_MAX_CONSUMERS (2) #define NVME_MAX_ASYNC_EVENTS (8) +#define NVME_ADMIN_TIMEOUT_PERIOD (60) /* in seconds */ #define NVME_DEFAULT_TIMEOUT_PERIOD (30) /* in seconds */ #define NVME_MIN_TIMEOUT_PERIOD (5) #define NVME_MAX_TIMEOUT_PERIOD (120) @@ -281,6 +282,7 @@ struct nvme_controller { uint32_t int_coal_threshold; /** timeout period in seconds */ + uint32_t admin_timeout_period; uint32_t timeout_period; /** doorbell stride */ diff --git a/sys/dev/nvme/nvme_qpair.c b/sys/dev/nvme/nvme_qpair.c index cd0057f444b8..2e9af4310a15 100644 --- a/sys/dev/nvme/nvme_qpair.c +++ b/sys/dev/nvme/nvme_qpair.c @@ -1158,6 +1158,8 @@ nvme_qpair_submit_tracker(struct nvme_qpair *qpair, struct nvme_tracker *tr) if (req->timeout) { if (req->cb_fn == nvme_completion_poll_cb) timeout = 1; + else if (qpair->id == 0) + timeout = ctrlr->admin_timeout_period; else timeout = ctrlr->timeout_period; tr->deadline = getsbinuptime() + timeout * SBT_1S; diff --git a/sys/dev/nvme/nvme_sysctl.c b/sys/dev/nvme/nvme_sysctl.c index ac0d507e2337..d6452a2e5492 100644 --- a/sys/dev/nvme/nvme_sysctl.c +++ b/sys/dev/nvme/nvme_sysctl.c @@ -132,8 +132,8 @@ nvme_sysctl_int_coal_threshold(SYSCTL_HANDLER_ARGS) static int nvme_sysctl_timeout_period(SYSCTL_HANDLER_ARGS) { - struct nvme_controller *ctrlr = arg1; - uint32_t newval = ctrlr->timeout_period; + uint32_t *ptr = arg1; + uint32_t newval = *ptr; int error = sysctl_handle_int(oidp, &newval, 0, req); if (error || (req->newptr == NULL)) @@ -143,7 +143,7 @@ nvme_sysctl_timeout_period(SYSCTL_HANDLER_ARGS) newval < NVME_MIN_TIMEOUT_PERIOD) { return (EINVAL); } else { - ctrlr->timeout_period = newval; + *ptr = newval; } return (0); @@ -352,10 +352,15 @@ nvme_sysctl_initialize_ctrlr(struct nvme_controller *ctrlr) nvme_sysctl_int_coal_threshold, "IU", "Interrupt coalescing threshold"); + SYSCTL_ADD_PROC(ctrlr_ctx, ctrlr_list, OID_AUTO, + "admin_timeout_period", CTLTYPE_UINT | CTLFLAG_RW | CTLFLAG_MPSAFE, + &ctrlr->admin_timeout_period, 0, nvme_sysctl_timeout_period, "IU", + "Timeout period for Admin queue (in seconds)"); + SYSCTL_ADD_PROC(ctrlr_ctx, ctrlr_list, OID_AUTO, "timeout_period", CTLTYPE_UINT | CTLFLAG_RW | CTLFLAG_MPSAFE, - ctrlr, 0, nvme_sysctl_timeout_period, "IU", - "Timeout period (in seconds)"); + &ctrlr->timeout_period, 0, nvme_sysctl_timeout_period, "IU", + "Timeout period for I/O queues (in seconds)"); SYSCTL_ADD_PROC(ctrlr_ctx, ctrlr_list, OID_AUTO, "num_cmds", CTLTYPE_S64 | CTLFLAG_RD | CTLFLAG_MPSAFE,