From nobody Fri Jan 21 02:27:02 2022 X-Original-To: dev-commits-src-branches@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 6C07219691B7; Fri, 21 Jan 2022 02:27:08 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Jg3DR1J7Pz4nnG; Fri, 21 Jan 2022 02:27:03 +0000 (UTC) (envelope-from git@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1642732024; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=3uGGIvZiDBIqN2q/Bfoa3mguJr42kYE3sbIyNuICYtc=; b=OW8gdN/FlViIUDi2cwLlWLMKiqrH3XHiREi5pmTUCTjUN+s/HfnCjB6DU2j1ry80gQ0kQP ViVFVW0bScW0Qla0TCHIqnelZwQPPH7LbjrULEhjcg6Xwup2Ta6kM606eVp6MgcUxY/uuY TXF2Irp2wy+INVDWQaiTKILAZ3jPjVtuJaes0QfFQkaDvtSmPN2K+9ZTtS1iuCEjyxvpBH HvmU/nSyeTD0LqCY0yYA4DIgOvE1D3j/vEUWZsj8rgCS+dKwRkQVyYdJkIl4B4auL4gC/q z+A9as1mQbKnffnPxmY0TAnFCN/3JtiX6g9gLkRPkn4H1vs5V3fOVGEXI7rmeg== Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id DD54913D06; Fri, 21 Jan 2022 02:27:02 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org ([127.0.1.44]) by gitrepo.freebsd.org (8.16.1/8.16.1) with ESMTP id 20L2R2wv001460; Fri, 21 Jan 2022 02:27:02 GMT (envelope-from git@gitrepo.freebsd.org) Received: (from git@localhost) by gitrepo.freebsd.org (8.16.1/8.16.1/Submit) id 20L2R20Q001459; Fri, 21 Jan 2022 02:27:02 GMT (envelope-from git) Date: Fri, 21 Jan 2022 02:27:02 GMT Message-Id: <202201210227.20L2R20Q001459@gitrepo.freebsd.org> To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-branches@FreeBSD.org From: Alexander Motin Subject: git: 86721e606c51 - stable/13 - nvme: Use adaptive spinning when polling for completion or state change List-Id: Commits to the stable branches of the FreeBSD src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-branches List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-dev-commits-src-branches@freebsd.org X-BeenThere: dev-commits-src-branches@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: mav X-Git-Repository: src X-Git-Refname: refs/heads/stable/13 X-Git-Reftype: branch X-Git-Commit: 86721e606c51f1012e851f2c930694eae7890cff Auto-Submitted: auto-generated ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1642732024; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=3uGGIvZiDBIqN2q/Bfoa3mguJr42kYE3sbIyNuICYtc=; b=DaGzywHTm9CMo1i+jHigg1H1YYAqDlCvmnffdqSIJSP8FslLZSCSRdsCt02wX75HTEQokk 0c1rpqJ+uyFHC8xFOlPe2DYVT2251gaZ0P52tTQpR+e+aneQlCczRzubtv+igvBGCEoBf+ mQuJ/4fpuYreHnZuOXfjxApLCAw16tw1EaOQFi6sZMLkwR1hsuMqOTLDVHsouMgjHidxgB ixxviMsTMp7NdFC2WnJfn/0QkWwNRhBZbE47+iEfpobLkqGgd4kC6VghhtM+pkYJDpWMQ2 CBOollVf2Gb+FCn6fLrjHDtg/avjsrXACd7uYHb2aWwkZh9EfLRQxmf+TtzSxA== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1642732024; a=rsa-sha256; cv=none; b=qXVA8U000RM0JSPyW06jY3Y3TEKtrbV5aTO/IdljM0k4v3j3SGpC1uEexH76uVou2mr/2s on4/rIHTLeGlnc8VH1Xhtgiueod4sUaG+oVnjq7PAQ8JW7Ehx5Y+LScr7rhBYWU8qQmybl 6RivDRuSCITKWyOuIJwd9hnlwMl6yPsVIplZ6rxftAuZ9ljyKYsnoZAKG1d+KAu2Xpcujg Ja8bL/OIxTsRepVGw3o5evISoNEa6f/PE90fn2KUpOempGKhFYGUeJ1nnaKT77HCpH50/3 bWJK42N7lUwf5ZBtvMY/bhyKki9JZWgWdvxsA+ZPB+rNkJxYuF53PZg2FxKRIQ== ARC-Authentication-Results: i=1; mx1.freebsd.org; none X-ThisMailContainsUnwantedMimeParts: N The branch stable/13 has been updated by mav: URL: https://cgit.FreeBSD.org/src/commit/?id=86721e606c51f1012e851f2c930694eae7890cff commit 86721e606c51f1012e851f2c930694eae7890cff Author: Warner Losh AuthorDate: 2021-10-01 17:32:48 +0000 Commit: Alexander Motin CommitDate: 2022-01-21 02:07:31 +0000 nvme: Use adaptive spinning when polling for completion or state change We only use nvme_completion_poll in the initialization path. The commands they queue and wait for finish quickly as they involve no I/O to the drive's media. These command take about 20-200 microsecnds each. Set the wait time to 1us and then increase it by 1.5 each successive iteration (max 1ms). This reduces initialization time by 80ms in cpervica's tests. Use this same technique waiting for RDY state transitions. This saves another 20ms. In total we're down from ~330ms to ~2ms. Tested by: cperciva Sponsored by: Netflix Reviewed by: mav Differential Review: https://reviews.freebsd.org/D32259 (cherry picked from commit 83581511d9476ef5084f47e3cc379be7191ae866) --- sys/dev/nvme/nvme_ctrlr.c | 11 ++++++++++- sys/dev/nvme/nvme_private.h | 35 +++++++++++++++++++++-------------- 2 files changed, 31 insertions(+), 15 deletions(-) diff --git a/sys/dev/nvme/nvme_ctrlr.c b/sys/dev/nvme/nvme_ctrlr.c index ca89e99d2934..aa41c8fe5540 100644 --- a/sys/dev/nvme/nvme_ctrlr.c +++ b/sys/dev/nvme/nvme_ctrlr.c @@ -260,10 +260,17 @@ nvme_ctrlr_fail_req_task(void *arg, int pending) mtx_unlock(&ctrlr->lock); } +/* + * Wait for RDY to change. + * + * Starts sleeping for 1us and geometrically increases it the longer we wait, + * capped at 1ms. + */ static int nvme_ctrlr_wait_for_ready(struct nvme_controller *ctrlr, int desired_val) { int timeout = ticks + MSEC_2_TICKS(ctrlr->ready_timeout_in_ms); + sbintime_t delta_t = SBT_1US; uint32_t csts; while (1) { @@ -278,7 +285,9 @@ nvme_ctrlr_wait_for_ready(struct nvme_controller *ctrlr, int desired_val) "within %d ms\n", desired_val, ctrlr->ready_timeout_in_ms); return (ENXIO); } - pause("nvmerdy", 1); + + pause_sbt("nvmerdy", delta_t, 0, C_PREL(1)); + delta_t = min(SBT_1MS, delta_t * 3 / 2); } return (0); diff --git a/sys/dev/nvme/nvme_private.h b/sys/dev/nvme/nvme_private.h index b00130910a82..9c7c1b46eab4 100644 --- a/sys/dev/nvme/nvme_private.h +++ b/sys/dev/nvme/nvme_private.h @@ -455,25 +455,32 @@ int nvme_shutdown(device_t dev); int nvme_detach(device_t dev); /* - * Wait for a command to complete using the nvme_completion_poll_cb. - * Used in limited contexts where the caller knows it's OK to block - * briefly while the command runs. The ISR will run the callback which - * will set status->done to true, usually within microseconds. If not, - * then after one second timeout handler should reset the controller - * and abort all outstanding requests including this polled one. If - * still not after ten seconds, then something is wrong with the driver, - * and panic is the only way to recover. + * Wait for a command to complete using the nvme_completion_poll_cb. Used in + * limited contexts where the caller knows it's OK to block briefly while the + * command runs. The ISR will run the callback which will set status->done to + * true, usually within microseconds. If not, then after one second timeout + * handler should reset the controller and abort all outstanding requests + * including this polled one. If still not after ten seconds, then something is + * wrong with the driver, and panic is the only way to recover. + * + * Most commands using this interface aren't actual I/O to the drive's media so + * complete within a few microseconds. Adaptively spin for one tick to catch the + * vast majority of these without waiting for a tick plus scheduling delays. Since + * these are on startup, this drastically reduces startup time. */ static __inline void nvme_completion_poll(struct nvme_completion_poll_status *status) { - int sanity = hz * 10; - - while (!atomic_load_acq_int(&status->done) && --sanity > 0) - pause("nvme", 1); - if (sanity <= 0) - panic("NVME polled command failed to complete within 10s."); + int timeout = ticks + 10 * hz; + sbintime_t delta_t = SBT_1US; + + while (!atomic_load_acq_int(&status->done)) { + if (timeout - ticks < 0) + panic("NVME polled command failed to complete within 10s."); + pause_sbt("nvme", delta_t, 0, C_PREL(1)); + delta_t = min(SBT_1MS, delta_t * 3 / 2); + } } static __inline void