From nobody Mon Jan 13 20:07:59 2025 X-Original-To: dev-commits-src-all@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4YX3GS1b68z5klkN; Mon, 13 Jan 2025 20:08:00 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R11" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4YX3GS0Vlfz3bLm; Mon, 13 Jan 2025 20:08:00 +0000 (UTC) (envelope-from git@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1736798880; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=ExZAnKtP+n+NfVb9/SxBvQtaOzMaOSSeJGWkR1OMFTM=; b=KfQotQ3qUfHxJVqIJhEOs/J2Y9T46WYjXyInMVhm+Pj7yfWTTXOUQcOoQPzpb3poLZm638 nyYb+l7hWPghYp0uzWwjoNJuY0NPyPBjeMKM9ZwX/wND13aMFggbgbMjRVvDqwCw5+7QR2 fXSwhoevQwOBhiD6HQyxqWLkkc+Lv3Lgt83GOBMeRZnD5iAO2XQAn08fWOgc8UTazWAYRo 8odAdvEL50LXy2nFj9einh1BXaXHqwrehkpzntssDEJ7j3qoUia6lb3/c8ozzxpy/MumbG BcIMxIJszS4cl4h0GvmV7uhRU7xaGvw9W9QWi9fNGdsPvC8MBDGXigl+mkekTg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1736798880; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=ExZAnKtP+n+NfVb9/SxBvQtaOzMaOSSeJGWkR1OMFTM=; b=kl7n5jf3mozR/y/YXPDO3pNBhVOrrvYpNMEiFRWtj0Fe35MT4K1hKmLcQPMs6Nya0MOhF4 MjudM/c2OyrbNfhvbdaG+KGoXa3HkAhgchkebNJq1GkaSC9smxzo0YnFDz04opUkC0vMrU biqxV3j3JSYo9zHJm9t8s3/4hgcoKs8DvSvhsf/365muYG6B+7mEqtFnrz6QlTsmlg6GjD ZzzhZfgDorAme0OBC2SKIEopFPx6Jb6L6kELY2kswG/85ZWHlYtGSE5GYr7W1xPYSnVRyX RELcv4u3GQoCMvIORKgPk/eLOQirEfu27w7TMrTzYtGrgoHQyLAObbSTa+jDRA== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1736798880; a=rsa-sha256; cv=none; b=SCc73gK0DmqMBQDXgs/h6JXPa2ir/kPWvXNKU5kY+mrDPIeWmjV8qe7YrW3ZAHfXZIQcHh HPc0nhmX17PfovqdqChW4n9vrCcRGKWZm88u/Gg4JRy4nKtlli3U5IqMSpFWdumSfmJ9JN BFIYgIShAEEp3U1uu3NIlbT4pip9oy1i8pSZ2W+UmshLBE43Gx7jcwBQKbcaJtARARwBtz q8fjPH4l8+T0gx4g7a1I82ERDPeWIgy1rU6paB5InXOsZGHsJLOPonXpBQFFQMPvCJ1V1Q nm8dBTcixAIfCnURhl4fmcrjNNVdx6skISmJrEqM4PztqkhoQOuIMC6b8vv72w== ARC-Authentication-Results: i=1; mx1.freebsd.org; none Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4YX3GR75dGzlnp; Mon, 13 Jan 2025 20:07:59 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org ([127.0.1.44]) by gitrepo.freebsd.org (8.18.1/8.18.1) with ESMTP id 50DK7xCg016895; Mon, 13 Jan 2025 20:07:59 GMT (envelope-from git@gitrepo.freebsd.org) Received: (from git@localhost) by gitrepo.freebsd.org (8.18.1/8.18.1/Submit) id 50DK7xim016892; Mon, 13 Jan 2025 20:07:59 GMT (envelope-from git) Date: Mon, 13 Jan 2025 20:07:59 GMT Message-Id: <202501132007.50DK7xim016892@gitrepo.freebsd.org> To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-main@FreeBSD.org From: John Baldwin Subject: git: 6751f65e6af1 - main - nvmf: Defer the post-sync shutdown handler to SHUTDOWN_PRI_LAST List-Id: Commit messages for all branches of the src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-all List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: dev-commits-src-all@freebsd.org Sender: owner-dev-commits-src-all@FreeBSD.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: jhb X-Git-Repository: src X-Git-Refname: refs/heads/main X-Git-Reftype: branch X-Git-Commit: 6751f65e6af15348abdc6106cf54c8335d45e49b Auto-Submitted: auto-generated The branch main has been updated by jhb: URL: https://cgit.FreeBSD.org/src/commit/?id=6751f65e6af15348abdc6106cf54c8335d45e49b commit 6751f65e6af15348abdc6106cf54c8335d45e49b Author: John Baldwin AuthorDate: 2024-10-31 15:03:41 +0000 Commit: John Baldwin CommitDate: 2025-01-13 20:04:29 +0000 nvmf: Defer the post-sync shutdown handler to SHUTDOWN_PRI_LAST nda(4) has its own shutdown handler that runs at SHUTDOWN_PRI_DEFAULT that calls ndaflush() that could run after the nvmf handler. Instead, give a the flush a chance to run before the graceful shutdown of the controller. While here, be a bit more defensive in the post-sync case and shutdown the consumers (sim and /dev/nvmeXnY devices) before destroying the queue pairs so that if any requests are submitted after the post-sync handler they fail gracefully instead of trying to use a destroyed queue pair. Reported by: Sony Arpita Das Sponsored by: Chelsio Communications --- sys/dev/nvmf/host/nvmf.c | 14 +++++++++++++- 1 file changed, 13 insertions(+), 1 deletion(-) diff --git a/sys/dev/nvmf/host/nvmf.c b/sys/dev/nvmf/host/nvmf.c index 09d5cecdfad6..77d3081243f6 100644 --- a/sys/dev/nvmf/host/nvmf.c +++ b/sys/dev/nvmf/host/nvmf.c @@ -554,7 +554,7 @@ nvmf_attach(device_t dev) sc->shutdown_pre_sync_eh = EVENTHANDLER_REGISTER(shutdown_pre_sync, nvmf_shutdown_pre_sync, sc, SHUTDOWN_PRI_FIRST); sc->shutdown_post_sync_eh = EVENTHANDLER_REGISTER(shutdown_post_sync, - nvmf_shutdown_post_sync, sc, SHUTDOWN_PRI_FIRST); + nvmf_shutdown_post_sync, sc, SHUTDOWN_PRI_LAST); return (0); out: @@ -776,6 +776,18 @@ nvmf_shutdown_post_sync(void *arg, int howto) callout_drain(&sc->ka_rx_timer); nvmf_shutdown_controller(sc); + + /* + * Quiesce consumers so that any commands submitted after this + * fail with an error. Notably, nda(4) calls nda_flush() from + * a post_sync handler that might be ordered after this one. + */ + for (u_int i = 0; i < sc->cdata->nn; i++) { + if (sc->ns[i] != NULL) + nvmf_shutdown_ns(sc->ns[i]); + } + nvmf_shutdown_sim(sc); + for (u_int i = 0; i < sc->num_io_queues; i++) { nvmf_destroy_qp(sc->io[i]); }