git: 6751f65e6af1 - main - nvmf: Defer the post-sync shutdown handler to SHUTDOWN_PRI_LAST

The branch main has been updated by jhb:

URL: https://cgit.FreeBSD.org/src/commit/?id=6751f65e6af15348abdc6106cf54c8335d45e49b

commit 6751f65e6af15348abdc6106cf54c8335d45e49b
Author:     John Baldwin <jhb@FreeBSD.org>
AuthorDate: 2024-10-31 15:03:41 +0000
Commit:     John Baldwin <jhb@FreeBSD.org>
CommitDate: 2025-01-13 20:04:29 +0000

    nvmf: Defer the post-sync shutdown handler to SHUTDOWN_PRI_LAST
    
    nda(4) has its own shutdown handler that runs at SHUTDOWN_PRI_DEFAULT
    that calls ndaflush() that could run after the nvmf handler.  Instead,
    give a the flush a chance to run before the graceful shutdown of the
    controller.
    
    While here, be a bit more defensive in the post-sync case and shutdown
    the consumers (sim and /dev/nvmeXnY devices) before destroying the
    queue pairs so that if any requests are submitted after the post-sync
    handler they fail gracefully instead of trying to use a destroyed
    queue pair.
    
    Reported by:    Sony Arpita Das <sonyarpitad@chelsio.com>
    Sponsored by:   Chelsio Communications
---
 sys/dev/nvmf/host/nvmf.c | 14 +++++++++++++-
 1 file changed, 13 insertions(+), 1 deletion(-)

diff --git a/sys/dev/nvmf/host/nvmf.c b/sys/dev/nvmf/host/nvmf.c
index 09d5cecdfad6..77d3081243f6 100644
--- a/sys/dev/nvmf/host/nvmf.c
+++ b/sys/dev/nvmf/host/nvmf.c
@@ -554,7 +554,7 @@ nvmf_attach(device_t dev)
 	sc->shutdown_pre_sync_eh = EVENTHANDLER_REGISTER(shutdown_pre_sync,
 	    nvmf_shutdown_pre_sync, sc, SHUTDOWN_PRI_FIRST);
 	sc->shutdown_post_sync_eh = EVENTHANDLER_REGISTER(shutdown_post_sync,
-	    nvmf_shutdown_post_sync, sc, SHUTDOWN_PRI_FIRST);
+	    nvmf_shutdown_post_sync, sc, SHUTDOWN_PRI_LAST);
 
 	return (0);
 out:
@@ -776,6 +776,18 @@ nvmf_shutdown_post_sync(void *arg, int howto)
 	callout_drain(&sc->ka_rx_timer);
 
 	nvmf_shutdown_controller(sc);
+
+	/*
+	 * Quiesce consumers so that any commands submitted after this
+	 * fail with an error.  Notably, nda(4) calls nda_flush() from
+	 * a post_sync handler that might be ordered after this one.
+	 */
+	for (u_int i = 0; i < sc->cdata->nn; i++) {
+		if (sc->ns[i] != NULL)
+			nvmf_shutdown_ns(sc->ns[i]);
+	}
+	nvmf_shutdown_sim(sc);
+
 	for (u_int i = 0; i < sc->num_io_queues; i++) {
 		nvmf_destroy_qp(sc->io[i]);
 	}