git: 12588ce02dd8 - main - PCI hot-plug: use dedicated taskqueue for device attach / detach

[Andriy Gapon]

The branch main has been updated by avg:

URL: https://cgit.FreeBSD.org/src/commit/?id=12588ce02dd835b332952d9fece5881d943554a9

commit 12588ce02dd835b332952d9fece5881d943554a9
Author:     Andriy Gapon <avg at FreeBSD.org>
AuthorDate: 2021-05-06 18:49:37 +0000
Commit:     Andriy Gapon <avg at FreeBSD.org>
CommitDate: 2021-05-06 18:49:37 +0000

    PCI hot-plug: use dedicated taskqueue for device attach / detach
    
    Attaching and detaching devices can be heavy-weight and detaching can
    sleep waiting for events.  For that reason using the system-wide
    single-threaded taskqueue_thread is not really appropriate.
    There is even a possibility for a deadlock if taskqueue_thread is used
    for detaching.
    
    In fact, there is an easy to reproduce deadlock involving nvme, pass
    and a sudden removal of an NVMe device.
    A pass peripheral would not release a reference on an nvme sim until
    pass_shutdown_kqueue() is executed via taskqueue_thread.  But the
    taskqueue's thread is blocked in nvme_detach() -> ... -> cam_sim_free()
    because of the outstanding reference.
    
    MFC after:      10 days
    Sponsored by:   CyberSecure
    Reviewed by:    mav, imp
    Differential Revision:  https://reviews.freebsd.org/D30144
---
 sys/dev/pci/pci_pci.c | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/sys/dev/pci/pci_pci.c b/sys/dev/pci/pci_pci.c
index 3a11d59f51fd..d6fbb06a61ac 100644
--- a/sys/dev/pci/pci_pci.c
+++ b/sys/dev/pci/pci_pci.c
@@ -925,6 +925,8 @@ SYSCTL_INT(_hw_pci, OID_AUTO, enable_pcie_hp, CTLFLAG_RDTUN,
     &pci_enable_pcie_hp, 0,
     "Enable support for native PCI-express HotPlug.");
 
+TASKQUEUE_DEFINE_THREAD(pci_hp);
+
 static void
 pcib_probe_hotplug(struct pcib_softc *sc)
 {
@@ -1154,7 +1156,7 @@ pcib_pcie_hotplug_update(struct pcib_softc *sc, uint16_t val, uint16_t mask,
 	 */
 	if (schedule_task &&
 	    (pcib_hotplug_present(sc) != 0) != (sc->child != NULL))
-		taskqueue_enqueue(taskqueue_thread, &sc->pcie_hp_task);
+		taskqueue_enqueue(taskqueue_pci_hp, &sc->pcie_hp_task);
 }
 
 static void
@@ -1449,7 +1451,7 @@ pcib_detach_hotplug(struct pcib_softc *sc)
 	error = pcib_release_pcie_irq(sc);
 	if (error)
 		return (error);
-	taskqueue_drain(taskqueue_thread, &sc->pcie_hp_task);
+	taskqueue_drain(taskqueue_pci_hp, &sc->pcie_hp_task);
 	callout_drain(&sc->pcie_ab_timer);
 	callout_drain(&sc->pcie_cc_timer);
 	callout_drain(&sc->pcie_dll_timer);