[Bug 260591] [attach/detach] [panic] insufficient multi-thread protection for probe/attach/detach

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=260591

            Bug ID: 260591
           Summary: [attach/detach] [panic] insufficient multi-thread
                    protection for probe/attach/detach
           Product: Base System
           Version: CURRENT
          Hardware: Any
                OS: Any
            Status: New
          Severity: Affects Some People
          Priority: ---
         Component: kern
          Assignee: bugs@FreeBSD.org
          Reporter: ghuckriede@blackberry.com

Created attachment 230290
  --> https://bugs.freebsd.org/bugzilla/attachment.cgi?id=230290&action=edit
main-n251848-3e01ee76f20 Panic

The protection for probe/attach/detach appears to be the 'Giant' mutex lock. 
This is insufficient to protect against multiple thread attaching/detaching at
the same time.  Some attach/detach actions require "sleeps", this releases the
'Giant' lock, allowing other threads to also attach/detach.  There are no
checks for `DS_ATTACHING` state in device_attach() and there is currently no
`DS_DETACHING` state to check in device_detach().

Steps to Reproduce:
This is easily reproduced with devices that "sleep" during attach/detach, as
Giant lock is dropped in these cases.

https://www.freebsd.org/cgi/man.cgi?locking(9). "Giant is dropped during
unbounded sleeps and reacquired after wakeup." 

igb devices enter e1000_get_cfg_done_i210()->safe_pause_sbt()->pause_sbt()
during detach, thus releasing the Giant lock.  As show in the attached
backtrace.

root@FBSDCURRENT:/ # devinfo -v | grep igb0
            igb0 pnpinfo vendor=0x8086 device=0x1533 subvendor=0x8086
subdevice=0x0002 class=0x020000 at slot=0 function=0 dbsf=pci0:6:0:0
handle=\_SB_.PCI0.RP05.PXSX
root@FBSDCURRENT:/ #

Terminal #1:
# sh
root@FBSDCURRENT:/ # while [ true ] ; do devctl attach pci0:6:0:0;devctl detach
pci0:6:0:0;done

Terminal #2:
# sh
root@FBSDCURRENT:/ # while [ true ] ; do devctl attach pci0:6:0:0;devctl detach
pci0:6:0:0;done

Actual Results:
This causes an immediate panic when Terminal #2 starts loop.

A backtrace of the 2 threads both running the devctl's devctl2_ioctl() is
attached (locally build kernel). `vmcore` file from "13.0-RELEASE" can also be
provided.

Expected Results:
The kernel should not panic/crash.  First thread should complete.  The second
thread should return an error (or wait until the other thread is complete).

Build Date & Hardware: 
Locally compiled with git updated Dec 21 2021:

FreeBSD FBSDCURRENT 14.0-CURRENT FreeBSD 14.0-CURRENT #3
main-n251848-3e01ee76f20: Tue Dec 21 00:33:45 EST 2021    
root@FBSDCURRENT:/usr/obj/usr/src/amd64.amd64/sys/GENERIC  amd64

Also reproducible on 13.0-RELEASE (Downloaded/Updated):

FreeBSD TrafficHammerTwoHanded 13.0-RELEASE FreeBSD 13.0-RELEASE #0
releng/13.0-n244733-ea31abc261f: Fri Apr  9 04:24:09 UTC 2021    
root@releng1.nyi.freebsd.org:/usr/obj/usr/src/amd64.amd64/sys/GENERIC  amd64

-- 
You are receiving this mail because:
You are the assignee for the bug.