Device nodes(/dev/daX) persists even after removing the device on FreeBSD10.3 Release

[Sumit Saxena]

Hi,

While doing some testing on Broadcom/LSI MegaRAID invader controller on
FreeBSD10.03 release, I am facing  problem of device node /dev/daX not
getting removed even after device is gone(removed/deleted).
Setup has multiple Virtual disks(VDs) created behind MegaRAID controller
and I am running IOs on these VDs. I observed that sometimes VDs are
deleted using Broadcom's management application but still device nodes of
these VDs are still present.
However  the device is not seen in "camcontrol devlist" output. This issue
is intermittent in nature.

Please find below some information/data collected from setup-

1. Output of camcontrol devlist:

root at MyBsd:~ # camcontrol devlist
<SEAGATE ST300MM0026 0003>         at scbus1 target 50 lun 0 (pass0,da0)
<AHCI SGPIO Enclosure 1.00 0001>   at scbus8 target 0 lun 0 (pass1,ses0)
<DELL MD1220 1.01>                 at scbus10 target 0 lun 0 (pass2,ses1)
<DELL MD1220 1.01>                 at scbus10 target 2 lun 0 (ses2,pass3)


2. See below dev nodes(/dev/daX)-

root at MyBsd:~ # ls /dev/da*
/dev/da0	/dev/da0p1	/dev/da0p2	/dev/da0p3	/dev/da1
/dev/da2	/dev/da37

da0 is OS drive and rest of these drives "da1" "da2" and "da37" are
deleted but still dev nodes is there with size 0 bytes. I tried
rescanning(camcontrol rescan all) but it does not help.


3. OS:
root at MyBsd:~ # uname -a
FreeBSD MyBsd 10.3-RELEASE FreeBSD 10.3-RELEASE #0 r297264: Fri Mar 25
02:10:02 UTC 2016
root at releng1.nyi.freebsd.org:/usr/obj/usr/src/sys/GENERIC  amd64

4.  dmesg logs pertaining to /dev/da37-

Oct  6 03:17:48 MyBsd kernel: da37: mrsas1:0:<Cisco MR9361-8i 4.67>35: s/n
00f5fd11568e82871f50d3c408b005060):  detached
Oct  6 03:17:48 MyBsd kernel: Periph destroyed
Oct  6 03:17:48 MyBsd kernel: (da37:
Oct  6 03:17:48 MyBsd kernel: mrsas1:0:
Oct  6 03:17:48 MyBsd kernel: 36:0): Periph destroyed
----
Oct  6 03:18:05 MyBsd kernel: (da35:mrsas1:0:50:0): got CAM status 0x208
Oct  6 03:18:05 MyBsd kernel: (da35:mrsas1:0:50:0): fatal error, failed to
attach to device
Oct  6 03:18:05 MyBsd kernel: (da37:mrsas1:0:51:0): got CAM status 0x208
Oct  6 03:18:05 MyBsd kernel: (da37:mrsas1:0:51:0): fatal error, failed to
attach to device
----
Oct  6 05:08:17 MyBsd kernel: cam_periph_alloc: attempt to re-allocate
valid device da37 rejected flags 0x18 refcount
1------------------------>here probably OS tries to assign da37 to some
newly created device but looks like da37 still has older reference.
Oct  6 05:08:17 MyBsd kernel: daasync: Unable to attach to new device due
to status 0x6


5. Device info for da37-

 root at MyBsd:/var/log # geom disk list da37
Geom name: da37
Providers:
1. Name: da37
   Mediasize: 0 (0B)
   Sectorsize: 0
   Mode: r0w0e0
   descr: LSI MR9361-8i
   lunid: 600605b008c4d3501f8782ab57c759db
   ident: 00db59c757ab82871f50d3c408b00506
   rotationrate: unknown
   fwsectors: 0
   fwheads: 0

root at MyBsd:/var/log #

6. after this I am able to create/delete VDs but bus scan is triggered
multiple times still these dev nodes- da2, da3, da37 exists.

Please let me know if anyone faced this problem. Any pointers/help will be
very much appreciated.


Thanks,
Sumit