problems with SAS JBODs 2

[Oliver Sech]

Hi!
 
I use FreeBSD with for a large ZFS pool (over 1PB) and I recently encountered a lot of problems with the JBODs. Generally everything works fine until I replug the shelves.
 
When I start with a clean system and attach a single shelf every thing seems fine.
-> 44 disks show up, I can use the enclosure services (sesutil) and the system continues to run without problems.
Once I disconnect the SAS cable, wait until all devices disapear and reconnect I get all sorts of problems.
-> a random number of disks shows up and the enclosure "ses" do not show up
Once I restart the system I can start over again.
 
On the server with the large pool there are only certain ports on the HBA that I can use, otherwise disks will be missing after a reboot and my ZFS pool won't go online.
I tried different firmware on the HBA. I tried the mpr.ko module from the broadcom site. (I replaced the one in /boot/kernel?)
I tested all the things above with a Linux as OS and everything seems to work.
 
 
Is there anything I'm missing? A command that can reset the SAS components?
 
 
FreeBSD version: 11.1-RELEASE-p11
HBA: broadcom lsi 9305-16e (latest firmware)
JBOD:SC847E2C-R1K28JBOD (two expanders, internally daisy chained)