problems with SAS JBODs 2

[Alan Somers]

On Tue, Jul 3, 2018 at 6:28 AM, Oliver Sech <CrimsonThunder at gmx.net> wrote:

> Hi!
>
> I use FreeBSD with for a large ZFS pool (over 1PB) and I recently
> encountered a lot of problems with the JBODs. Generally everything works
> fine until I replug the shelves.
>
> When I start with a clean system and attach a single shelf every thing
> seems fine.
> -> 44 disks show up, I can use the enclosure services (sesutil) and the
> system continues to run without problems.
> Once I disconnect the SAS cable, wait until all devices disapear and
> reconnect I get all sorts of problems.
> -> a random number of disks shows up and the enclosure "ses" do not show up
> Once I restart the system I can start over again.
>
> On the server with the large pool there are only certain ports on the HBA
> that I can use, otherwise disks will be missing after a reboot and my ZFS
> pool won't go online.
> I tried different firmware on the HBA. I tried the mpr.ko module from the
> broadcom site. (I replaced the one in /boot/kernel?)
> I tested all the things above with a Linux as OS and everything seems to
> work.
>
>
> Is there anything I'm missing? A command that can reset the SAS components?
>
>
> FreeBSD version: 11.1-RELEASE-p11
> HBA: broadcom lsi 9305-16e (latest firmware)
> JBOD:SC847E2C-R1K28JBOD (two expanders, internally daisy chained)
> _______________________________________________
> freebsd-scsi at freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-scsi
> To unsubscribe, send any mail to "freebsd-scsi-unsubscribe at freebsd.org"
>

1) Are the expanders daisy chained?  Some SAS expanders don't work reliably
when daisy chained.   Best to direct connect each one to the server.
2) Are the expanders connected in multipath or single path?  You need
geom_multipath if you're going to do that.
3) Are you attempting to use wide ports (two SAS cables connecting each
expander to the HBA).  If do, you'll need to make sure that each pair of
SAS cables goes to the same HBA chip (not merely the same card, as some
cards contain two HBA chips).
4) Are you trying to remove an expander while ZFS is active on that
expander?  That will suspend your pool, and ZFS doesn't always recover from
a suspended state.

-Alan