problems with SAS JBODs 2

[Oliver Sech]

I tested a few additional things. I don't think this is a multipath, daisy chain nor a SAS wide ports problem.
I can reproduce the problem with just a single connection to an Expander/JBOD.

Test:
* physically disconnect all shelves
* reboot system
* connect one shelf via SAS cable
* check number of disks (after a reboot everything always shows up)
* disconnect the shelf and wait (geom disk list still shows most disks.)
* connect the shelf (missing disks)

Tested Hardware:
* Supermicro SAS3 847E2C-R1K28JBOD     + SAS3 LSI 9305-16e ( internal daisy chain + wide links) 
* Supermicro SAS3 847E2C-R1K28JBOD     + SAS3 LSI 9305-16e (straight HBA <-> EXPANDER connection. (no wide links, no daisy chain))
* Supermicro SAS2 SC847E26-RJBOD1      + SAS3 LSI 9305-16e (internal daisy chain)
* Promise    SAS2 VTrak 830            + SAS3 LSI 9305-16e (straight HBA <-> EXPANDER connection.)



On 07/04/2018 12:15 PM, Oliver Sech wrote:
>> 1) Are the expanders daisy chained?  Some SAS expanders don't work reliably
>> when daisy chained.   Best to direct connect each one to the server.
> At the moment I have 1 JBOD connected to 1 HBA Port with 1 cable (4 lanes?).
> Unfortunately the JBOD has 24 slots in the front and 20 in the back and, those are connected via a internal SAS daisy chaining.
> I could rewire and connect each backplane directly to the server, but unfortunately I do not have enough ports..
> 
> JOBD Model: Supermicro 847E2C-R1K28JBOD
> 
>> 2) Are the expanders connected in multipath or single path?  You need
>> geom_multipath if you're going to do that.
> See answer 1. There is a single path from the host to the first expander.
> 
>> 3) Are you attempting to use wide ports (two SAS cables connecting each
>> expander to the HBA).  If do, you'll need to make sure that each pair of
>> SAS cables goes to the same HBA chip (not merely the same card, as some
>> cards contain two HBA chips).
> see 1. The last time I opened one of those JBODs there were 8 SAS cables between the Front and Back expander. I assume that wide ports are being used.
> (2 expanders per backplane as well)
> 
>> 4) Are you trying to remove an expander while ZFS is active on that
>> expander?  That will suspend your pool, and ZFS doesn't always recover from
>> a suspended state.
> I'm testing with a new unused disk shelf that was never part of the ZFS pool. There were
> _______________________________________________
> freebsd-scsi at freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-scsi
> To unsubscribe, send any mail to "freebsd-scsi-unsubscribe at freebsd.org"