problems with SAS JBODs 2

[Oliver Sech]

sorry for sending dead links earlier...
(Here is a link for the previous files: https://www.dropbox.com/s/5dlwizrzy48vme3/freebsd_sas.zip?dl=0 )
Here is the link for the new logs: https://www.dropbox.com/s/7bbt1fipg2a50oq/freebsd_sas2.zip?dl=0

notes:
logfile: "1_clean_boot_without_shelves_dmesg"

while booting with no shelves are attached it actually resets something:
mpr0: mpr_mapping_check_devices: Enclosure XX is missing from the topology. Update its missing count.
mpr0: _mapping_commit_enc_entry: Writing DPM entry XX for enclosure.

logfile: "3_shelf_disconnected_geom"
the only disks that really are connected are ada0, ada1, da0
everything else cannot get accessed

Hardware: 
Promise    SAS2 VTrak 830 (Full of SATA disks) +  LSI 9305-16e

Oliver

On 07/10/2018 05:48 PM, Stephen Mcconnell wrote:
> Ken, I looked at the logs and I don't see anything in them that suggests
> that the driver is not adding any of the devices. In fact, I don't see
> anything that looks strange at all. This looks like a different problem than
> the other one you mentioned. What do you think?
> 
> Steve
> 
>> -----Original Message-----
>> From: Stephen Mcconnell [mailto:stephen.mcconnell at broadcom.com]
>> Sent: Tuesday, July 10, 2018 9:28 AM
>> To: 'Oliver Sech'; 'FreeBSD-scsi'
>> Subject: RE: problems with SAS JBODs 2
>>
>> Hi Oliver, I can't get to your links. Can you try to send the logs in
>> another
>> way?
>>
>> Steve
>>
>>> -----Original Message-----
>>> From: owner-freebsd-scsi at freebsd.org [mailto:owner-freebsd-
>>> scsi at freebsd.org] On Behalf Of Oliver Sech
>>> Sent: Tuesday, July 10, 2018 9:14 AM
>>> To: FreeBSD-scsi
>>> Subject: Re: problems with SAS JBODs 2
>>>
>>> I tested a few additional things. I don't think this is a multipath,
>>> daisy
>> chain
>>> nor a SAS wide ports problem.
>>> I can reproduce the problem with just a single connection to an
>>> Expander/JBOD.
>>>
>>> Test:
>>> * physically disconnect all shelves
>>> * reboot system
>>> * connect one shelf via SAS cable
>>> * check number of disks (after a reboot everything always shows up)
>>> * disconnect the shelf and wait (geom disk list still shows most disks.)
>>> * connect the shelf (missing disks)
>>>
>>> Tested Hardware:
>>> * Supermicro SAS3 847E2C-R1K28JBOD     + SAS3 LSI 9305-16e ( internal
>> daisy
>>> chain + wide links)
>>> * Supermicro SAS3 847E2C-R1K28JBOD     + SAS3 LSI 9305-16e (straight HBA
>> <-
>>>> EXPANDER connection. (no wide links, no daisy chain))
>>> * Supermicro SAS2 SC847E26-RJBOD1      + SAS3 LSI 9305-16e (internal
>>> daisy
>>> chain)
>>> * Promise    SAS2 VTrak 830            + SAS3 LSI 9305-16e (straight HBA
>>> <->
>>> EXPANDER connection.)
>>>
>>>
>>>
>>> On 07/04/2018 12:15 PM, Oliver Sech wrote:
>>>>> 1) Are the expanders daisy chained?  Some SAS expanders don't work
>>> reliably
>>>>> when daisy chained.   Best to direct connect each one to the server.
>>>> At the moment I have 1 JBOD connected to 1 HBA Port with 1 cable (4
>>> lanes?).
>>>> Unfortunately the JBOD has 24 slots in the front and 20 in the back
>>>> and,
>>> those are connected via a internal SAS daisy chaining.
>>>> I could rewire and connect each backplane directly to the server, but
>>> unfortunately I do not have enough ports..
>>>>
>>>> JOBD Model: Supermicro 847E2C-R1K28JBOD
>>>>
>>>>> 2) Are the expanders connected in multipath or single path?  You need
>>>>> geom_multipath if you're going to do that.
>>>> See answer 1. There is a single path from the host to the first
>>>> expander.
>>>>
>>>>> 3) Are you attempting to use wide ports (two SAS cables connecting
>> each
>>>>> expander to the HBA).  If do, you'll need to make sure that each pair
>>>>> of
>>>>> SAS cables goes to the same HBA chip (not merely the same card, as
>> some
>>>>> cards contain two HBA chips).
>>>> see 1. The last time I opened one of those JBODs there were 8 SAS
>>>> cables
>>> between the Front and Back expander. I assume that wide ports are being
>>> used.
>>>> (2 expanders per backplane as well)
>>>>
>>>>> 4) Are you trying to remove an expander while ZFS is active on that
>>>>> expander?  That will suspend your pool, and ZFS doesn't always
>>>>> recover
>>> from
>>>>> a suspended state.
>>>> I'm testing with a new unused disk shelf that was never part of the
>>>> ZFS
>>> pool. There were
>>>> _______________________________________________
>>>> freebsd-scsi at freebsd.org mailing list
>>>> https://lists.freebsd.org/mailman/listinfo/freebsd-scsi
>>>> To unsubscribe, send any mail to
>>>> "freebsd-scsi-unsubscribe at freebsd.org"
>>> _______________________________________________
>>> freebsd-scsi at freebsd.org mailing list
>>> https://lists.freebsd.org/mailman/listinfo/freebsd-scsi
>>> To unsubscribe, send any mail to "freebsd-scsi-unsubscribe at freebsd.org"