problems with SAS JBODs 2

[Stephen Mcconnell]

Hi Oliver, I can't get to your links. Can you try to send the logs in
another way?

Steve

> -----Original Message-----
> From: owner-freebsd-scsi at freebsd.org [mailto:owner-freebsd-
> scsi at freebsd.org] On Behalf Of Oliver Sech
> Sent: Tuesday, July 10, 2018 9:14 AM
> To: FreeBSD-scsi
> Subject: Re: problems with SAS JBODs 2
>
> I tested a few additional things. I don't think this is a multipath, daisy
> chain
> nor a SAS wide ports problem.
> I can reproduce the problem with just a single connection to an
> Expander/JBOD.
>
> Test:
> * physically disconnect all shelves
> * reboot system
> * connect one shelf via SAS cable
> * check number of disks (after a reboot everything always shows up)
> * disconnect the shelf and wait (geom disk list still shows most disks.)
> * connect the shelf (missing disks)
>
> Tested Hardware:
> * Supermicro SAS3 847E2C-R1K28JBOD     + SAS3 LSI 9305-16e ( internal
> daisy
> chain + wide links)
> * Supermicro SAS3 847E2C-R1K28JBOD     + SAS3 LSI 9305-16e (straight HBA
> <-
> > EXPANDER connection. (no wide links, no daisy chain))
> * Supermicro SAS2 SC847E26-RJBOD1      + SAS3 LSI 9305-16e (internal daisy
> chain)
> * Promise    SAS2 VTrak 830            + SAS3 LSI 9305-16e (straight HBA
> <->
> EXPANDER connection.)
>
>
>
> On 07/04/2018 12:15 PM, Oliver Sech wrote:
> >> 1) Are the expanders daisy chained?  Some SAS expanders don't work
> reliably
> >> when daisy chained.   Best to direct connect each one to the server.
> > At the moment I have 1 JBOD connected to 1 HBA Port with 1 cable (4
> lanes?).
> > Unfortunately the JBOD has 24 slots in the front and 20 in the back and,
> those are connected via a internal SAS daisy chaining.
> > I could rewire and connect each backplane directly to the server, but
> unfortunately I do not have enough ports..
> >
> > JOBD Model: Supermicro 847E2C-R1K28JBOD
> >
> >> 2) Are the expanders connected in multipath or single path?  You need
> >> geom_multipath if you're going to do that.
> > See answer 1. There is a single path from the host to the first
> > expander.
> >
> >> 3) Are you attempting to use wide ports (two SAS cables connecting each
> >> expander to the HBA).  If do, you'll need to make sure that each pair
> >> of
> >> SAS cables goes to the same HBA chip (not merely the same card, as some
> >> cards contain two HBA chips).
> > see 1. The last time I opened one of those JBODs there were 8 SAS cables
> between the Front and Back expander. I assume that wide ports are being
> used.
> > (2 expanders per backplane as well)
> >
> >> 4) Are you trying to remove an expander while ZFS is active on that
> >> expander?  That will suspend your pool, and ZFS doesn't always recover
> from
> >> a suspended state.
> > I'm testing with a new unused disk shelf that was never part of the ZFS
> pool. There were
> > _______________________________________________
> > freebsd-scsi at freebsd.org mailing list
> > https://lists.freebsd.org/mailman/listinfo/freebsd-scsi
> > To unsubscribe, send any mail to "freebsd-scsi-unsubscribe at freebsd.org"
> _______________________________________________
> freebsd-scsi at freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-scsi
> To unsubscribe, send any mail to "freebsd-scsi-unsubscribe at freebsd.org"