problems with SAS JBODs 2

[Stephen Mcconnell]

Ken, I looked at the logs and I don't see anything in them that suggests
that the driver is not adding any of the devices. In fact, I don't see
anything that looks strange at all. This looks like a different problem than
the other one you mentioned. What do you think?

Steve

> -----Original Message-----
> From: Stephen Mcconnell [mailto:stephen.mcconnell at broadcom.com]
> Sent: Tuesday, July 10, 2018 9:28 AM
> To: 'Oliver Sech'; 'FreeBSD-scsi'
> Subject: RE: problems with SAS JBODs 2
>
> Hi Oliver, I can't get to your links. Can you try to send the logs in
> another
> way?
>
> Steve
>
> > -----Original Message-----
> > From: owner-freebsd-scsi at freebsd.org [mailto:owner-freebsd-
> > scsi at freebsd.org] On Behalf Of Oliver Sech
> > Sent: Tuesday, July 10, 2018 9:14 AM
> > To: FreeBSD-scsi
> > Subject: Re: problems with SAS JBODs 2
> >
> > I tested a few additional things. I don't think this is a multipath,
> > daisy
> chain
> > nor a SAS wide ports problem.
> > I can reproduce the problem with just a single connection to an
> > Expander/JBOD.
> >
> > Test:
> > * physically disconnect all shelves
> > * reboot system
> > * connect one shelf via SAS cable
> > * check number of disks (after a reboot everything always shows up)
> > * disconnect the shelf and wait (geom disk list still shows most disks.)
> > * connect the shelf (missing disks)
> >
> > Tested Hardware:
> > * Supermicro SAS3 847E2C-R1K28JBOD     + SAS3 LSI 9305-16e ( internal
> daisy
> > chain + wide links)
> > * Supermicro SAS3 847E2C-R1K28JBOD     + SAS3 LSI 9305-16e (straight HBA
> <-
> > > EXPANDER connection. (no wide links, no daisy chain))
> > * Supermicro SAS2 SC847E26-RJBOD1      + SAS3 LSI 9305-16e (internal
> > daisy
> > chain)
> > * Promise    SAS2 VTrak 830            + SAS3 LSI 9305-16e (straight HBA
> > <->
> > EXPANDER connection.)
> >
> >
> >
> > On 07/04/2018 12:15 PM, Oliver Sech wrote:
> > >> 1) Are the expanders daisy chained?  Some SAS expanders don't work
> > reliably
> > >> when daisy chained.   Best to direct connect each one to the server.
> > > At the moment I have 1 JBOD connected to 1 HBA Port with 1 cable (4
> > lanes?).
> > > Unfortunately the JBOD has 24 slots in the front and 20 in the back
> > > and,
> > those are connected via a internal SAS daisy chaining.
> > > I could rewire and connect each backplane directly to the server, but
> > unfortunately I do not have enough ports..
> > >
> > > JOBD Model: Supermicro 847E2C-R1K28JBOD
> > >
> > >> 2) Are the expanders connected in multipath or single path?  You need
> > >> geom_multipath if you're going to do that.
> > > See answer 1. There is a single path from the host to the first
> > > expander.
> > >
> > >> 3) Are you attempting to use wide ports (two SAS cables connecting
> each
> > >> expander to the HBA).  If do, you'll need to make sure that each pair
> > >> of
> > >> SAS cables goes to the same HBA chip (not merely the same card, as
> some
> > >> cards contain two HBA chips).
> > > see 1. The last time I opened one of those JBODs there were 8 SAS
> > > cables
> > between the Front and Back expander. I assume that wide ports are being
> > used.
> > > (2 expanders per backplane as well)
> > >
> > >> 4) Are you trying to remove an expander while ZFS is active on that
> > >> expander?  That will suspend your pool, and ZFS doesn't always
> > >> recover
> > from
> > >> a suspended state.
> > > I'm testing with a new unused disk shelf that was never part of the
> > > ZFS
> > pool. There were
> > > _______________________________________________
> > > freebsd-scsi at freebsd.org mailing list
> > > https://lists.freebsd.org/mailman/listinfo/freebsd-scsi
> > > To unsubscribe, send any mail to
> > > "freebsd-scsi-unsubscribe at freebsd.org"
> > _______________________________________________
> > freebsd-scsi at freebsd.org mailing list
> > https://lists.freebsd.org/mailman/listinfo/freebsd-scsi
> > To unsubscribe, send any mail to "freebsd-scsi-unsubscribe at freebsd.org"