LSI2008 controller clobbers first disk with new LSI mps driver

Wed Feb 29 16:38:35 UTC 2012

Hi Jason,

I have started discussion with LSI internal folks to get better clarity on this issue. Since our key person is on vacation, we may get clarity on this next week.
I cannot provide some temporary workaround in upstream(because this is against our design), but if you want to use for your environment, I can provide you some temporary patch.

Doug,

Thanks for providing your view and I have convey this to our architect.

~ Kashyap

> -----Original Message-----
> From: Douglas Gilbert [mailto:dgilbert at interlog.com]
> Sent: Tuesday, February 28, 2012 5:11 AM
> To: Jason Wolfe
> Cc: Desai, Kashyap; freebsd-scsi at freebsd.org; McConnell, Stephen
> Subject: Re: LSI2008 controller clobbers first disk with new LSI mps
> driver
> 
> On 12-02-27 02:59 PM, Jason Wolfe wrote:
> > On Wed, Feb 22, 2012 at 9:11 AM, Desai, Kashyap<Kashyap.Desai at lsi.com>
> wrote:
> >>
> >>
> >>> -----Original Message-----
> >>> From: Douglas Gilbert [mailto:dgilbert at interlog.com]
> >>> Sent: Wednesday, February 22, 2012 8:52 PM
> >>> To: Desai, Kashyap
> >>> Cc: Jason Wolfe; freebsd-scsi at freebsd.org; McConnell, Stephen
> >>> Subject: Re: LSI2008 controller clobbers first disk with new LSI mps
> >>> driver
> >>>
> >>> On 12-02-22 03:39 AM, Desai, Kashyap wrote:
> >>>> Here is a possible root cause of this issue.
> >>>>
> >>>> Enclosure which you are using in your setup (might be) not
> configured
> >>> properly.
> >>>>
> >>>> You have Enclosure with 12 Slots + 1 SES Device.
> >>>> See below detail from the log.
> >>>>
> >>>>      EventDataLength: 5
> >>>>      AckRequired: 0
> >>>>      Event: SasEnclDeviceStatusChange (0x1d)
> >>>>      EventContext: 0x0
> >>>>      EnclosureHandle: 0x2
> >>>>      ReasonCode: Added
> >>>>      PhysicalPort: 0
> >>>>      NumSlots: 13
> >>>>      StartSlot: 0
> >>>>      PhyBits: 0xff
> >>>>
> >>>> StartSlot is 0 in this case.
> >>>> Correct behavior should be each device on your enclosure must have
> >>> different slot number starting from 0 till 12.
> >>>> I have doubt that SES device has not configured well and it is
> using
> >>> slot-0 as default. This can create issue for actual device which is
> >>> connected to slot-0.
> >>>> So In your setup you will have slot-0 till slot-11 assigned for
> actual
> >>> Phys of your enclosures and again slot-0 is assigned for SES device
> >>> instead of Slot-12.
> >>>
> >>> No. SAS-2 expanders typically have an integral SES device on an
> >>> expander _virtual_ phy (see SMP DISCOVER (LIST) response). Once
> >>> you see that virtual phy flag the slot number is irrelevant.
> >>
> >> Doug,
> >>
> >> I need some more info so that I can understand your point better.
> >>
> >> I have one Enclosure setup on FreeBSD. Here is smp_discover output.
> (smp_discover_list is failing for me)
> >>
> >> phy   0: inaccessible (phy vacant)
> >>   phy   1: inaccessible (phy vacant)
> >>   phy   2: inaccessible (phy vacant)
> >>   phy   3: inaccessible (phy vacant)
> >>   phy   4:S:attached:[500605b012345888:03  i(SSP+STP+SMP)]  6 Gbps
> >>   phy   5:S:attached:[500605b012345888:02  i(SSP+STP+SMP)]  6 Gbps
> >>   phy   6:S:attached:[500605b012345888:01  i(SSP+STP+SMP)]  6 Gbps
> >>   phy   7:S:attached:[500605b012345888:00  i(SSP+STP+SMP)]  6 Gbps
> >>   phy  12:D:attached:[5000c5003bc2c389:00  t(SSP)]  6 Gbps
> >>   phy  13:D:attached:[500000e116ee91e2:00  t(SSP)]  6 Gbps
> >>   phy  14:D:attached:[5000c5003bc308e5:00  t(SSP)]  6 Gbps
> >>   phy  15:D:attached:[5000c5003bc2f0d1:00  t(SSP)]  6 Gbps
> >>   phy  16:D:attached:[5000c5003bc2ff3d:00  t(SSP)]  6 Gbps
> >>   phy  17:D:attached:[5000c5003bae5fdd:00  t(SSP)]  6 Gbps
> >>   phy  18:D:attached:[5000c5003bae5eb1:00  t(SSP)]  6 Gbps
> >>   phy  19:D:attached:[5000c5003bc2d135:00  t(SSP)]  6 Gbps
> >>   phy  20:D:attached:[5000c5003baea36d:00  t(SSP)]  6 Gbps
> >>   phy  21:D:attached:[5000c5003bc2a8c9:00  t(SSP)]  6 Gbps
> >>   phy  22:D:attached:[5000c5003bc237a9:00  t(SSP)]  6 Gbps
> >>   phy  23:D:attached:[5000c5003bc2cec1:00  t(SSP)]  6 Gbps
> >>   phy  24:D:attached:[500000e01d92cb52:00  t(SSP)]  3 Gbps
> >>   phy  25:D:attached:[500000e01d74cfb2:00  t(SSP)]  3 Gbps
> >>   phy  26:D:attached:[500000e01d656052:00  t(SSP)]  3 Gbps
> >>   phy  27:D:attached:[500000e01d7cad52:00  t(SSP)]  3 Gbps
> >>   phy  28:D:attached:[500c04f2b64cdd1c:00  t(SATA)]  3 Gbps
> >>   phy  29:D:attached:[500c04f2b64cdd1d:00  t(SATA)]  3 Gbps
> >>   phy  30:D:attached:[500000e01d73c262:00  t(SSP)]  3 Gbps
> >>   phy  31:D:attached:[500000e01d536b22:00  t(SSP)]  3 Gbps
> >>   phy  32:D:attached:[500000e01d92cab2:00  t(SSP)]  3 Gbps
> >>   phy  33:D:attached:[500000e01afd8792:00  t(SSP)]  3 Gbps
> >>   phy  34:D:attached:[5000c5003bc30301:00  t(SSP)]  6 Gbps
> >>   phy  35:D:attached:[5000c5003bb09a69:00  t(SSP)]  6 Gbps
> >>   phy  36:D:attached:[500c04f2b64cdd3d:00  V i(SSP) t(SSP)]  6 Gbps<-
> -- This has virtual phy set.
> >>
> >> What I understood from your explanation is if we have virt_phy field
> set, we should not trust slot for that entry.
> >> You are suggesting to use phy index instead of slot. Just for info:
> But how to see Slot details mapping with phy ?
> 
> Kashyap,
> I haven't written a SAS discover algorithm but there
> must be plenty of examples out there. One way to do it
> is to find all the phy_ids attached to targets, in this
> case there are SAS (SSP) and SATA targets. Each SATA
> target phy_id will correspond to one SATA disk (or could be an
> ATAPI device (e.g. DVD/BD player)). The SSP targets are a
> bit trickier because two (or more) phys could be connected
> to the same target (either a wide port or multiple (target)
> ports). With a wide port each component phy has the same
> attached SAS address (so above you have a wide initiator
> port (phy ids 4,5,6,7) but no wide target ports). If a
> SAS disk has multiple target ports connected, FreeBSD
> probably has a device node for each. So for each SCSI (SSP)
> target port you need a REPORT LUNS command issued on LUN 0
> (or the REPORT LUNS well known logical unit) to find the
> LUs it contains. A device node is created for each LU.
> 
> Anyway I'm sure many folks in LSI know the SAS discover
> process better than I do. Ask them :-) Surely most of
> the above is already done in your HBA's firmware.
> 
> 
> BTW I don't think slot numbers are reliable and don't apply
> to things on virtual phys so they will just cause you
> problems when used in the discover process, as this thread
> attests. The BIOS on LSI's HBAs does a discover process
> but is only interested in bootable devices so SES devices
> don't appear.
> 
> 
> Doug Gilbert
> 
> > Kashyap,
> >
> > Let me know if there are any changes agreed upon, I'm happy to test
> > out patches as this is affecting a large number of our machines.  I
> > can only imagine the same for others as they start to upgrade, as this
> > is standard SuperMicro hardware.
> >
> > Thanks,
> > Jason
> >