LSI2008 controller clobbers first disk with new LSI mps driver

Mon Feb 27 23:41:38 UTC 2012

On 12-02-27 02:59 PM, Jason Wolfe wrote:
> On Wed, Feb 22, 2012 at 9:11 AM, Desai, Kashyap<Kashyap.Desai at lsi.com>  wrote:
>>
>>
>>> -----Original Message-----
>>> From: Douglas Gilbert [mailto:dgilbert at interlog.com]
>>> Sent: Wednesday, February 22, 2012 8:52 PM
>>> To: Desai, Kashyap
>>> Cc: Jason Wolfe; freebsd-scsi at freebsd.org; McConnell, Stephen
>>> Subject: Re: LSI2008 controller clobbers first disk with new LSI mps
>>> driver
>>>
>>> On 12-02-22 03:39 AM, Desai, Kashyap wrote:
>>>> Here is a possible root cause of this issue.
>>>>
>>>> Enclosure which you are using in your setup (might be) not configured
>>> properly.
>>>>
>>>> You have Enclosure with 12 Slots + 1 SES Device.
>>>> See below detail from the log.
>>>>
>>>>      EventDataLength: 5
>>>>      AckRequired: 0
>>>>      Event: SasEnclDeviceStatusChange (0x1d)
>>>>      EventContext: 0x0
>>>>      EnclosureHandle: 0x2
>>>>      ReasonCode: Added
>>>>      PhysicalPort: 0
>>>>      NumSlots: 13
>>>>      StartSlot: 0
>>>>      PhyBits: 0xff
>>>>
>>>> StartSlot is 0 in this case.
>>>> Correct behavior should be each device on your enclosure must have
>>> different slot number starting from 0 till 12.
>>>> I have doubt that SES device has not configured well and it is using
>>> slot-0 as default. This can create issue for actual device which is
>>> connected to slot-0.
>>>> So In your setup you will have slot-0 till slot-11 assigned for actual
>>> Phys of your enclosures and again slot-0 is assigned for SES device
>>> instead of Slot-12.
>>>
>>> No. SAS-2 expanders typically have an integral SES device on an
>>> expander _virtual_ phy (see SMP DISCOVER (LIST) response). Once
>>> you see that virtual phy flag the slot number is irrelevant.
>>
>> Doug,
>>
>> I need some more info so that I can understand your point better.
>>
>> I have one Enclosure setup on FreeBSD. Here is smp_discover output. (smp_discover_list is failing for me)
>>
>> phy   0: inaccessible (phy vacant)
>>   phy   1: inaccessible (phy vacant)
>>   phy   2: inaccessible (phy vacant)
>>   phy   3: inaccessible (phy vacant)
>>   phy   4:S:attached:[500605b012345888:03  i(SSP+STP+SMP)]  6 Gbps
>>   phy   5:S:attached:[500605b012345888:02  i(SSP+STP+SMP)]  6 Gbps
>>   phy   6:S:attached:[500605b012345888:01  i(SSP+STP+SMP)]  6 Gbps
>>   phy   7:S:attached:[500605b012345888:00  i(SSP+STP+SMP)]  6 Gbps
>>   phy  12:D:attached:[5000c5003bc2c389:00  t(SSP)]  6 Gbps
>>   phy  13:D:attached:[500000e116ee91e2:00  t(SSP)]  6 Gbps
>>   phy  14:D:attached:[5000c5003bc308e5:00  t(SSP)]  6 Gbps
>>   phy  15:D:attached:[5000c5003bc2f0d1:00  t(SSP)]  6 Gbps
>>   phy  16:D:attached:[5000c5003bc2ff3d:00  t(SSP)]  6 Gbps
>>   phy  17:D:attached:[5000c5003bae5fdd:00  t(SSP)]  6 Gbps
>>   phy  18:D:attached:[5000c5003bae5eb1:00  t(SSP)]  6 Gbps
>>   phy  19:D:attached:[5000c5003bc2d135:00  t(SSP)]  6 Gbps
>>   phy  20:D:attached:[5000c5003baea36d:00  t(SSP)]  6 Gbps
>>   phy  21:D:attached:[5000c5003bc2a8c9:00  t(SSP)]  6 Gbps
>>   phy  22:D:attached:[5000c5003bc237a9:00  t(SSP)]  6 Gbps
>>   phy  23:D:attached:[5000c5003bc2cec1:00  t(SSP)]  6 Gbps
>>   phy  24:D:attached:[500000e01d92cb52:00  t(SSP)]  3 Gbps
>>   phy  25:D:attached:[500000e01d74cfb2:00  t(SSP)]  3 Gbps
>>   phy  26:D:attached:[500000e01d656052:00  t(SSP)]  3 Gbps
>>   phy  27:D:attached:[500000e01d7cad52:00  t(SSP)]  3 Gbps
>>   phy  28:D:attached:[500c04f2b64cdd1c:00  t(SATA)]  3 Gbps
>>   phy  29:D:attached:[500c04f2b64cdd1d:00  t(SATA)]  3 Gbps
>>   phy  30:D:attached:[500000e01d73c262:00  t(SSP)]  3 Gbps
>>   phy  31:D:attached:[500000e01d536b22:00  t(SSP)]  3 Gbps
>>   phy  32:D:attached:[500000e01d92cab2:00  t(SSP)]  3 Gbps
>>   phy  33:D:attached:[500000e01afd8792:00  t(SSP)]  3 Gbps
>>   phy  34:D:attached:[5000c5003bc30301:00  t(SSP)]  6 Gbps
>>   phy  35:D:attached:[5000c5003bb09a69:00  t(SSP)]  6 Gbps
>>   phy  36:D:attached:[500c04f2b64cdd3d:00  V i(SSP) t(SSP)]  6 Gbps<--- This has virtual phy set.
>>
>> What I understood from your explanation is if we have virt_phy field set, we should not trust slot for that entry.
>> You are suggesting to use phy index instead of slot. Just for info: But how to see Slot details mapping with phy ?

Kashyap,
I haven't written a SAS discover algorithm but there
must be plenty of examples out there. One way to do it
is to find all the phy_ids attached to targets, in this
case there are SAS (SSP) and SATA targets. Each SATA
target phy_id will correspond to one SATA disk (or could be an
ATAPI device (e.g. DVD/BD player)). The SSP targets are a
bit trickier because two (or more) phys could be connected
to the same target (either a wide port or multiple (target)
ports). With a wide port each component phy has the same
attached SAS address (so above you have a wide initiator
port (phy ids 4,5,6,7) but no wide target ports). If a
SAS disk has multiple target ports connected, FreeBSD
probably has a device node for each. So for each SCSI (SSP)
target port you need a REPORT LUNS command issued on LUN 0
(or the REPORT LUNS well known logical unit) to find the
LUs it contains. A device node is created for each LU.

Anyway I'm sure many folks in LSI know the SAS discover
process better than I do. Ask them :-) Surely most of
the above is already done in your HBA's firmware.

BTW I don't think slot numbers are reliable and don't apply
to things on virtual phys so they will just cause you
problems when used in the discover process, as this thread
attests. The BIOS on LSI's HBAs does a discover process
but is only interested in bootable devices so SES devices
don't appear.

Doug Gilbert

> Kashyap,
>
> Let me know if there are any changes agreed upon, I'm happy to test
> out patches as this is affecting a large number of our machines.  I
> can only imagine the same for others as they start to upgrade, as this
> is standard SuperMicro hardware.
>
> Thanks,
> Jason
>