ses/pass devices (enclosure/processor devices) not all showing up?

[Alan Somers]

On Wed, Sep 9, 2015 at 5:17 PM, John De Boskey <jwd at freebsd.org> wrote:
> ----- Alan Somers's Original Message -----
>> On Tue, Sep 8, 2015 at 9:35 PM, John De Boskey <jwd at freebsd.org> wrote:
>> > Hi Folks -
>> >
>> >    I have a shelf with 84 sata drives. All drives show up
>> > correctly and are accessible. The shelf appears to have
>> > multiple processor devices and one enclosure device internally.
>> > For instance:
>> >
>> > # camcontrol devlist | grep XYRATEX
>> > <XYRATEX DEFAULT-SD-R24 3034>      at scbus7 target 159 lun 0 (pass18)
>> > <XYRATEX DEFAULT-SD-R36 3034>      at scbus7 target 188 lun 0 (pass47)
>> > <XYRATEX DEFAULT-SD-R36 3034>      at scbus7 target 217 lun 0 (pass76)
>> > <XYRATEX DEFAULT-SD-R24 3034>      at scbus7 target 232 lun 0 (pass91)
>> >
>> > # camcontrol devlist | grep ses
>> > <DELL SC280-01-E6EBD 3034>         at scbus7 target 144 lun 0 (ses0,pass3)
>> >
>> > # camcontrol smprg pass18 | grep 'Number of Phys:'
>> > Number of Phys: 25
>> > # camcontrol smprg pass47 | grep 'Number of Phys:'
>> > Number of Phys: 37
>> > # camcontrol smprg pass76 | grep 'Number of Phys:'
>> > Number of Phys: 37
>> > # camcontrol smprg pass91 | grep 'Number of Phys:'
>> > Number of Phys: 25
>> > # camcontrol smprg ses0 | grep 'Number of Phys:'
>> > Number of Phys: 37
>> >
>> > # camcontrol smpphylist pass18
>> > 25 PHYs:
>> > PHY  Attached SAS Address
>> >   0  0x5000c500585f4b52   <SEAGATE ST4000NM0023 GE09>       (pass4,da0)
>
> ...
>
>> > I have linked dmesg, camcontrol devlist, and sas2ircu output below:
>> >
>> > http://people.freebsd.org/~jwd/sespass/dmesg.txt   mps messaging enabled.
>> >
>> > http://people.freebsd.org/~jwd/sespass/devlist.txt
>> >
>> > http://people.freebsd.org/~jwd/sespass/sas2ircu.txt
>> >
>> > Thanks,
>> > John
>
> Hi Alan,
>
>> I'm not sure exactly what you're asking.  Are you just wondering why
>
>    In a simple form, I'm trying to figure out why 'camcontrol smpphyslist ses0'
> produces no output, while 'camcontrol smpphyslist pass18' does.
>
>> your pass devices don't also have ses nodes?  I think I know why.
>> Your JBOD problem has five SAS expander chips, though ses0 might
>> actually be some other kind of SAS target chip.  pass18, pass47,
>> pass76, and pass91 are configured to report ses0's SAS Address as the
>> address of their SEP.  The LSI HBA's interpretation is that there is
>
>    Could you expand on the concept of the SAS Address being reported?
> When I look at the address of the devices in the sas2ircu output they
> are unique.  For example:


Each expander IC actually contains several logically independent
devices.  First there is the expander itself.  FreeBSD doesn't have
any visibility into that.  But you can see the address if you do
"sg_ses -p 10 ses0" and look for "attached SAS address".  In your
case, you should see four unique attached SAS addresses.  Second,
there is the SEP (SCSI Enclosure Processor).  This is the SCSI target
that ends up getting a device node in FreeBSD.  Third, the expander
will automatically generate a SAS address for any attached SATA
devices.

Now, I don't understand all the details, but somehow your four slave
expanders are telling your HBA that their disks are managed by the SEP
whose SAS address is the address of ses0.  As you've seen, that
information is not contained with the SMP Report General response.
It's probably some other SMP command, but I don't know which.


>
> Device is a Processor device
>   Enclosure #                             : 2
>   Slot #                                  : 1
>   SAS Address                             : 50050cc-1-0d2f-fabe
>   State                                   : Standby (SBY)
>   Manufacturer                            : XYRATEX
>   Model Number                            : DEFAULT-SD-R24
>   Firmware Revision                       : 3034
>   Serial No                               : PMCSIERA
>
> Device is a Processor device
>   Enclosure #                             : 2
>   Slot #                                  : 2
>   SAS Address                             : 50050cc-1-0d2f-fafe
>   State                                   : Standby (SBY)
>   Manufacturer                            : XYRATEX
>   Model Number                            : DEFAULT-SD-R36
>   Firmware Revision                       : 3034
>   Serial No                               : PMCSIERA
>
>    ie: they show as unique. However, they don't appear to be unique
> as reported by camcontrol smprg. I haven't been able to determine
> where in the code this linkage is being done yet.
>
>
>> only one SES processor.  So FreeBSD reports one ses device, and the
>> other expanders just show up as pass devices.  Unless the manufacturer
>> royally screwed up (I doubt it), ses0 will report info for all 84
>> disks in its SES status pages.
>
> camcontrol smpphylist ses0 - returns no information.
> camcontrol smpphylist pass18 - returns a drive list
> camcontrol smpphylist pass47 - returns a drive list
> camcontrol smpphylist pass76 - returns a drive list
> camcontrol smpphylist pass91 - returns a drive list
>
>> Bill is right, sg3_utils is your friend.  However, I doubt you'll see
>> any secondary subenclosures.  That feature isn't much used, and it's
>> not necessary in order for ses0 to report all 84 drives.  Running
>> these two commands will probably tell you most of what you need to
>> know:
>>
>> sg_ses -p 1 ses0
>> sg_ses -p 2 ses0
>
>    I have run the above commands and linked the output below.
>
> http://people.freebsd.org/~jwd/sespass/sgsesp1.txt
>
> http://people.freebsd.org/~jwd/sespass/sgsesp2.txt   descriptor for each disk
>
>    Thoughts?


It's as I expected.  ses0 is reporting all 84 disk slots.  You can use
sg_ses on ses0 for pretty much all drive management tasks.  The only
reason to use smpphylist and smppc, IMHO, is to disable a drive's phy.
That's mostly useful for simulating a drive pull, or for disabling a
device that is broken and doesn't probe cleanly.

-Alan


>
> Thanks,
> John
>
>> -Alan