ses/pass devices (enclosure/processor devices) not all showing up?

[Alan Somers]

On Tue, Sep 8, 2015 at 9:35 PM, John De Boskey <jwd at freebsd.org> wrote:
> Hi Folks -
>
>    I have a shelf with 84 sata drives. All drives show up
> correctly and are accessible. The shelf appears to have
> multiple processor devices and one enclosure device internally.
> For instance:
>
> # camcontrol devlist | grep XYRATEX
> <XYRATEX DEFAULT-SD-R24 3034>      at scbus7 target 159 lun 0 (pass18)
> <XYRATEX DEFAULT-SD-R36 3034>      at scbus7 target 188 lun 0 (pass47)
> <XYRATEX DEFAULT-SD-R36 3034>      at scbus7 target 217 lun 0 (pass76)
> <XYRATEX DEFAULT-SD-R24 3034>      at scbus7 target 232 lun 0 (pass91)
>
> # camcontrol devlist | grep ses
> <DELL SC280-01-E6EBD 3034>         at scbus7 target 144 lun 0 (ses0,pass3)
>
> # camcontrol smprg pass18 | grep 'Number of Phys:'
> Number of Phys: 25
> # camcontrol smprg pass47 | grep 'Number of Phys:'
> Number of Phys: 37
> # camcontrol smprg pass76 | grep 'Number of Phys:'
> Number of Phys: 37
> # camcontrol smprg pass91 | grep 'Number of Phys:'
> Number of Phys: 25
> # camcontrol smprg ses0 | grep 'Number of Phys:'
> Number of Phys: 37
>
> # camcontrol smpphylist pass18
> 25 PHYs:
> PHY  Attached SAS Address
>   0  0x5000c500585f4b52   <SEAGATE ST4000NM0023 GE09>       (pass4,da0)
>   1  0x5000c500585f4476   <SEAGATE ST4000NM0023 GE09>       (pass5,da1)
>   2  0x5000c500585f6efe   <SEAGATE ST4000NM0023 GE09>       (pass6,da2)
>   3  0x5000c500585f14d6   <SEAGATE ST4000NM0023 GE09>       (pass7,da3)
>   4  0x5000c500585f683e   <SEAGATE ST4000NM0023 GE09>       (pass8,da4)
>   5  0x5000c500585f15e6   <SEAGATE ST4000NM0023 GE09>       (pass9,da5)
>   6  0x5000c500585f49ba   <SEAGATE ST4000NM0023 GE09>       (pass10,da6)
>   7  0x5000c500585f14ea   <SEAGATE ST4000NM0023 GE09>       (pass11,da7)
>   8  0x5000c500585f5ec6   <SEAGATE ST4000NM0023 GE09>       (pass12,da8)
>   9  0x5000c500585f0d12   <SEAGATE ST4000NM0023 GE09>       (pass13,da9)
>  10  0x5000c500585f620a   <SEAGATE ST4000NM0023 GE09>       (pass14,da10)
>  11  0x5000c500585f0d1e   <SEAGATE ST4000NM0023 GE09>       (pass15,da11)
>  12  0x5000c500585f64ae   <SEAGATE ST4000NM0023 GE09>       (pass16,da12)
>  13  0x5000c500585f27aa   <SEAGATE ST4000NM0023 GE09>       (pass17,da13)
>  14  0x0000000000000000
>  15  0x0000000000000000
>  16  0x0000000000000000
>  17  0x0000000000000000
>  18  0x0000000000000000
>  19  0x0000000000000000
>  20  0x50050cc1177513bf
>  21  0x50050cc1177513bf
>  22  0x50050cc1177513bf
>  23  0x50050cc1177513bf
>  24  0x50050cc10d2fe7be
>
>
> (sas2ircu processor devices)
>
> # camcontrol smpphylist pass18 -q | wc -l
>       14
> # camcontrol smpphylist pass47 -q | wc -l
>       28
> # camcontrol smpphylist pass76 -q | wc -l
>       28
> # camcontrol smpphylist pass91 -q | wc -l
>       14
>
> Which adds up to the 84 expected drives. No devices show up under ses0:
>
> (sas2ircu enclosure services device)
>
> # camcontrol smpphylist ses0 -q | wc -l
>        0
>
> I don't know that I understand the hierarchy of the backplane attachments.
> Should the 4 pass only processor devices actually be ses devices? Or should
> they be linked up under ses0 and all the devices show up under ses0?
>
> Any thoughts or ideas on how to fix this up(debug) would be appreciated.
>
> I have linked dmesg, camcontrol devlist, and sas2ircu output below:
>
> http://people.freebsd.org/~jwd/sespass/dmesg.txt   mps messaging enabled.
>
> http://people.freebsd.org/~jwd/sespass/devlist.txt
>
> http://people.freebsd.org/~jwd/sespass/sas2ircu.txt
>
> Thanks,
> John

I'm not sure exactly what you're asking.  Are you just wondering why
your pass devices don't also have ses nodes?  I think I know why.
Your JBOD problem has five SAS expander chips, though ses0 might
actually be some other kind of SAS target chip.  pass18, pass47,
pass76, and pass91 are configured to report ses0's SAS Address as the
address of their SEP.  The LSI HBA's interpretation is that there is
only one SES processor.  So FreeBSD reports one ses device, and the
other expanders just show up as pass devices.  Unless the manufacturer
royally screwed up (I doubt it), ses0 will report info for all 84
disks in its SES status pages.

Bill is right, sg3_utils is your friend.  However, I doubt you'll see
any secondary subenclosures.  That feature isn't much used, and it's
not necessary in order for ses0 to report all 84 drives.  Running
these two commands will probably tell you most of what you need to
know:

sg_ses -p 1 ses0
sg_ses -p 2 ses0

-Alan