Inferring SAS expander topology

Tue May 30 20:09:19 UTC 2017

Thanks! It looks like smp_discover will do what I need, though is not particularly user-friendly for this specific application. 

I needed to use both "smp_discover sesX" in addition to "smp_discover --phy=Y --brief sesX" in order to correlate SAS interfaces with the addresses of devices attached to them.

	- .Dustin

> On May 30, 2017, at 12:31 PM, Alan Somers <asomers at freebsd.org> wrote:
> 
> On Tue, May 30, 2017 at 11:00 AM, Dustin Wenz <dustinwenz at ebureau.com> wrote:
>> I'm working on a server that has 6 LSI SAS expanders connected to it. Two of them are throwing intermittent command errors and need some attention. Because the host only has four external 8088 ports, I know some of the expanders are chained through each other.
>> 
>> Before I can send parts for replacement, I need to determine if the two problematic expanders are either daisy-chained or directly connected to the host. On Linux, this is trivial; I would just browse the device topology in /sys/. However, this machine is running FreeBSD 10.3, and I am unable to find a way to do this. I have sg3_utils available, as well as any build-in FreeBSD tools. Using this software, is it possible to determine if a SAS expander is chained off of another?
>> 
>>        - .Dustin
> 
> I don't think sg3_utils will help you.  You want sysutils/smp_utils
> instead.  If you install that and then run "smp_discover /dev/ses0",
> it will show you what each phy is connected to.  Usually, there will
> be four phys connected to the upstream port.  You can tell by their
> EUI64s whether they're connected directly to the HBA or to another
> expander.  And you can tell which expander by comparing the exact
> EUI64 to each other expander's SEP phy.  Note that with LSI expanders,
> the SEP's address usually differs from the expander's address by a few
> bits in the last byte.  For example, if the expander's address is
> 0x50000000000000ff, then the SEP's address might by
> 0x50000000000000ffd.  Here's some example output from one of my
> systems:
> 
> # smp_discover /dev/ses3
>  phy   0:U:attached:[500093d23000a000:00  t(SATA)]  6 Gbps
>  phy   1:U:attached:[500093d23000a001:00  t(SATA)]  6 Gbps
>  phy   2:U:attached:[500093d23000a002:00  t(SATA)]  6 Gbps
>  phy   3:U:attached:[500093d23000a003:00  t(SATA)]  6 Gbps
>  phy   4:U:attached:[500093d23000a17f:11 exp t(SMP)]  6 Gbps
>         <- Connected to the other expander
>  phy   5:U:attached:[500093d23000a17f:10 exp t(SMP)]  6 Gbps
>         <- Connected to the other expander
>  phy   6:U:attached:[500093d23000a17f:09 exp t(SMP)]  6 Gbps
>         <- Connected to the other expander
>  phy   7:U:attached:[500093d23000a17f:08 exp t(SMP)]  6 Gbps
>         <- Connected to the other expander
>  phy   8:U:attached:[500093d23000a1bf:11 exp t(SMP)]  6 Gbps
>         <- Connected to a third expander
>  phy   9:U:attached:[500093d23000a1bf:10 exp t(SMP)]  6 Gbps
>         <- Connected to a third expander
>  phy  10:U:attached:[500093d23000a1bf:09 exp t(SMP)]  6 Gbps
>        <- Connected to a third expander
>  phy  11:U:attached:[500093d23000a1bf:08 exp t(SMP)]  6 Gbps
>        <- Connected to a third expander
>  phy  16:U:attached:[500093d23000a010:00  t(SATA)]  6 Gbps
>  phy  17:U:attached:[500093d23000a011:00  t(SATA)]  6 Gbps
>  phy  18:U:attached:[500093d23000a012:00  t(SATA)]  6 Gbps
>  phy  19:U:attached:[500093d23000a013:00  t(SATA)]  6 Gbps
>  phy  20:U:attached:[500093d23000a014:00  t(SATA)]  6 Gbps
>  phy  21:U:attached:[500093d23000a015:00  t(SATA)]  6 Gbps
>  phy  22:U:attached:[500093d23000a016:00  t(SATA)]  6 Gbps
>  phy  23:U:attached:[500093d23000a017:00  t(SATA)]  6 Gbps
>  phy  25:U:attached:[500093d23000a019:00  t(SATA)]  6 Gbps
>  phy  26:U:attached:[500093d23000a01a:00  t(SATA)]  6 Gbps
>  phy  29:U:attached:[500093d23000a01d:00  t(SATA)]  6 Gbps
>  phy  30:U:attached:[500093d23000a01e:00  t(SATA)]  6 Gbps
>  phy  31:U:attached:[500093d23000a01f:00  t(SATA)]  6 Gbps
>  phy  33:U:attached:[500093d23000a021:00  t(SATA)]  6 Gbps
>  phy  34:U:attached:[500093d23000a022:00  t(SATA)]  6 Gbps
>  phy  35:U:attached:[500093d23000a023:00  t(SATA)]  6 Gbps
>  phy  36:U:attached:[500093d23000a024:00  t(SATA)]  6 Gbps
>  phy  37:U:attached:[500093d23000a025:00  t(SATA)]  6 Gbps
>  phy  38:U:attached:[500093d23000a026:00  t(SATA)]  6 Gbps
>  phy  39:U:attached:[500093d23000a027:00  t(SATA)]  6 Gbps
>  phy  40:U:attached:[500605b008a93990:04  i(SSP+STP+SMP)]  6 Gbps
>  <- Connected to an LSI HBA
>  phy  41:U:attached:[500605b008a93990:07  i(SSP+STP+SMP)]  6 Gbps
>  <- Connected to an LSI HBA
>  phy  42:U:attached:[500605b008a93990:05  i(SSP+STP+SMP)]  6 Gbps
>  <- Connected to an LSI HBA
>  phy  43:U:attached:[500605b008a93990:06  i(SSP+STP+SMP)]  6 Gbps
>  <- Connected to an LSI HBA
>  phy  48:D:attached:[500093d23000a03d:00  V i(SMP) t(SSP)]  12 Gbps
>      <- This expander's SEP
> 
> # smp_discover /dev/pass121
>  phy   0:U:attached:[500093d23000a140:00  t(SATA)]  6 Gbps
>  phy   1:U:attached:[500093d23000a141:00  t(SATA)]  6 Gbps
>  phy   2:U:attached:[500093d23000a142:00  t(SATA)]  6 Gbps
>  phy   3:U:attached:[500093d23000a143:00  t(SATA)]  6 Gbps
>  phy   4:U:attached:[500093d23000a144:00  t(SATA)]  6 Gbps
>  phy   5:U:attached:[500093d23000a145:00  t(SATA)]  6 Gbps
>  phy   6:U:attached:[500093d23000a146:00  t(SATA)]  6 Gbps
>  phy   7:U:attached:[500093d23000a147:00  t(SATA)]  6 Gbps
>  phy   8:U:attached:[500093d23000a03f:07 exp t(SMP)]  6 Gbps
>         <- Connected to the other expander
>  phy   9:U:attached:[500093d23000a03f:06 exp t(SMP)]  6 Gbps
>         <- Connected to the other expander
>  phy  10:U:attached:[500093d23000a03f:05 exp t(SMP)]  6 Gbps
>        <- Connected to the other expander
>  phy  11:U:attached:[500093d23000a03f:04 exp t(SMP)]  6 Gbps
>        <- Connected to the other expander
>  phy  12:U:attached:[500093d23000a14c:00  t(SATA)]  6 Gbps
>  phy  14:U:attached:[500093d23000a14e:00  t(SATA)]  6 Gbps
>  phy  15:U:attached:[500093d23000a14f:00  t(SATA)]  6 Gbps
>  phy  16:U:attached:[500093d23000a150:00  t(SATA)]  6 Gbps
>  phy  17:U:attached:[500093d23000a151:00  t(SATA)]  6 Gbps
>  phy  18:U:attached:[500093d23000a152:00  t(SATA)]  6 Gbps
>  phy  19:U:attached:[500093d23000a153:00  t(SATA)]  6 Gbps
>  phy  20:U:attached:[500093d23000a154:00  t(SATA)]  6 Gbps
>  phy  21:U:attached:[500093d23000a155:00  t(SATA)]  6 Gbps
>  phy  22:U:attached:[500093d23000a156:00  t(SATA)]  6 Gbps
>  phy  23:U:attached:[500093d23000a157:00  t(SATA)]  6 Gbps
>  phy  24:U:attached:[500093d23000a158:00  t(SATA)]  6 Gbps
>  phy  25:U:attached:[500093d23000a159:00  t(SATA)]  6 Gbps
>  phy  26:U:attached:[500093d23000a15a:00  t(SATA)]  6 Gbps
>  phy  27:U:attached:[500093d23000a15b:00  t(SATA)]  6 Gbps
>  phy  28:U:attached:[500093d23000a15c:00  t(SATA)]  6 Gbps
>  phy  29:U:attached:[500093d23000a15d:00  t(SATA)]  6 Gbps
>  phy  30:U:attached:[500093d23000a15e:00  t(SATA)]  6 Gbps
>  phy  32:U:attached:[500093d23000a160:00  t(SATA)]  6 Gbps
>  phy  33:U:attached:[500093d23000a161:00  t(SATA)]  6 Gbps
>  phy  34:U:attached:[500093d23000a162:00  t(SATA)]  6 Gbps
>  phy  35:U:attached:[500093d23000a163:00  t(SATA)]  6 Gbps
>  phy  36:U:attached:[500093d23000a164:00  t(SATA)]  6 Gbps
>  phy  37:U:attached:[500093d23000a165:00  t(SATA)]  6 Gbps
>  phy  38:U:attached:[500093d23000a166:00  t(SATA)]  6 Gbps
>  phy  39:U:attached:[500093d23000a167:00  t(SATA)]  6 Gbps
>  phy  40:D:attached:[500093d23000a17d:00  V i(SMP) t(SSP)]  12 Gbps
>    <- This expander's SEP
> 
> -Alan