Inferring SAS expander topology

Alan Somers asomers at freebsd.org
Tue May 30 17:31:35 UTC 2017 

On Tue, May 30, 2017 at 11:00 AM, Dustin Wenz <dustinwenz at ebureau.com> wrote:
> I'm working on a server that has 6 LSI SAS expanders connected to it. Two of them are throwing intermittent command errors and need some attention. Because the host only has four external 8088 ports, I know some of the expanders are chained through each other.
>
> Before I can send parts for replacement, I need to determine if the two problematic expanders are either daisy-chained or directly connected to the host. On Linux, this is trivial; I would just browse the device topology in /sys/. However, this machine is running FreeBSD 10.3, and I am unable to find a way to do this. I have sg3_utils available, as well as any build-in FreeBSD tools. Using this software, is it possible to determine if a SAS expander is chained off of another?
>
>         - .Dustin

I don't think sg3_utils will help you.  You want sysutils/smp_utils
instead.  If you install that and then run "smp_discover /dev/ses0",
it will show you what each phy is connected to.  Usually, there will
be four phys connected to the upstream port.  You can tell by their
EUI64s whether they're connected directly to the HBA or to another
expander.  And you can tell which expander by comparing the exact
EUI64 to each other expander's SEP phy.  Note that with LSI expanders,
the SEP's address usually differs from the expander's address by a few
bits in the last byte.  For example, if the expander's address is
0x50000000000000ff, then the SEP's address might by
0x50000000000000ffd.  Here's some example output from one of my
systems:

# smp_discover /dev/ses3
  phy   0:U:attached:[500093d23000a000:00  t(SATA)]  6 Gbps
  phy   1:U:attached:[500093d23000a001:00  t(SATA)]  6 Gbps
  phy   2:U:attached:[500093d23000a002:00  t(SATA)]  6 Gbps
  phy   3:U:attached:[500093d23000a003:00  t(SATA)]  6 Gbps
  phy   4:U:attached:[500093d23000a17f:11 exp t(SMP)]  6 Gbps
         <- Connected to the other expander
  phy   5:U:attached:[500093d23000a17f:10 exp t(SMP)]  6 Gbps
         <- Connected to the other expander
  phy   6:U:attached:[500093d23000a17f:09 exp t(SMP)]  6 Gbps
         <- Connected to the other expander
  phy   7:U:attached:[500093d23000a17f:08 exp t(SMP)]  6 Gbps
         <- Connected to the other expander
  phy   8:U:attached:[500093d23000a1bf:11 exp t(SMP)]  6 Gbps
         <- Connected to a third expander
  phy   9:U:attached:[500093d23000a1bf:10 exp t(SMP)]  6 Gbps
         <- Connected to a third expander
  phy  10:U:attached:[500093d23000a1bf:09 exp t(SMP)]  6 Gbps
        <- Connected to a third expander
  phy  11:U:attached:[500093d23000a1bf:08 exp t(SMP)]  6 Gbps
        <- Connected to a third expander
  phy  16:U:attached:[500093d23000a010:00  t(SATA)]  6 Gbps
  phy  17:U:attached:[500093d23000a011:00  t(SATA)]  6 Gbps
  phy  18:U:attached:[500093d23000a012:00  t(SATA)]  6 Gbps
  phy  19:U:attached:[500093d23000a013:00  t(SATA)]  6 Gbps
  phy  20:U:attached:[500093d23000a014:00  t(SATA)]  6 Gbps
  phy  21:U:attached:[500093d23000a015:00  t(SATA)]  6 Gbps
  phy  22:U:attached:[500093d23000a016:00  t(SATA)]  6 Gbps
  phy  23:U:attached:[500093d23000a017:00  t(SATA)]  6 Gbps
  phy  25:U:attached:[500093d23000a019:00  t(SATA)]  6 Gbps
  phy  26:U:attached:[500093d23000a01a:00  t(SATA)]  6 Gbps
  phy  29:U:attached:[500093d23000a01d:00  t(SATA)]  6 Gbps
  phy  30:U:attached:[500093d23000a01e:00  t(SATA)]  6 Gbps
  phy  31:U:attached:[500093d23000a01f:00  t(SATA)]  6 Gbps
  phy  33:U:attached:[500093d23000a021:00  t(SATA)]  6 Gbps
  phy  34:U:attached:[500093d23000a022:00  t(SATA)]  6 Gbps
  phy  35:U:attached:[500093d23000a023:00  t(SATA)]  6 Gbps
  phy  36:U:attached:[500093d23000a024:00  t(SATA)]  6 Gbps
  phy  37:U:attached:[500093d23000a025:00  t(SATA)]  6 Gbps
  phy  38:U:attached:[500093d23000a026:00  t(SATA)]  6 Gbps
  phy  39:U:attached:[500093d23000a027:00  t(SATA)]  6 Gbps
  phy  40:U:attached:[500605b008a93990:04  i(SSP+STP+SMP)]  6 Gbps
  <- Connected to an LSI HBA
  phy  41:U:attached:[500605b008a93990:07  i(SSP+STP+SMP)]  6 Gbps
  <- Connected to an LSI HBA
  phy  42:U:attached:[500605b008a93990:05  i(SSP+STP+SMP)]  6 Gbps
  <- Connected to an LSI HBA
  phy  43:U:attached:[500605b008a93990:06  i(SSP+STP+SMP)]  6 Gbps
  <- Connected to an LSI HBA
  phy  48:D:attached:[500093d23000a03d:00  V i(SMP) t(SSP)]  12 Gbps
      <- This expander's SEP

# smp_discover /dev/pass121
  phy   0:U:attached:[500093d23000a140:00  t(SATA)]  6 Gbps
  phy   1:U:attached:[500093d23000a141:00  t(SATA)]  6 Gbps
  phy   2:U:attached:[500093d23000a142:00  t(SATA)]  6 Gbps
  phy   3:U:attached:[500093d23000a143:00  t(SATA)]  6 Gbps
  phy   4:U:attached:[500093d23000a144:00  t(SATA)]  6 Gbps
  phy   5:U:attached:[500093d23000a145:00  t(SATA)]  6 Gbps
  phy   6:U:attached:[500093d23000a146:00  t(SATA)]  6 Gbps
  phy   7:U:attached:[500093d23000a147:00  t(SATA)]  6 Gbps
  phy   8:U:attached:[500093d23000a03f:07 exp t(SMP)]  6 Gbps
         <- Connected to the other expander
  phy   9:U:attached:[500093d23000a03f:06 exp t(SMP)]  6 Gbps
         <- Connected to the other expander
  phy  10:U:attached:[500093d23000a03f:05 exp t(SMP)]  6 Gbps
        <- Connected to the other expander
  phy  11:U:attached:[500093d23000a03f:04 exp t(SMP)]  6 Gbps
        <- Connected to the other expander
  phy  12:U:attached:[500093d23000a14c:00  t(SATA)]  6 Gbps
  phy  14:U:attached:[500093d23000a14e:00  t(SATA)]  6 Gbps
  phy  15:U:attached:[500093d23000a14f:00  t(SATA)]  6 Gbps
  phy  16:U:attached:[500093d23000a150:00  t(SATA)]  6 Gbps
  phy  17:U:attached:[500093d23000a151:00  t(SATA)]  6 Gbps
  phy  18:U:attached:[500093d23000a152:00  t(SATA)]  6 Gbps
  phy  19:U:attached:[500093d23000a153:00  t(SATA)]  6 Gbps
  phy  20:U:attached:[500093d23000a154:00  t(SATA)]  6 Gbps
  phy  21:U:attached:[500093d23000a155:00  t(SATA)]  6 Gbps
  phy  22:U:attached:[500093d23000a156:00  t(SATA)]  6 Gbps
  phy  23:U:attached:[500093d23000a157:00  t(SATA)]  6 Gbps
  phy  24:U:attached:[500093d23000a158:00  t(SATA)]  6 Gbps
  phy  25:U:attached:[500093d23000a159:00  t(SATA)]  6 Gbps
  phy  26:U:attached:[500093d23000a15a:00  t(SATA)]  6 Gbps
  phy  27:U:attached:[500093d23000a15b:00  t(SATA)]  6 Gbps
  phy  28:U:attached:[500093d23000a15c:00  t(SATA)]  6 Gbps
  phy  29:U:attached:[500093d23000a15d:00  t(SATA)]  6 Gbps
  phy  30:U:attached:[500093d23000a15e:00  t(SATA)]  6 Gbps
  phy  32:U:attached:[500093d23000a160:00  t(SATA)]  6 Gbps
  phy  33:U:attached:[500093d23000a161:00  t(SATA)]  6 Gbps
  phy  34:U:attached:[500093d23000a162:00  t(SATA)]  6 Gbps
  phy  35:U:attached:[500093d23000a163:00  t(SATA)]  6 Gbps
  phy  36:U:attached:[500093d23000a164:00  t(SATA)]  6 Gbps
  phy  37:U:attached:[500093d23000a165:00  t(SATA)]  6 Gbps
  phy  38:U:attached:[500093d23000a166:00  t(SATA)]  6 Gbps
  phy  39:U:attached:[500093d23000a167:00  t(SATA)]  6 Gbps
  phy  40:D:attached:[500093d23000a17d:00  V i(SMP) t(SSP)]  12 Gbps
    <- This expander's SEP

-Alan

More information about the freebsd-scsi mailing list