camcontrol sata affiliations

[Alan Somers]

On Tue, Feb 16, 2016 at 4:45 AM, David Ford <david.ford at ouce.ox.ac.uk> wrote:
> Hello,
>
> I have a number of dual homed SAS disk chasses, with a mixture of SAS and SATA drives. As expected, the SAS drives appear to both hosts, and the SATA drives appear on a single host, which gets the SAS affiliation.
>
> From the host with the SATA drive visible:
>
> [root at backup-san1 ~]# camcontrol smpphylist /dev/ses0
> 26 PHYs:
> PHY  Attached SAS Address
>   0  0x0000000000000000
>   1  0x0000000000000000
>   2  0x50080e53c2b8f002   <ATA ST1000DM003-1ER1 CC45>       (da33,pass36)
>   3  0x5000cca01ab1a139   <IBM-ESXS HUS723030ALS64 J210>    (pass0,da0)
>   4  0x0000000000000000
>   5  0x0000000000000000
>   6  0x0000000000000000
>   7  0x5000c50041affc01   <IBM-ESXS ST33000650SS BC36>      (pass2,da2)
>   8  0x0000000000000000
>   9  0x0000000000000000
>  10  0x5000cca03ea41585   <IBM-ESXS HUS723030ALS64 J3K7>    (pass1,da1)
>  11  0x0000000000000000
>  12  0x500605b004f24f20
>  13  0x500605b004f24f20
>  14  0x500605b004f24f20
>  15  0x500605b004f24f20
>  16  0x0000000000000000
>  17  0x0000000000000000
>  18  0x0000000000000000
>  19  0x0000000000000000
>  20  0x0000000000000000
>  21  0x0000000000000000
>  22  0x0000000000000000
>  23  0x0000000000000000
>  24  0x50080e53c2b8f03d
>  25  0x000000000000003e
>
> From the other host:
>
> root at backup-san-02:~ # camcontrol smpphylist /dev/ses0
> 26 PHYs:
> PHY  Attached SAS Address
>   0  0x0000000000000000
>   1  0x0000000000000000
>   2  0x0000000000000000
>   3  0x5000cca01ab1a13a   <IBM-ESXS HUS723030ALS64 J210>    (pass2,da1)
>   4  0x0000000000000000
>   5  0x0000000000000000
>   6  0x0000000000000000
>   7  0x5000c50041affc02   <IBM-ESXS ST33000650SS BC36>      (pass1,da0)
>   8  0x0000000000000000
>   9  0x0000000000000000
>  10  0x5000cca03ea41586   <IBM-ESXS HUS723030ALS64 J3K7>    (pass3,da2)
>  11  0x0000000000000000
>  12  0x500605b004f27920
>  13  0x500605b004f27920
>  14  0x500605b004f27920
>  15  0x500605b004f27920
>  16  0x0000000000000000
>  17  0x0000000000000000
>  18  0x0000000000000000
>  19  0x0000000000000000
>  20  0x0000000000000000
>  21  0x0000000000000000
>  22  0x0000000000000000
>  23  0x0000000000000000
>  24  0x50080e53c1e1803d
>  25  0x000000000000003e
>
>
> I can successfully clear the affiliation:
>
> [root at backup-san1 ~]# camcontrol  smppc /dev/ses0 -p 2 -o clearaffiliation
> [root at backup-san1 ~]# smp_rep_phy_sata --phy=2 /dev/ses0
> Report phy SATA response:
>   expander change count: 74
>   phy identifier: 2
>   STP I_T nexus loss occurred: 0
>   affiliations supported: 1
>   affiliation valid: 0
>   STP SAS address: 0x50080e53c2b8f002
>   register device to host FIS:
>     34 00 50 01 01 00 00 00 00 00 00 00 01 00 00 00 00 00 00 00
>   affiliated STP initiator SAS address: 0x0
>   STP I_T nexus loss SAS address: 0x0
>   affiliation context: 0
>   current affiliation contexts: 0
>   maximum affiliation contexts: 1
>
> However from the other host:
>
> root at backup-san-02:~ # camcontrol smppc /dev/ses0 -p 2 -o sataportsel
>
> appears to do nothing - the output of camcontrol smpphylist /dev/ses0 and it does not appear on a rescan, or if I attempt to hard reset it.
>
> root at backup-san-02:~ # smp_rep_phy_sata --phy=2 /dev/ses0
> Report phy SATA result: Phy does not support SATA
>
> The systems are running Freebsd 10.2, and I have tested with both the mps and the mpr driver on different systems, the behaviour is identical.
>
> Either I'm missing a crucial step in this process, or it's a bug. Does anyone have any suggestions.
>
> Thanks
>
> David
>

You aren't missing anything.  This is just a difference between SATA
and SAS.  SAS drives have two ports, and SATA drives have only one.
Most (all?) multipath JBODs like yours have two separate expander
chips.  They connect every slot's first port to the first expander and
every slot's second port to the second expander.  That results in a
chassis with no SPOF.  With such hardware, there's no way to connect a
SATA drive to both servers.  And with more complicated hardware that
uses a single expander chip combined with SAS zoning to connect a SATA
drive to two servers, you're stuck with a SPOF.

-Alan