/dev/enc@ SES doc?

[Alan Somers]

If you have 3 shelves each with dual expanders, then you should have six
/dev/ses devices.  The fact that you see 12 means that something is cabled
incorrectly.  It sounds like each expander has two SAS ports (each of which
has 4 SAS lanes).  Are there really only two LSI 9300-16e cards?  It sounds
like there must be three.  What you _should_ do is connect both of each
expanders' ports to the same HBA.  That way it will form a wide port of 8
lanes.  If you connect a single expander to two different HBAs, then the OS
sees them as separate expanders and doesn't realize that they're really the
same.
-Alan

On Mon, Apr 23, 2018 at 7:39 PM, John <jwd at freebsd.org> wrote:

> ----- Alan Somers's Original Message -----
> > The ses(4) man page describes the ioctls supported by the /dev/sesX
> nodes.
> > The /dev/enc at ... nodes, however, are simply aliases for /dev/da* nodes
>
>    I've been poking around in the code and am probably missing
> something obvious.
>
>    I have 3 shelves, dual expanders each, 4 cables from a pair
> of LSI 9300-16e cards. I have one disk in each shelf, bay 1
> in the 1st shelf, bay 2 in the 2nd, bay 3 in the 3rd. One drive
> shows up on each cable/channel:
>
> WBN06J1R0000E8283APS - ses1-da0   ses9-da10 ses3-da4 ses7-da6
> WBN0414K0000E81627LM - ses11-da11 ses2-da2  ses5-da5 ses8-da8
> WBN05GME0000E8280DY5 - ses0-da1   ses4-da3  ses6-da7 ses10-da9
>
>    And the readout of /dev/enc@* (trailing ses annotation added by hand):
>
> # find /dev/enc@* -name 'da*'
> /dev/enc at n500304801f25c63d/type at 0/slot at 3/elmdesc at Slot02/da3 (ses4)
> /dev/enc at n500304801f25c6bd/type at 0/slot at 3/elmdesc at Slot02/da1 (ses0)
> /dev/enc at n500304801f25c73d/type at 0/slot at 1/elmdesc at Slot00/da4 (ses3)
> /dev/enc at n500304801f25c7bd/type at 0/slot at 1/elmdesc at Slot00/da0 (ses1)
> /dev/enc at n500304801f25c83d/type at 0/slot at 2/elmdesc at Slot01/da5 (ses5)
> /dev/enc at n500304801f25c8bd/type at 0/slot at 2/elmdesc at Slot01/da2 (ses2)
>
>    The 1st 6 ses units are shown - the 2nd 6 ses units are missing (ses6
> -> ses11).
>
>    I think the following represents the issue, not sure how to fix it?
>
> # for i in /dev/ses*; do
>      camcontrol smpphylist $i 2>/dev/null | grep ses
>   done | awk '{print $2;}'
> 0x500304801f25c6bd
> 0x500304801f25c7bd
> 0x500304801f25c6bd
> 0x500304801f25c8bd
> 0x500304801f25c8bd
> 0x500304801f25c73d
> 0x500304801f25c63d
> 0x500304801f25c83d
> 0x500304801f25c63d
> 0x500304801f25c73d
> 0x500304801f25c83d
> 0x500304801f25c7bd
> # for i in /dev/ses*; do
>      camcontrol smpphylist $i 2>/dev/null | grep ses
>   done | awk '{print $2;}' | sort -u
> 0x500304801f25c63d
> 0x500304801f25c6bd
> 0x500304801f25c73d
> 0x500304801f25c7bd
> 0x500304801f25c83d
> 0x500304801f25c8bd
>
>    The ses unit adds a level of uniqueness not provided by just the
> logical identifier?
>
>    Thoughts?
>
> Thanks,
> John
>
> > On Fri, Apr 13, 2018 at 9:07 AM, John <jwd at freebsd.org> wrote:
> >
> > > Hi Folks,
> > >
> > >    Just curious...
> > >
> > >    Is there any doc for the /dev/enc at .... entries that map
> > > any attached SES enabled storage?
> > >
> > >    cam/scsi/scsi_enc_ses.c does contain a good amount of info.
> > >
> > > Thanks!
> > > John
>