Missing drive paths on SAS3008 with expanders

[Alan Somers]

On Wed, Feb 28, 2018 at 1:55 PM, Richard Kunert <rkunert at wisc.edu> wrote:

> Greetings!
>
> We have a number of SuperMicro 60 bay JBODs with SAS3 dual expander
> backplanes. They are connected to servers with dual SAS3008 (Dell) HBAs.
> Running FreeBSD 11.1-Release.
>
> Initially I connected the JBODs to the servers in a way that made sense to
> me, with each HBA connected to one of the “primary” SAS interfaces on the
> backplanes plus another connection to one of the “secondary” SAS
> interfaces. That resulted in four paths to each drive. These are 60 drive
> boxes so the devices went from da1 to da240 (a Dell PERC H330 takes up da0).
>
> Then I saw that SuperMicro recommended a different cabling scheme, with
> both primary SAS interfaces connected to one HBA and both secondary
> interfaces to the other HBA. These are new servers so I tried this and only
> got two paths per drive for a total of 120.
>
> The problem is when I change the cabling back I don’t see all the drive
> paths. I see 180 out of the 240 possible, and "camcontrol devlist" only
> shows 6 of the 8 expander channels. There should be four front and four
> rear channels:
>
> camcontrol devlist | grep LSI
>
> <LSI-F SAS3x48Front 0c04>          at scbus2 target 48 lun 0 (ses0,pass31)
> <LSI-F SAS3x48Front 0c04>          at scbus2 target 146 lun 0 (ses1,pass62)
> <LSI-R SAS3x48Rear 0c04>           at scbus2 target 248 lun 0 (ses2,pass93)
> <LSI-F SAS3x48Front 0c04>          at scbus3 target 48 lun 0 (ses3,pass124)
> <LSI-F SAS3x48Front 0c04>          at scbus3 target 146 lun 0
> (ses4,pass155)
> <LSI-R SAS3x48Rear 0c04>           at scbus3 target 248 lun 0
> (ses5,pass186)
>
>
> So is something cached here that I need to clear out? Does the mpr driver
> store something on disk? I have tried the obvious steps of powering off and
> unplugging the server and storage. We have several of these and they are
> acting the same way so I don’t think there is a hardware issue. But on boot
> the missing devices aren’t discovered at all. SuperMicro knows nothing of
> FreeBSD, their first question was what hardware RAID card we’re using.
>
> Thanks much!
>
> Richard
>
>
SAS hard drives have two ports each, so it's impossible to have four truly
independent paths to each drive.  I'm guessing that each JBOD has two
backplanes, each of which has two expanders.  On each backplane, both
expanders are probably connected to all drives, one through the primary
port and one through the secondary.  I'm also going to guess that each
expander has two external SAS ports.  That would make 8 physical SAS ports
on the back of the JBOD.  If you connect both ports from the same expander
into the same HBA, they'll form a wide port.  Logically it's a single
connection, but it uses all 8 SAS lanes.  In this case, the OS will only
see one path to each drive.  However, if you connect both ports from the
same expander into different HBAs, then you'll have two separate
connections, because there are two separate SAS initiators.  In this case,
the OS will see two paths to each drive.  Does that explain what you're
seeing?

In any case, four paths are overkill.  It will be fine to have just two
paths to each drive.  Really, a single path should be fine, because
expander failures are pretty rare.  Your server will crash more often than
an expander will fail.  But if you want multipathing, then I recommend
using just two paths per drive.  Connect all of your primary expanders to
one HBA, and all of the secondary expanders to the other.  If you're using
SSDs, then use wide ports.  But if all of your drives are spinning rust,
then wide ports won't be necessary.  They won't hurt, though.

If you try this and still don't see two paths per drive, reply with a
cabling diagram.

-Alan