Missing drive paths on SAS3008 with expanders

[Richard Kunert]

Greetings!

We have a number of SuperMicro 60 bay JBODs with SAS3 dual expander backplanes. They are connected to servers with dual SAS3008 (Dell) HBAs. Running FreeBSD 11.1-Release.

Initially I connected the JBODs to the servers in a way that made sense to me, with each HBA connected to one of the “primary” SAS interfaces on the backplanes plus another connection to one of the “secondary” SAS interfaces. That resulted in four paths to each drive. These are 60 drive boxes so the devices went from da1 to da240 (a Dell PERC H330 takes up da0).

Then I saw that SuperMicro recommended a different cabling scheme, with both primary SAS interfaces connected to one HBA and both secondary interfaces to the other HBA. These are new servers so I tried this and only got two paths per drive for a total of 120.

The problem is when I change the cabling back I don’t see all the drive paths. I see 180 out of the 240 possible, and "camcontrol devlist" only shows 6 of the 8 expander channels. There should be four front and four rear channels:

camcontrol devlist | grep LSI

<LSI-F SAS3x48Front 0c04>          at scbus2 target 48 lun 0 (ses0,pass31)
<LSI-F SAS3x48Front 0c04>          at scbus2 target 146 lun 0 (ses1,pass62)
<LSI-R SAS3x48Rear 0c04>           at scbus2 target 248 lun 0 (ses2,pass93)
<LSI-F SAS3x48Front 0c04>          at scbus3 target 48 lun 0 (ses3,pass124)
<LSI-F SAS3x48Front 0c04>          at scbus3 target 146 lun 0 (ses4,pass155)
<LSI-R SAS3x48Rear 0c04>           at scbus3 target 248 lun 0 (ses5,pass186)


So is something cached here that I need to clear out? Does the mpr driver store something on disk? I have tried the obvious steps of powering off and unplugging the server and storage. We have several of these and they are acting the same way so I don’t think there is a hardware issue. But on boot the missing devices aren’t discovered at all. SuperMicro knows nothing of FreeBSD, their first question was what hardware RAID card we’re using.

Thanks much!

Richard