LSI SAS 3008 card - 35 out of 36 disks detected

[Jason Wolfe]

On Mon, Dec 8, 2014 at 11:20 AM, Alan Somers <asomers at freebsd.org> wrote:
> On Mon, Dec 8, 2014 at 8:58 AM, Justin O'Conor
> <oconnor at crystal.harvard.edu> wrote:
>> Hi All,
>> Thanks, this is encouraging. smp_discover ses0|1 see 36 sata disks. This is from the 10.1 install.
>>
>
> There are certainly some inconsistencies in the smp_discover
> responses.  For example, the SEP on ses0 (phy identifier: 36) has
> "connector type: SAS virtual connector" and "connector element index:
> 24"  But the SEP on ses1 (phy identifier: 28) has "connector type: No
> information" and "connector element index: 0".  Also note that "phy
> identifier: 12" on ses1 has "connector element index: 0".  That would
> be the first slot on the rear expander, if the slots and phys are
> numbered the same way.  My best guess is that phy 12 and phy 28 mapped
> to the same map_idx in mpr_mapping.c:1168.  So the information for the
> SEP overwrote the information for the first disk slot.  If my guess is
> true, then recabling your chassis as you suggested wouldn't help.
> However, you might try the attached but untested patch.  It will
> prevent the SEP from being added to the mapping table while printing a
> useful error message.  If I'm correct, then the patch will let you use
> all 36 disk slots, but you won't have ses1 anymore.
>
> In the meantime, I'll try to reproduce your problem.  I have all the
> required equipment in my lab.
>
> -Alan
>

I believe this is the same issue we ran into a few years ago on the
LSI2008, where the ses0 device would map over the boot disk.  It's
long and spans over multiple months, so just the relevant bits:

Initial report:
https://lists.freebsd.org/pipermail/freebsd-scsi/2012-February/005243.html

The issue seems to be a shortcoming in the detection method where it
has no problem assigning the ses over an already mapped disk, LSI's
initial response was to use the LSI config utility and map the drives
manually:
https://lists.freebsd.org/pipermail/freebsd-scsi/2012-February/005243.html
https://lists.freebsd.org/pipermail/freebsd-scsi/2012-February/005267.html
https://lists.freebsd.org/pipermail/freebsd-scsi/2012-March/005343.html

This was not an option for us as we had 2000 of these devices in the
field, and entering the LSI BIOS would be a large undertaking. In the
end after an internal dialogue with LSI guys, Kashyap was kind enough
to write a one off for us, that never made it upstream.  It simply
assigns the ses device to max target + 1 when a conflict is found.
The core issue seems to be with the way LSI detects and assigns
devices on FreeBSD, so this is by no means 'proper', but it's sound
enough so resolve the issue for us on the LSI2008.  In case it's
interesting to anyone:

http://nitrology.com/mps_fix-fbsd10stable.diff

Jason