LSI - MR-Fusion controller driver <mrsas> patch and man page

[Desai, Kashyap]

> -----Original Message-----
> From: Borja Marcos [mailto:borjam at sarenet.es]
> Sent: Tuesday, March 25, 2014 1:42 PM
> To: Doug Ambrisko
> Cc: Desai, Kashyap; scottl at netflix.com; Radford, Adam; Kenneth D. Merry;
> sean_bruno at yahoo.com; Mankani, Krishnaraddi; dwhite at ixsystems.com;
> Maloy, Joe; jpaetzel at freebsd.org; freebsd-scsi at freebsd.org; McConnell,
> Stephen
> Subject: Re: LSI - MR-Fusion controller driver <mrsas> patch and man page
> 
> 
> On Mar 24, 2014, at 6:45 PM, Doug Ambrisko wrote:
> 
> > On Mon, Mar 24, 2014 at 09:03:43AM +0100, Borja Marcos wrote:
> > |
> > | On Mar 21, 2014, at 5:09 PM, Doug Ambrisko wrote:
> > |
> > | > Do you have a simple test case for that?  There were issues in the
> > | > SCSI command translation which should have been fixed via
> > | > switching it to use the CAM translation code.  Can you try it via a RAID
> volume of one disk?
> > |
> > | Yes, I tried all those possibilities. "syspd" devices, passthrough
> > | (same result, corrruption) and when creating single-disk RAID0
> > | volumes it worked, no corruption.
> >
> > Okay, so you don't have and simple test case to show this.  What are
> > you currently doing?  I would think the syspd and the single disk
> > RAID0 should be very similar.  I'll take a look at that.
> 
> Yes, sorry for not being so explicit.
> 
> The server  I tried has a backplane, with 23 disks fitted. At first  I assumed it
> would work, so I just created a raidz2 vdev  and begun running benchmarks.
> That's where I noticed  that it was corrupting data badly.
> 
> The disks were set as "JBOD" with the Invader's built-in configuration tool
> and visible as mfisyspd drives.
> 
> So I reduced it to using a single disk. Choosing one of them at random, I did a
> newfs. Mounted, and executed
> bonnie++. I got a panic after  some time. To avoid the inconvenience of
> bonnie++panics, I moved to ZFS (single disk vdev)
> so that I could still detect corruption.
> 
> The tests I did were:
> 
> - Disk in JBOD/syspd mode, accessed through /dev/mfisyspd -> corruption
> - Disk in JBOD/syspd mode, passthrough enabled, accessed as /dev/daX ->
> corruption
> - 1 disk RAID0 volume, accessed as /dev/mfid0 -> NO corruption
> 
> And I understand that syspd and RAID0 should be really different. I
> understand that it's a shallow layer doing little more than passing through
> commands? I should read the driver thoroughly it seems... Pity there are no
> available documents.
> 
> > | Just in case I tried with SSDs (Samsung EVO  840 and OCZ Vertex 4)
> > | and Seagate SAS disks.
> >
> > So it seems generic.  Size of disk might be useful.
> 
> 
> The Samsung are 1 TB, the OCZ are the 512 MB ones, and the Seagates are
> 146 GB.
> The SSDs (Samsung and OCZ) are SATA disks, the Seagate is SAS. The Seagate
> was corrupted as well in the same test cases, so it doesn't seem to be linked
> to SATA-on-SAS or to TRIM. TRIM, anyway, doesn't work on "mfisyspd"
> devices, which is a problem if you want to run ZFS on SSD disks.
> 
> > | Actually I think that converting passthrough disks to "mfisyspd" or
> > | an equivalent is a bad idea, unless, of course, there's a compelling
> > | reason I don't realize. For instance, if you are using SSDs you need
> > | access to the TRIM command.
> > |
> > | So, if I can vote, I vote with arms and legs for "da" devices in
> > | case it's a passthough.
> >
> > Nothing is planned to be removed from mfi.  However, LSI would like
> > mrsas to be used on newer cards.  We've let people LSI know that
> > people use the pass through mode and having the logical volumes come
> > up as /dev/daX and pass through would be confusing.  Granted mrsas
> > doesn't support pass through so that isn't a problem.
> 
> Pass through is a critical feature when you are running ZFS. I think it's a real
> problem if the new driver (the only one that will support newer cards, I
> guess) won't support pass-through.
> 
> And of course I know that you can just buy making sure that you get HBAs
> instead of RAID cards, but  so many manufacturers bundle those RAID
> controllers in fixed  configurations that it wouldn't be practical.
> 
> The ideal solution would be to have logical volumes come up as /dev/mfid,
> /dev/mrsas, /dev/mfisyspd or whatever, and passthrough devices as real
> passthrough devices like /dev/da. That would avoid confusion.
> 
> So, please, LSI, consider it. It would really add to the versatility of your cards
> and there's nothing to lose.

Borja:

<mrsas> driver will attach Raid volume and JBOD (SysPD) to the CAM layer.  It is not good to expose hidden raid volume or what we called as pass-through device here to the OS for many reason..  Other than management things like SMART monitor, we cannot/should not do file system IO on pass-through devices.  With <mfi> it might be true that user always do file system IO on <mfiX> deivce and consider /dev/daX as pass-through device... With <mrsas> all device will be seen as <daX>. You cannot identify which will be a pass-through and which is configured device by LSI config utils.

It is not a complex code change if pass-through device is required for <mrsas>, but it is just a matter of no use and more error prone to expose devices as pass-through. 
None of the LSI driver does this including <mps> and <mrsas> in FreeBSD + <megaraid_sas> and <mpt2sas>/<mpt3sas> in Linux.

If you can express what functionality you think it is missing, if there is not pass-through device ?  Are you doing ZFS (File system IO) on Pass-through device. ? 
If yes, then why can't you create JBOD/SysPD  for that purpose?

> 
> > | It's not a pressing need, I installed a LSI2008 card and I connected
> > | the SAS backplane to it, so for now the Invader is a non-problem,
> > | but if it will help to run some tests with the mps driver I can
> > | certainly do it.
> >
> > The mps driver is a different thing, mrsas is LSI's replacement for
> > mfi with newer cards.  So if you could try syspd and single RAID 0
> > test with mrsas that would be great to see if that shows the same
> > pattern of syspd failing with mfi and passing with single RAID 0.
> > Again, the logical volumes will show up as /dev/da* with no pass
> > through devices.  If I can reproduce the problem then I can look into
> > here or others might be able to find the solution.
> 
> I'll try, thanks :)
> 
> 
> 
> 
> 
> Borja.
> 
> 
>