Very poor performance from Dell/LSI Logic SAS 3000 series SATA/SAS RAID controller FreeBSD 6.3

[Scott Long]

Chris Dillon wrote:
> Quoting Scott Long <scottl at samsco.org>:
> 
>> For data reliability, you really don't want it enabled by default.  The
>> problem is that SATA/ATA performs so poorly without it that everyone
>> turns it on and lives with the consequences.  The tweak that I
>> recommended puts it in line with what the FreeBSD ATA driver has been
>> doing for years.
> 
> Doesn't SATA NCQ solve this particular performance vs. reliability 
> problem since it safely allows multiple outstanding write requests?  Of 
> course that means the SATA RAID controller would have to use NCQ on the 
> drives and would probably also need its own non-volatile cache.  I've 
> always assumed this is how SCSI/SAS drives (with TCQ) perform as well as 
> they do without sacrificing data integrity.

Yes and no.  NCQ gets you 90% the way there, but the lack of an ordered
tag operation in the NCQ protocol means that i/o streams can be starved,
forcing you to do unpleasant i/o scheduling hacks.  But yes, it helps
quite a bit, and I have a prototype driver already working that supports
NCQ and performs very well with write cache turned off.

> 
> We recently bought a new HP DL380G5 server with a P800 SAS RAID 
> controller, MSA60 external drive shelf with 12 750GB SATA drives, 
> 11-drive RAID5 array w/ hot-spare (a few too many drives in a single 
> RAID5 array, I know, but I'm experimenting).  The system is running 
> Windows Server 2K3 R2.  Without telling the P800 to enable the SATA WC 
> (it has an option to do so, off by default), when doing a drag and drop 
> file copy of several very large files from the internal SAS array to the 
> external SATA array it writes 300MB/sec.  I briefly enabled the 
> "Physical Drive Write Cache" on the controller just a few minutes ago 
> and ran another test and didn't notice any difference in write speed.  I 
> can only assume from this that the P800 is using NCQ on the SATA drives.
> 

The cache and queueing mechanism on most IOP raid cards will smooth over 
the performance problems with ATA/SATA, so your results aren't too 
surprising.

Scott