mfi driver performance

[matt]

On 09/13/12 13:13, Garrett Cooper wrote:
> On Thu, Sep 13, 2012 at 12:54 PM, matt <sendtomatt at gmail.com> wrote:
>> On 09/10/12 19:31, Garrett Cooper wrote:
> ...
>
>> It seems hw.mfi.max_cmds is read only. The performance is pretty close to
>> expected with no nvram or bbu on this card and commodity disks from 1.5
>> years ago, as far as I'm concerned. I'd love better write performance, but
>> it's probably being held back by the single platter in the mirror when it is
>> writing far from its edge.
> Try loader.conf:
>
> $ grep -r hw.mfi.max_cmds /sys/dev/mfi/
> /sys/dev/mfi/mfi.c:TUNABLE_INT("hw.mfi.max_cmds", &mfi_max_cmds);
>
> Cheers,
> -Garrett
Here are the results for differing values of max_cmds with same test 
conditions as against mps

Original mfi performance (max_cmds=128)

Version  1.96       ------Sequential Output------ --Sequential Input- 
--Random-
Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- 
--Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  
/sec %CP
flatline.local  32G   125  99 71443  24 53177  21   317  99 220280 33 
255.3  52
Latency               533ms     566ms    1134ms   86565us 357ms     252ms
Version  1.96       ------Sequential Create------ --------Random 
Create--------
flatline.local      -Create-- --Read--- -Delete-- -Create-- --Read--- 
-Delete--
               files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  
/sec %CP
                  16 22347  94 12389  30 16804 100 18729  99 27798 99  
5317  99
Latency             33818us     233ms     558us   26581us 75us   12319us
1.96,1.96,flatline.local,1,1347329123,32G,,125,99,71443,24,53177,21,317,99,220280,33,255.3,52,16,,,,,22347,94,12389,30,16804,100,18729,99,27798,99,5317,99,533ms,566ms,1134ms,86565us,357ms,252ms,33818us,233ms,558us,26581us,75us,12319us

max_cmds=256

Version  1.96       ------Sequential Output------ --Sequential Input- 
--Random-
Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- 
--Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  
/sec %CP
flatline.local  32G   125  99 70856  24 53503  21   327  98 232650 33 
265.1  60
Latency               637ms     522ms    1050ms     121ms 318ms     339ms
Version  1.96       ------Sequential Create------ --------Random 
Create--------
flatline.local      -Create-- --Read--- -Delete-- -Create-- --Read--- 
-Delete--
               files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  
/sec %CP
                  16 17126  76 11865  31 17134  99 18265  99 27169 100  
5006  99
Latency               114ms     522ms     875us   24250us 87us   14324us
1.96,1.96,flatline.local,1,1347580235,32G,,125,99,70856,24,53503,21,327,98,232650,33,265.1,60,16,,,,,17126,76,11865,31,17134,99,18265,99,27169,100,5006,99,637ms,522ms,1050ms,121ms,318ms,339ms,114ms,522ms,875us,24250us,87us,14324us

max_cmds=64

Version  1.96       ------Sequential Output------ --Sequential Input- 
--Random-
Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- 
--Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  
/sec %CP
flatline.local  32G   125  99 71161  24 54035  21   288  90 229860 34 
254.2  62
Latency               310ms     378ms     809ms     567ms 308ms     447ms
Version  1.96       ------Sequential Create------ --------Random 
Create--------
flatline.local      -Create-- --Read--- -Delete-- -Create-- --Read--- 
-Delete--
               files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  
/sec %CP
                  16 22570  95 14243  35 13170  99 23503  99 +++++ +++ 
22225  99
Latency             18111us     282ms    1165us   24786us 117us      80us
1.96,1.96,flatline.local,1,1347584224,32G,,125,99,71161,24,54035,21,288,90,229860,34,254.2,62,16,,,,,22570,95,14243,35,13170,99,23503,99,+++++,+++,22225,99,310ms,378ms,809ms,567ms,308ms,447ms,18111us,282ms,1165us,24786us,117us,80us

Still digesting the differences, but 256 seems to get more random seeks 
and better sequential reads at the expense of higher latencies (some 
probably identical). I think with lots of small files like a buildworld, 
it looks like 64 would excel slightly more than 128, but the differences 
between 128 and 64 are less extreme than the difference between 128 and 
256. Interestingly, sequential read appears better at 64 and 256 than 
128, but I assume this is a testing fluke...sample set is small.

Matt