7.0-Release and 3ware 9550SXU w/BBU - horrible write performance

[Erik Stian Tefre]

Ivan Voras wrote:
> Erik Stian Tefre wrote:
> 
>> Results from bonnie++ with default settings on my box (7.0-RELEASE,
>> amd64, 480 MB system memory, BBU, write cache enabled, 7 drive raid-5,
>> storsave = balance, NCQ on):
>>
>> Version 1.93d       ------Sequential Output------ --Sequential Input-
>> --Random-
>> Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block--
>> --Seeks--
>> Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP
>> /sec %CP
>> xxxxxxxxxxxxxxxx 1G   275  83 72440  23 24990   6   644  83 76548  15
>> 460.6   6
> 
> Was the array busy when you did the test? 72-76 MB MB/s is very slow for
> a 7-drive array in RAID-5. FreeBSD's disk IO can be sluggish but I doubt
> it would get that slow. On a RAID10 array with 6 drives I can get ~~ 200
> MB/s in both directions (though on a different controller). RAID5 is
> basically striped so your setup should on average yield close to 6 times
> the single drive performance, something in the area of 350 MB/s.

It was completely idle. Changing vfs.read_max to 80 triples the 
sequential read performance, see bonnie++ output below (run on the same 
box, nothing changed except vfs.read_max). I tried setting it to 256 as 
recommended by 3ware, but 80 seems to be the sweet spot.
3ware performance tips for FreeBSD: 
http://www.3ware.com/KB/article.aspx?id=14852

I have measured similar write performance on 2 other servers with the 
same controller model and the same number of disks.

By the way, the newer 9650SE controllers seem to be a lot faster, I 
remember getting around 350 MB/s of sequential reads/writes from an idle 
16 drive raid-6 array. (Sorry, I have no idle 9650SE to run bonnie++ on 
at the moment.)

bonnie++ with vfs_max=80:
Version 1.93d       ------Sequential Output------ --Sequential Input- 
--Random-
Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- 
--Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP 
/sec %CP
xxxxxxxxxxxxxxxx 1G   285  83 88193  28 26858   6   690  89 218642  59 
406.5   5
Latency             44353us     557ms     542ms   90795us     209ms 
361ms
Version 1.93d       ------Sequential Create------ --------Random 
Create--------
xxxxxxxxxxxxxxxxxxx -Create-- --Read--- -Delete-- -Create-- --Read--- 
-Delete--
               files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP 
/sec %CP
                  16  8204  19 +++++ +++ 22163  36  9925  25 +++++ +++ 
+++++ +++
Latency               585ms     571us    3844us     257ms   24015us 
768us

--
Erik