More Controllers != Higher Through Put

[Ivan Voras]

On 07/07/10 01:35, Stephen Sanders wrote:
> I'm wondering if anyone has heard of this.
> 
> I've a system with a 3ware 9650 servicing 4 7200RPM Segate 1TB drives
> and the motherboard servicing 2 7200 RPM Segate 1TB drives.

So far so good.

> The 4 disk array is RAID 6 while the 2 disk array is RAID 1.  The drives
> should deliver about 100MB/s.

Ok, so you've reduced the 4-drive array's write performance nearly to
equivalent of 2 drives and the 2-drive array to 1 drive. It should be
even worse for random IOs.

Since FreeBSD doesn't support RAID-6 I guess you are using hardware RAID?

For the 2-disk RAID-1 : you are probably using software RAID, right?
(on-board "SATA RAID" controllers usually are just software
implementations).

> 1.  The most the 4 disk array is developing is 250MB/s write performance

This is too much. It almost looks like something is caching what
shouldn't be cached. How did you get this result?

I'd expect less than 200 MB/s sequential writes on a 4-drive RAID-6 with
100 MB/s drives.

> while the 2 disk array is coming in at 90MB/s write performance.  

This is as expected - write performance of any size RAID-1 is equivalent
of 1 drive or less.

> The 4 disk array seems slow.

Nope - the contrary should be true. It looks like you are doing
something you shouldn't if you get that much performance, or your test
is overly simplistic (e.g. you're testing cache).

> 2. Attempting to write to both arrays simultaneously causes the rate on
> the 4 disk array to drop to 150MB/s and the 2 disk array drops to 60MB/s

Are you running on an Atom CPU? What kind of system are you using?

> I'd expect the 4 disk array should look more like 300+MB/s while the 2
> disk array is about right.

No, you cannot get 300 MB/s from simple RAID6 of 4 drives in any direction.

Think about it: all the data needs to be a) written as-is to 2 of the
drives, then b) parity/ECCs calculated and c) the same amount of data
written to 2 more drives. You cannot get write performance of more than
2 drives equivalent in this scheme, and will probably be worse. For
reads, only if your RAID controller is very, very smart (meaning: it
probably isn't), you can recover some performance by using this
parity/ECC data to reconstruct more data than is read from the two
"plain" drives. I think ZFS does this in a limited way.

> I don't get why there should be a 'coupling' between the rates on
> separate controllers. 

This is the only thing which is puzzling a bit. I

> The system is running FreeBSD 8.0, has 16GB of RAM in the system, and
> the test program is using O_DIRECT for writes in order to avoid the page
> daemon.

You should use some benchmark which knows how to deal with OS cache, for
example bonnie++ from the ports.

Use a benchmark with random IO to see just how horrible your RAID-6
performance will be for random writes.