zfs performance degradation

[Paul Kraus]

On Sep 24, 2015, at 11:11, Dmitrijs <war at dim.lv> wrote:

> Nope, no compression, no deduplication, only pure zfs. Even no prefetch, as it is not recommended for machines 4Gb RAM and below.

I am very surprised that ZFS is CPU limited on that system. My N54L has less CPU performance than that and I easily get 60 MB/sec via CIFS (Samba) from a Mac or Windows client.

> I've tested performance with 40Gb file on 4Gb ram machine, so cache should not count so much.

Yup. BTW, b == bits B == bytes (and to be pedantic, GB == one billion bytes while GiB == 2^30 bytes)

> I really hoped that I could get from 2HDD MIRROR at least 1.5x read performance of a single HDD, but it's more tricky as you explained.

Yup, remember that ZFS has _lots_ of metadata per fs block and it needs to read all of that as well as the data.

> Now I'm not sure what configuration will make better performance for 4 HDD - raid10 or raid-z2? Or two separate mirrors? Need directions for scale things up in the future.

Of all the questions you have asked that one is the easiest to answer … a zpool which has 2 vdevs each of which is a 2-way mirror will have roughly double the performance of a zpool that has one vdev that is a 4 drive RAIDz2. Performance scales with the number of vdevs, not the number of drives. I know that is not obvious at first, but when you look at the design of ZFS (all top level vdevs are striped across) it makes perfect sense.

So a 2 x 2-way mirror will be faster than a 4 drive RAIDz2. At a cost, the MTTDL (Mean Time To Data Loss) will be better for the RAIDz2 than the 2 x 2-way mirror. See Richard Ellings post here http://blog.richardelling.com/2010/02/zfs-data-protection-comparison.html for a comparison of relative MTTDL for ZFS configurations.

Note that I use 3-way mirrors where I need _both_ performance and reliability and RAIDz2 where I need mostly reliability and performance is secondary. But … back when I was managing lots of data (2007 - 2012), I did use RADIz2 in production for critical data, but we had 22 top level vdevs, each a 5 drive RAIDz2 and 10 hot spares. Striping data across 22 RAIDz2 gave us the performance we needed with the reliability.

> I've got http://ark.intel.com/products/78867/Intel-Celeron-Processor-J1900-2M-Cache-up-to-2_42-GHz
> And 4Gb RAM.

That should beat the MicroServer in terms of CPU. I had 8 GB in my N36L and upgraded to 16 GB in my later N54L. I keep referencing these as they are more similar to your setup than my bigger SuperMicro servers. Also note that these MicroServers were the first generation, not the later Gen 8. I paid about US$200 to US$300 for the bare chassis with CPU and 2 GB RAM. The Gen 8 are much more expensive and I am looking for something cheaper for my clients now.

> Thought it would be sufficient, but now I'm in doubt.

I think that 4 GB is slightly low for a file server, but it should not be too bad. The CPU should be fine. What are the drives themselves ? [Because with only 4 GB RAM you _will_ feel the effect of drive performance, and it is random I/Ops that really matter for ZFS]

> I can live with reduced performance for my 1st NAS, but would be nice to have clear performance requirements in mind for planing future storage boxes.
> 
> I see QNAPs and Synology NAS, they use like 1Ghz CPU and 1Gb of RAM for 4 HDD, so either I'm doing it wrong, either those NASes don't have performance (or safety?) at all.

Do they calculate checksums for end-to-end data integrity ?
What is their performance like ?

The data integrity and reliability features of ZFS do come at a cost.

> HP Proliant MicroServer is nice, but i've made my diskless system 2-3 times cheaper (200euro vs 530/650euro), so I need a reason or recomendation to spend x2x3 money on the thing, which specification looks the same.

As I said above, I have moved away from the MicroServer Gen 8 due to cost. The original MicroServer was a very sweet deal and I wish I had bought a couple of them last December when I could have gotten them for US$200 each :-( 

--
Paul Kraus
paul at kraus-haus.org