RAM amount recommendations for ZFS pools with ZIL and L2ARC on SSD

[Tom Evans]

On Fri, Mar 22, 2013 at 2:26 PM, Zeus Panchenko <zeus at ibs.dn.ua> wrote:
> hi all,
>
> while discovering the subj, I found recommendations at:
> http://doc.freenas.org/index.php/Hardware_Recommendations#RAM
>
> - ---------------------------------------------------------------------
> ... a general rule of thumb is 1 GB of RAM for every 1TB of storage ...
> If you plan to use ZFS deduplication, a general rule of thumb is 5 GB
> RAM per TB of storage to be deduplicated.
> - ---------------------------------------------------------------------
>
>
> so, are these recommendations correct for ZFS pools with ZIL and L2ARC
> on SSD configurations?
>
> are there corellations between "RAM amount" and "with/without ZIL, L2ARC
> on separate devices pool configuration" ?
>

Well, they wouldn't hurt. Personally, I've used much less than that -
I have a 12 x 1.5 TB server with 8 GB RAM, which is only 0.5GB/TB.
It's replacement will have significantly more, but only because RAM is
cheap(er).

Dedupe is a special beast, to get decent performance the whole dedupe
table must fit into memory, Oracle have a sizing guide:

http://www.oracle.com/technetwork/articles/servers-storage-admin/o11-113-size-zfs-dedup-1354231.html

If you don't have a ZIL, a portion of your pool is used instead as
ZIL. RAM is never used for ZIL. Remember that the ZIL is only for
synchronous writes, which most writes are not.

RAM is mainly used for the ARC cache, and RAM is a lot faster than an
SSD, so the more RAM used for the ARC cache the better. Adding an SSD
L2ARC in addition will increase the amount of things that can be kept
cached in a (relatively) slow cache. If your problem is that you need
that data in a faster cache, then you need more RAM.

I think I am right in saying that only things that have been read can
be stored in ARC, so how effective it is would depend upon your
workload - how much read things do you need in the cache?

Cheers

Tom