Re: [zfs] recordsize: unexpected increase of disk usage when increasing it

Am 18.01.22 um 17:07 schrieb Florent Rivoire:
> On Tue, Jan 18, 2022 at 3:23 PM Alan Somers <asomers@freebsd.org> wrote:
>> However, I would suggest that you don't bother.  With a 128kB recsize,
>> ZFS has something like a 1000:1 ratio of data:metadata.  In other
>> words, increasing your recsize can save you at most 0.1% of disk
>> space.  Basically, it doesn't matter.  What it _does_ matter for is
>> the tradeoff between write amplification and RAM usage.  1000:1 is
>> comparable to the disk:ram of many computers.  And performance is more
>> sensitive to metadata access times than data access times.  So
>> increasing your recsize can help you keep a greater fraction of your
>> metadata in ARC.  OTOH, as you remarked increasing your recsize will
>> also increase write amplification.
> 
> In the attached zdb files (for 128K recordsize), we can see that the
> "L0 ZFS plain file" objects are using 99.89% in my test zpool. So the
> ratio in my case is exactly 1000:1 like you said.
> I had that rule-of-thumb in mind, but thanks for reminding me !
> 
> As quickly mentioned in my first email, the context is that I'm
> considering using a mirror of SSDs as "special devices" for a new
> zpool which will still be mainly made of magnetic HDDs (raidz2 of
> 5x3TB).

Why don't you use the SSDs as L2ARC and let the system place often
used data and meta-data in that cache?

I'm using 1/2 TB of a 1 TB SSD as L2ARC for a pool of comparable
size, and with the persistence offered by OpenZFS it is already
filled with relevant data when the system boots.

With zstd-2 compression I get a compression ratio of typically
2 to 3 for my data, which makes the SSD effectively hold more
than 1 TB, since the L2ARC stores compressed blocks. (BTW: The
L2ARC meta-data is kept in the ARC, therefore too big an L2ARC
can reduce the amount of RAM available for other data in the ARC
and for programs more than it helps ...)

After 1,5 years, smartcl reports 14 TB written, which is a small
fraction of the specified write resilience of my SSD (less than 3%).

In my work-loads I see an ARC efficiency of about 98,5% and a
L2ARC efficiency of 80%, resulting in 1/5 of 1.5% = 0.3% of disk
reads actually being served by the rotating disks.

Regards, STefan