Maildir on ZFS

[mike tancsa]

Havent done benchmarking to see what sort of difference tweaking
primarycache makes on our mail server. But I tried on a very busy sftp
server where some directories had 500,000+ files in them.  It didnt seem
to make a difference performance wise for me, at least when fiddling
with min cache sizes etc.  Lots of RAM and fast disks.  Our mail spool
is on the equiv of raid10 and thats "spinning rust" and it keeps up well.  

I would approach it as your hardware should be able to meet its
functional requirements in the default configuration.  Then, looking at
zfs-stats and cache statistics, see what makes a difference for
tuning.   I dont think its gonna be a game changer either way.  Not sure
about record sizes.

L2ARC doesnt seem to help for our situation (RELENG_11)


ARC Efficiency:                                 35.40b
        Cache Hit Ratio:                98.39%  34.83b
        Cache Miss Ratio:               1.61%   571.23m
        Actual Hit Ratio:               98.28%  34.79b

        Data Demand Efficiency:         98.90%  3.45b
        Data Prefetch Efficiency:       75.07%  23.39m

        CACHE HITS BY CACHE LIST:
          Most Recently Used:           4.94%   1.72b
          Most Frequently Used:         94.94%  33.07b
          Most Recently Used Ghost:     0.05%   17.29m
          Most Frequently Used Ghost:   0.16%   54.30m

        CACHE HITS BY DATA TYPE:
          Demand Data:                  9.79%   3.41b
          Prefetch Data:                0.05%   17.56m
          Demand Metadata:              87.97%  30.64b
          Prefetch Metadata:            2.19%   762.76m

        CACHE MISSES BY DATA TYPE:
          Demand Data:                  6.62%   37.83m
          Prefetch Data:                1.02%   5.83m
          Demand Metadata:              74.66%  426.48m
          Prefetch Metadata:            17.70%  101.09m

------------------------------------------------------------------------

L2 ARC Summary: (HEALTHY)
        Passed Headroom:                        29.46m
        Tried Lock Failures:                    3.05m
        IO In Progress:                         27.85k
        Low Memory Aborts:                      4
        Free on Write:                          2.41m
        Writes While Full:                      31.85k
        R/W Clashes:                            0
        Bad Checksums:                          0
        IO Errors:                              0
        SPA Mismatch:                           17.47b

L2 ARC Size: (Adaptive)                         136.03  GiB
        Header Size:                    0.11%   156.63  MiB

L2 ARC Evicts:
        Lock Retries:                           440
        Upon Reading:                           7

L2 ARC Breakdown:                               571.23m
        Hit Ratio:                      4.53%   25.88m
        Miss Ratio:                     95.47%  545.35m
        Feeds:                                  10.88m

L2 ARC Buffer:
        Bytes Scanned:                          1022.25 TiB
        Buffer Iterations:                      10.88m
        List Iterations:                        43.51m
        NULL List Iterations:                   362.24k

L2 ARC Writes:
        Writes Sent:                    100.00% 3.94m

    ---Mike

On 8/28/2020 6:03 PM, Mason Loring Bliss wrote:
> I sent this to BSD Now a month ago and haven't heard anything back, but I
> bet there are opinions to be had in here! Thanks in advance.
>
>
>
> I'm migrating a mail server to a FreeBSD jail, and I've seen conflicting
> notes about how best to tune ZFS for holding accounts with Maildirs. My
> current thinking is that I don't want to change recordsize as it's an upper
> limit, not a minimum, but that I might want to set primarycache=metadata so
> I'm caching information about Maildir contents, but not the Maildir
> contents themselves. But are these valid thoughts, and what else is there
> to think about? The problem is that I don't know what I don't know, and I'm
> not sure about what I do know, so I'd love some advise about how best to
> tune a dataset for Maildirs.
>
> Thanks!
>