speeding up zfs send | recv

[mike tancsa]

For offsite storage, I have been doing a zfs send across a 10G link and
noticed something I don't understand with respect to speed.  I have a
number of datasets I am sending, one which contains a great many
Maildirs / files and others that a few large 30-60G files (vm disk
images).  When I am sending the data set with the mail spool (many small
files and directories), transfer tends to be markedly slower.  Looking
at the cacti graph, it seems to hover around 500Mb/s through an
aes128-gcm cipher when sending the mail spool, vs sending the dataset
that has the VMs on it, around 2.5Gb/s (both on a 5min average)...

Why would the mail spool send be so slow compared to the sends where
datasets only have a few large files ?

One thing I am wondering is, could it be due to the amount of snapshots
I have ? For each, I have about 60-100 snapshots. I am only sending a
copy based on the latest snapshot, but I guess that's a lot of
calculations to go through in order to get a complete image. However, I
would have thought that would impact both types of datasets equally ?
e.g. on my oldest mailspool snapshot, I see 60G of difference from the
oldest snapshot on a dataset that's about 600GB in size

By contrast, the dataset with VM images, is 300G and the oldest snapshot
shows just 16G of difference and has a total of 93 snapshots.

Is there anything I can do to speed up the send ? The recv side has lots
of spare CPU. I dont see the disk blocking at all.  The sending side is
pretty busy, but I would imagine equally busy across all data sets.

sender is a recent RELENG_12, recv side is RELENG_13

as a side note, is zstd ever nice!  On a different dataset that has a
lot of big ass json files. I am seeing refcompressratio  22.15x   vs
13.19x  for the old lz4.

    ---Mike