speeding up zfs send | recv
mike tancsa
mike at sentex.net
Thu May 13 14:45:51 UTC 2021
For offsite storage, I have been doing a zfs send across a 10G link and
noticed something I don't understand with respect to speed. I have a
number of datasets I am sending, one which contains a great many
Maildirs / files and others that a few large 30-60G files (vm disk
images). When I am sending the data set with the mail spool (many small
files and directories), transfer tends to be markedly slower. Looking
at the cacti graph, it seems to hover around 500Mb/s through an
aes128-gcm cipher when sending the mail spool, vs sending the dataset
that has the VMs on it, around 2.5Gb/s (both on a 5min average)...
Why would the mail spool send be so slow compared to the sends where
datasets only have a few large files ?
One thing I am wondering is, could it be due to the amount of snapshots
I have ? For each, I have about 60-100 snapshots. I am only sending a
copy based on the latest snapshot, but I guess that's a lot of
calculations to go through in order to get a complete image. However, I
would have thought that would impact both types of datasets equally ?
e.g. on my oldest mailspool snapshot, I see 60G of difference from the
oldest snapshot on a dataset that's about 600GB in size
By contrast, the dataset with VM images, is 300G and the oldest snapshot
shows just 16G of difference and has a total of 93 snapshots.
Is there anything I can do to speed up the send ? The recv side has lots
of spare CPU. I dont see the disk blocking at all. The sending side is
pretty busy, but I would imagine equally busy across all data sets.
sender is a recent RELENG_12, recv side is RELENG_13
as a side note, is zstd ever nice! On a different dataset that has a
lot of big ass json files. I am seeing refcompressratio 22.15x vs
13.19x for the old lz4.
---Mike
More information about the freebsd-fs
mailing list