FreeBSD restartable send/receive over WAN

[Shiva Bhanujan]

thanks for all the pointers for the compression algorithms.  I ran a few tests to compare compression overhead.  These are local zfs send/receive, and no network is involved.

time zfs send -v <src-snapshot> | <compress> | <uncompress> | zfs receive -s <dst-snapshot>

Here are the performance results that I got.

no compression:
real    0m20.892s
user    0m0.000s
sys     0m5.587s

xz -0:
real    8m38.569s
user    10m28.551s
sys     0m6.866s

pxz -0:
real    4m38.448s
user    10m55.863s
sys     0m13.324s

gzip:
real    3m51.297s
user    4m12.035s
sys     0m4.696s

lz4:
real    0m29.912s
user    0m16.543s
sys     0m10.810s


lz4 has the least overhead in terms of time.   pxz/xz seem to be prohibitive give the above results.  Unless, there is something basic I'm missing?

I was really hoping that compressed sends would be available, as that would actively eliminate this overhead, given that we use lz4 as the compression algorithm when writing to disks.



________________________________
From: Karli Sjöberg [karli.sjoberg at slu.se]
Sent: Sunday, February 26, 2017 8:41 AM
To: Gary Palmer
Cc: Shiva Bhanujan; Jeremy Faulkner; freebsd-fs at freebsd.org
Subject: Re: FreeBSD restartable send/receive over WAN


Den 26 feb. 2017 4:16 em skrev Gary Palmer <gpalmer at freebsd.org>:
>
> On Sun, Feb 26, 2017 at 02:08:59PM +0000, Shiva Bhanujan wrote:
> > The compression that we use on our ZFS filesystems is lz4.  So, if I have to pipe it through a compression algorithm, that'd be uncompressing and compressing it 4 times.
> >
> > disk (lz4) -> zfs send (uncompress) -> compress (gzip) -> (network) -> uncompress (gzip) -> zfs recv (compress) -> disk (lz4)
> >
> > isn't this quite expensive?  We have to transfer multi terabyte files on a WAN link.  I'm also of the understanding that gzip by itself is single-threaded, so that'd peg one of the CPUs to 100%.  there might be other compression algorithms that can be used, but sending the ZFS as it is compressed on the filesystem is something that would be optimal, and would reduce the overhead of the additional [de]compressions that are taking place?
>
> Without going into the efficiency part of your message:
>
> archivers/pigz: Parallel GZIP
> archivers/pbzip2: Parallel BZIP2
> archivers/pixz: Parallel, indexing version of XZ
> archivers/pxz: Parallel LZMA compressor using liblzma

Also worth mentioning is, obviously:
archivers/lz4

:)

/K

>
> Regards,
>
> Gary
>
> >
> > ________________________________________
> > From: owner-freebsd-fs at freebsd.org [owner-freebsd-fs at freebsd.org] on behalf of Jeremy Faulkner [gldisater at gmail.com]
> > Sent: Saturday, February 25, 2017 4:03 PM
> > To: freebsd-fs at freebsd.org
> > Subject: Re: FreeBSD restartable send/receive over WAN
> >
> > Pipe it through a compressor
> >
> > On 2017-02-25 2:09 PM, Shiva Bhanujan wrote:
> > > Hi,
> > >
> > > I just tried restartable send/receive in 10.3 and it works like a charm.  I was wondering if compressed send has made its way into FreeBSD?  I checked 10.3 and 11.0-RELEASE, and I don't see the -c/--compressed option.  Any pointers?
> > >
> > > Regards,
> > > Shiva
> > >
> > >
> > > ________________________________________
> > > From: owner-freebsd-fs at freebsd.org [owner-freebsd-fs at freebsd.org] on behalf of Adam Nowacki [nowakpl at platinum.linux.pl]
> > > Sent: Thursday, February 16, 2017 10:41 AM
> > > To: freebsd-fs at freebsd.org
> > > Subject: Re: FreeBSD restartable send/receive over WAN
> > >
> > > On 2017-02-16 19:22, Shiva Bhanujan wrote:
> > >> Hello,
> > >>
> > >> I was wondering if restartable send/receive is available in FreeBSD?  We're running 10.2 and have a requirement of sending and receiving ZFS snapshots over a WAN link.  The snapshots could be more than a few terabytes.
> > >>
> > >> Can somebody please give me pointers, and if this feature is or isn't available in FreeBSD?
> > >
> > > FreeBSD 10.3 and later.
> > >
> > > _______________________________________________
> > > freebsd-fs at freebsd.org mailing list
> > > https://lists.freebsd.org/mailman/listinfo/freebsd-fs
> > > To unsubscribe, send any mail to "freebsd-fs-unsubscribe at freebsd.org"
> > > _______________________________________________
> > > freebsd-fs at freebsd.org mailing list
> > > https://lists.freebsd.org/mailman/listinfo/freebsd-fs
> > > To unsubscribe, send any mail to "freebsd-fs-unsubscribe at freebsd.org"
> > >
> > _______________________________________________
> > freebsd-fs at freebsd.org mailing list
> > https://lists.freebsd.org/mailman/listinfo/freebsd-fs
> > To unsubscribe, send any mail to "freebsd-fs-unsubscribe at freebsd.org"
> > _______________________________________________
> > freebsd-fs at freebsd.org mailing list
> > https://lists.freebsd.org/mailman/listinfo/freebsd-fs
> > To unsubscribe, send any mail to "freebsd-fs-unsubscribe at freebsd.org"
> >
> _______________________________________________
> freebsd-fs at freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-fs
> To unsubscribe, send any mail to "freebsd-fs-unsubscribe at freebsd.org"