Re: speeding up zfs send | recv (update)

From: Miroslav Lachman <000.fbsd_at_quip.cz>
Date: Wed, 22 Feb 2023 21:03:44 UTC 
On 22/02/2023 21:13, mike tancsa wrote:

> aes128-gcm@openssh.com is what I settled on for the cipher.   If you 
> just blast dd if=/dev/zero | ssh are you able to achieve close to 
> wirespeed ?  As (I think) I mentioned in this thread, different zfs 
> datasets transmit at different speeds.  Ones with tens of thousands of 
> small files are much slower than those with a few multi gigabit files. 
> The disk seems to be the limiting factor for me as graphing "time spent 
> in IO" via telegraf, shows the disks pretty well 100% busy.

Yes, I read about it. Which is also interesting as I thought when zfs 
sends the whole filesystem it will not walk file by file as will rsync do.

> dd if=/dev/zero count=20000 bs=1m status=progress | ssh -c 
> aes128-gcm@openssh.com mdtancsa@coldstore "cat - > /dev/null"
>    20650655744 bytes (21 GB, 19 GiB) transferred 18.001s, 1147 MB/s

>   dd if=/dev/zero count=20000 bs=1m status=progress | ssh -c 
> chacha20-poly1305@openssh.com mdtancsa@coldstore "cat - > /dev/null"
>    20481835008 bytes (20 GB, 19 GiB) transferred 43.000s, 476 MB/s

> dd if=/dev/zero count=20000 bs=1m status=progress | ssh -c aes128-ctr 
> mdtancsa@coldstore "cat - > /dev/null"
>    20781727744 bytes (21 GB, 19 GiB) transferred 29.001s, 717 MB/s
> 

Interresting numbers. I think I am the only one who get best speed with 
chacha20-poly1305@openssh.com

# dd if=/dev/zero count=2000 bs=1m status=progress | ssh -c 
aes128-gcm@openssh.com quip@aa.bb.cc.dd "cat - > /dev/null"
   2074083328 bytes (2074 MB, 1978 MiB) transferred 50.003s, 41 MB/s


# dd if=/dev/zero count=2000 bs=1m status=progress | ssh -c 
chacha20-poly1305@openssh.com quip@aa.bb.cc.dd "cat - > /dev/null"
   2069889024 bytes (2070 MB, 1974 MiB) transferred 33.006s, 63 MB/s


# dd if=/dev/zero count=2000 bs=1m status=progress | ssh -c aes128-ctr 
quip@aa.bb.cc.dd "cat - > /dev/null"
   2091909120 bytes (2092 MB, 1995 MiB) transferred 45.007s, 46 MB/s


It seems the speed of SSH is limited by single core performance which is 
very poor on this machine (Intel(R) Pentium(R) Dual  CPU  E2160). Even 
if CPU has 50% idle, ssh runs on 99.8% of single core.

I know there were some HPN patches to ssh, beside that is there any 
option I can try to use less CPU?

I will play with cpuset to pin ssh on one core and everything else on 
the other core.


Kind regards
Miroslav Lachman