Re: can sftp be made multi-threaded?

void <void@f-m.fm> wrote:

> Hello hackers@
>
> Can sftp be made multi-threaded? Would rsync-over-ssh accomplish the same thing?
>
> The issue I'm having is with large backups taking ages to transfer
> even with both source and destination being on gigabit fibre.
>
> It looks like sftp is single-threaded and the bandwidth per thread is
> something like 5-50 Mbps and it's nowhere near saturating the bandwidth. 
> Would multi-threaded sftp (if it were possible)
> or rsync-over-ssh work multi-threaded on a single huge backup file?
>
> In my example, the ssh systems are both OpenSSH_9.3p1, OpenSSL 3.0.9 30 May 2023

rsync just spawns an ssh command, so would probably behave similarly.

Another thing, scp transfers from my test Rpi2 are much slower than the network
can handle due to the CPU use, which hits 100% on one cpu whilst it's running.
So, check that CPU isn't the bottleneck too.

As for the speed, I just tested sftp to transfer a file of random data, 2 GB in
size from one FreeBSD box in London to another in France:

The final result was:

 100% 2000MB  43.5MB/s   00:46  (Note, that's MegaBYTES/s)

That is WITHOUT HPN. I have compiled ssh both sides with HPN, but in this case,
at least, enabling it makes no difference.

The London box is pretty old, and is a virtual host scheduled to be decomissioned.
It is running an old openssl 1.X, openssh 8.8 and is a single core 2.4Ghz amd64 box.

The France box is a 4 core bare metal 3.1Ghz and64 running openssh 9.2 and openssl 1.1.1

Anything more I can tell you that may help?

Cheers, Jamie