Re: can sftp be made multi-threaded?

Why don’t you just use xargs -P until you’ve exhausted your CPU capacity?

On Fri, Aug 11, 2023 at 10:10 AM void <void@f-m.fm> wrote:

> On Fri, Aug 11, 2023 at 12:21:33PM +0100, Jamie Landeg-Jones wrote:
>
> >rsync just spawns an ssh command, so would probably behave similarly.
>
> I'm hoping that rsync will spawn many ssh. Need to look at max sessions
> on both ends of the connection.
>
> Since encountering the described problem, the person at the other
> end is away for the week so have not been able to test thoroughly.
> What I have been able to test shows that there is spiky latency
> in the connection, as well as slow speed, single-threaded.
>
> >Another thing, scp transfers from my test Rpi2 are much slower than the
> network
> >can handle due to the CPU use, which hits 100% on one cpu whilst it's
> running.
> >So, check that CPU isn't the bottleneck too.
>
> Yup. That won't be happening here. Dual xenon with 56 cores at remote
> end and same (but with 32 cores) at this end
>
> >As for the speed, I just tested sftp to transfer a file of random data, 2
> GB in
> >size from one FreeBSD box in London to another in France:
> >
> >The final result was:
> >
> > 100% 2000MB  43.5MB/s   00:46  (Note, that's MegaBYTES/s)
>
> I ran a similar test.
> Sending system is on synchronous gigabit fibre on US east coast,
> receiving system is near London on 110/21 fibre (so, gigabit in the sending
> direction):
>
> 100% 2000MB   7.2MB/s   04:36
>
> using rsync -azP : 2,097,152,000 100%    6.81MB/s    0:04:53 (xfr#1,
> to-chk=0/1)
>
> the speed fluctulates a lot. Both systems are quiet in a network and OS
> sense
> for the duration of the test.
>
> >The London box is pretty old, and is a virtual host scheduled to be
> decomissioned.
> >It is running an old openssl 1.X, openssh 8.8 and is a single core 2.4Ghz
> amd64 box.
> >
> >The France box is a 4 core bare metal 3.1Ghz and64 running openssh 9.2
> and openssl 1.1.1
>
> both ends here are running very recent -current, so ssl/ssh is
> OpenSSH_9.3p1, OpenSSL 3.0.9 30 May 2023
>
> >Anything more I can tell you that may help?
>
> Thanks very much for your input. I'm certain it's not a freebsd problem.
>
> --
>
>