Re: can sftp be made multi-threaded?

If it’s just a single file, split it into chunks.

On Fri, Aug 11, 2023 at 12:25 PM Joe Schaefer <joesuf4@gmail.com> wrote:

> Why don’t you just use xargs -P until you’ve exhausted your CPU capacity?
>
> On Fri, Aug 11, 2023 at 10:10 AM void <void@f-m.fm> wrote:
>
>> On Fri, Aug 11, 2023 at 12:21:33PM +0100, Jamie Landeg-Jones wrote:
>>
>> >rsync just spawns an ssh command, so would probably behave similarly.
>>
>> I'm hoping that rsync will spawn many ssh. Need to look at max sessions
>> on both ends of the connection.
>>
>> Since encountering the described problem, the person at the other
>> end is away for the week so have not been able to test thoroughly.
>> What I have been able to test shows that there is spiky latency
>> in the connection, as well as slow speed, single-threaded.
>>
>> >Another thing, scp transfers from my test Rpi2 are much slower than the
>> network
>> >can handle due to the CPU use, which hits 100% on one cpu whilst it's
>> running.
>> >So, check that CPU isn't the bottleneck too.
>>
>> Yup. That won't be happening here. Dual xenon with 56 cores at remote
>> end and same (but with 32 cores) at this end
>>
>> >As for the speed, I just tested sftp to transfer a file of random data,
>> 2 GB in
>> >size from one FreeBSD box in London to another in France:
>> >
>> >The final result was:
>> >
>> > 100% 2000MB  43.5MB/s   00:46  (Note, that's MegaBYTES/s)
>>
>> I ran a similar test.
>> Sending system is on synchronous gigabit fibre on US east coast,
>> receiving system is near London on 110/21 fibre (so, gigabit in the
>> sending
>> direction):
>>
>> 100% 2000MB   7.2MB/s   04:36
>>
>> using rsync -azP : 2,097,152,000 100%    6.81MB/s    0:04:53 (xfr#1,
>> to-chk=0/1)
>>
>> the speed fluctulates a lot. Both systems are quiet in a network and OS
>> sense
>> for the duration of the test.
>>
>> >The London box is pretty old, and is a virtual host scheduled to be
>> decomissioned.
>> >It is running an old openssl 1.X, openssh 8.8 and is a single core
>> 2.4Ghz amd64 box.
>> >
>> >The France box is a 4 core bare metal 3.1Ghz and64 running openssh 9.2
>> and openssl 1.1.1
>>
>> both ends here are running very recent -current, so ssl/ssh is
>> OpenSSH_9.3p1, OpenSSL 3.0.9 30 May 2023
>>
>> >Anything more I can tell you that may help?
>>
>> Thanks very much for your input. I'm certain it's not a freebsd problem.
>>
>> --
>>
>>