Re: can sftp be made multi-threaded?

% split -n $(nproc) foo.pdf; ls x* | xargs -P 0 -J % scp %
 user@bar.example.com:%; rm x*; ssh user@bar.example.com sh -c “cat x* >
foo.pdf; rm x*”

On Fri, Aug 11, 2023 at 12:27 PM Joe Schaefer <joesuf4@gmail.com> wrote:

> If it’s just a single file, split it into chunks.
>
> On Fri, Aug 11, 2023 at 12:25 PM Joe Schaefer <joesuf4@gmail.com> wrote:
>
>> Why don’t you just use xargs -P until you’ve exhausted your CPU capacity?
>>
>> On Fri, Aug 11, 2023 at 10:10 AM void <void@f-m.fm> wrote:
>>
>>> On Fri, Aug 11, 2023 at 12:21:33PM +0100, Jamie Landeg-Jones wrote:
>>>
>>> >rsync just spawns an ssh command, so would probably behave similarly.
>>>
>>> I'm hoping that rsync will spawn many ssh. Need to look at max sessions
>>> on both ends of the connection.
>>>
>>> Since encountering the described problem, the person at the other
>>> end is away for the week so have not been able to test thoroughly.
>>> What I have been able to test shows that there is spiky latency
>>> in the connection, as well as slow speed, single-threaded.
>>>
>>> >Another thing, scp transfers from my test Rpi2 are much slower than the
>>> network
>>> >can handle due to the CPU use, which hits 100% on one cpu whilst it's
>>> running.
>>> >So, check that CPU isn't the bottleneck too.
>>>
>>> Yup. That won't be happening here. Dual xenon with 56 cores at remote
>>> end and same (but with 32 cores) at this end
>>>
>>> >As for the speed, I just tested sftp to transfer a file of random data,
>>> 2 GB in
>>> >size from one FreeBSD box in London to another in France:
>>> >
>>> >The final result was:
>>> >
>>> > 100% 2000MB  43.5MB/s   00:46  (Note, that's MegaBYTES/s)
>>>
>>> I ran a similar test.
>>> Sending system is on synchronous gigabit fibre on US east coast,
>>> receiving system is near London on 110/21 fibre (so, gigabit in the
>>> sending
>>> direction):
>>>
>>> 100% 2000MB   7.2MB/s   04:36
>>>
>>> using rsync -azP : 2,097,152,000 100%    6.81MB/s    0:04:53 (xfr#1,
>>> to-chk=0/1)
>>>
>>> the speed fluctulates a lot. Both systems are quiet in a network and OS
>>> sense
>>> for the duration of the test.
>>>
>>> >The London box is pretty old, and is a virtual host scheduled to be
>>> decomissioned.
>>> >It is running an old openssl 1.X, openssh 8.8 and is a single core
>>> 2.4Ghz amd64 box.
>>> >
>>> >The France box is a 4 core bare metal 3.1Ghz and64 running openssh 9.2
>>> and openssl 1.1.1
>>>
>>> both ends here are running very recent -current, so ssl/ssh is
>>> OpenSSH_9.3p1, OpenSSL 3.0.9 30 May 2023
>>>
>>> >Anything more I can tell you that may help?
>>>
>>> Thanks very much for your input. I'm certain it's not a freebsd problem.
>>>
>>> --
>>>
>>>