Re: can sftp be made multi-threaded?

Put the split files on tmpfs for full effect

On Fri, Aug 11, 2023 at 12:51 PM Joe Schaefer <joesuf4@gmail.com> wrote:

> % split -n $(nproc) foo.pdf; ls x* | xargs -P 0 -J % scp %
>  user@bar.example.com:%; rm x*; ssh user@bar.example.com sh -c “cat x* >
> foo.pdf; rm x*”
>
> On Fri, Aug 11, 2023 at 12:27 PM Joe Schaefer <joesuf4@gmail.com> wrote:
>
>> If it’s just a single file, split it into chunks.
>>
>> On Fri, Aug 11, 2023 at 12:25 PM Joe Schaefer <joesuf4@gmail.com> wrote:
>>
>>> Why don’t you just use xargs -P until you’ve exhausted your CPU capacity?
>>>
>>> On Fri, Aug 11, 2023 at 10:10 AM void <void@f-m.fm> wrote:
>>>
>>>> On Fri, Aug 11, 2023 at 12:21:33PM +0100, Jamie Landeg-Jones wrote:
>>>>
>>>> >rsync just spawns an ssh command, so would probably behave similarly.
>>>>
>>>> I'm hoping that rsync will spawn many ssh. Need to look at max sessions
>>>> on both ends of the connection.
>>>>
>>>> Since encountering the described problem, the person at the other
>>>> end is away for the week so have not been able to test thoroughly.
>>>> What I have been able to test shows that there is spiky latency
>>>> in the connection, as well as slow speed, single-threaded.
>>>>
>>>> >Another thing, scp transfers from my test Rpi2 are much slower than
>>>> the network
>>>> >can handle due to the CPU use, which hits 100% on one cpu whilst it's
>>>> running.
>>>> >So, check that CPU isn't the bottleneck too.
>>>>
>>>> Yup. That won't be happening here. Dual xenon with 56 cores at remote
>>>> end and same (but with 32 cores) at this end
>>>>
>>>> >As for the speed, I just tested sftp to transfer a file of random
>>>> data, 2 GB in
>>>> >size from one FreeBSD box in London to another in France:
>>>> >
>>>> >The final result was:
>>>> >
>>>> > 100% 2000MB  43.5MB/s   00:46  (Note, that's MegaBYTES/s)
>>>>
>>>> I ran a similar test.
>>>> Sending system is on synchronous gigabit fibre on US east coast,
>>>> receiving system is near London on 110/21 fibre (so, gigabit in the
>>>> sending
>>>> direction):
>>>>
>>>> 100% 2000MB   7.2MB/s   04:36
>>>>
>>>> using rsync -azP : 2,097,152,000 100%    6.81MB/s    0:04:53 (xfr#1,
>>>> to-chk=0/1)
>>>>
>>>> the speed fluctulates a lot. Both systems are quiet in a network and OS
>>>> sense
>>>> for the duration of the test.
>>>>
>>>> >The London box is pretty old, and is a virtual host scheduled to be
>>>> decomissioned.
>>>> >It is running an old openssl 1.X, openssh 8.8 and is a single core
>>>> 2.4Ghz amd64 box.
>>>> >
>>>> >The France box is a 4 core bare metal 3.1Ghz and64 running openssh 9.2
>>>> and openssl 1.1.1
>>>>
>>>> both ends here are running very recent -current, so ssl/ssh is
>>>> OpenSSH_9.3p1, OpenSSL 3.0.9 30 May 2023
>>>>
>>>> >Anything more I can tell you that may help?
>>>>
>>>> Thanks very much for your input. I'm certain it's not a freebsd problem.
>>>>
>>>> --
>>>>
>>>>