Re: can sftp be made multi-threaded?

void <void@f-m.fm> wrote:

> On Fri, Aug 11, 2023 at 12:21:33PM +0100, Jamie Landeg-Jones wrote:
>
> >rsync just spawns an ssh command, so would probably behave similarly.
>
> I'm hoping that rsync will spawn many ssh. Need to look at max sessions 
> on both ends of the connection.

No. There are various "hacks" that use xargs or "fpart" or "parallel"
to split the tree into subtrees, and run parallel, independent copies
of rsync on each tree, but they are a bit hacky, and you lose efficiency
if there are cross-tree hardlinks of changed files.

This is also intended for multiple files, so won't help unless you
split the file, and if you're going to do all that for a single file, there's
no point using rsync!!

> >The final result was:
> >
> > 100% 2000MB  43.5MB/s   00:46  (Note, that's MegaBYTES/s)
>
> I ran a similar test.
> Sending system is on synchronous gigabit fibre on US east coast,
> receiving system is near London on 110/21 fibre (so, gigabit in the sending
> direction):
>
> 100% 2000MB   7.2MB/s   04:36
>
> using rsync -azP : 2,097,152,000 100%    6.81MB/s    0:04:53 (xfr#1, to-chk=0/1)
>
> the speed fluctulates a lot. Both systems are quiet in a network and OS sense
> for the duration of the test.

I forgot something important in my original post:

I *HAVE* tweaked network settings, many moons ago, and I forgot about it.

I currently have both machines using the cubic congestion control rather
than the default newreno.

I can't remember why, or even if I tried the other methods, or even if they
existed at the time, so this may be an avenue worth persuing?

https://lafibre.info/images/doc/201207_TCP_Congestion_Control_Comparison.pdf

man mod_cc

and:

 SEE ALSO
	cc_cdg(4), cc_chd(4), cc_cubic(4), cc_dctcp(4), cc_hd(4), cc_htcp(4), cc_newreno(4), cc_vegas(4), tcp(4), mod_cc(9)

Finally, have you seen "udt"? It's a reliable-data transfer method over UDP,
designed to be much faster than TCP (though I don't know how it stacks up
against the above mentioned alternate CC methods)

/usr/ports/net/udt   and  https://udt.sourceforge.io/index.html

The problem with udt is it isn't encrypted, so you'll need to run some
encryption over that.

Even if you don't use udt, it can provide a quick test to what speeds the
mcahines could get under better circumstances - just install the port on
both ends, and run udt-sendfile on one end and udt-recvfile on the other.

> Thanks very much for your input. I'm certain it's not a freebsd problem.

No worries!  I hope it helps a bit.

Cheers, Jamie