Re: ftp-master rsync client issue?

Philip,

Thanks for your help. Now rsync is connecting but getting slow and
almost stopped after a few minutes:

$ /usr/local/bin/rsync -vaHz --delete --progress --stats
ftp-master.freebsd.org::FreeBSD/ <ftp-pub>/FreeBSD/
==============================================================================
This is the FreeBSD master archive for the ftp area /pub/FreeBSD
It is available for primary mirrors in order to ensure rapid archive
updates and distribution.

rsync -vaHz --delete rsync://ftp-master.freebsd.org/FreeBSD/ ~ftp/pub/FreeBSD/

Contact information etc: http://www.FreeBSD.org/doc/en/articles/hubs/

==============================================================================
********    MINIMUM RSYNC VERSION REQUIRED = 3.00 (prefer 3.1.0+)    *********
==============================================================================

receiving incremental file list
./
TIMESTAMP
             35 100%   34.18kB/s    0:00:00 (xfr#1, ir-chk=1006/1013)
dir.sizes
          3,110 100%  337.46kB/s    0:00:00 (xfr#2, ir-chk=1005/1013)
development/
development/TIMESTAMP
             35 100%    3.42kB/s    0:00:00 (xfr#3, ir-chk=1027/1041)
development/tarballs/
development/tarballs/doc_main.tar.gz
        851,895   0%  392.42kB/s    0:04:33  ^C^C

Latency is high (expected) and no obvious loss.

> ping -c 10 ftp-master.freebsd.org
PING ftp-master.freebsd.org (96.47.72.39): 56 data bytes
64 bytes from 96.47.72.39: icmp_seq=0 ttl=50 time=206.468 ms
64 bytes from 96.47.72.39: icmp_seq=1 ttl=50 time=205.864 ms
64 bytes from 96.47.72.39: icmp_seq=2 ttl=50 time=205.773 ms
64 bytes from 96.47.72.39: icmp_seq=3 ttl=50 time=205.814 ms
64 bytes from 96.47.72.39: icmp_seq=4 ttl=50 time=205.780 ms
64 bytes from 96.47.72.39: icmp_seq=5 ttl=50 time=205.958 ms
64 bytes from 96.47.72.39: icmp_seq=6 ttl=50 time=205.775 ms
64 bytes from 96.47.72.39: icmp_seq=7 ttl=50 time=205.817 ms
64 bytes from 96.47.72.39: icmp_seq=8 ttl=50 time=206.643 ms
64 bytes from 96.47.72.39: icmp_seq=9 ttl=50 time=205.772 ms

--- ftp-master.freebsd.org ping statistics ---
10 packets transmitted, 10 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 205.772/205.966/206.643/0.302 ms

I used to sync with ftp-master all the time before this connection issue.
I can sync with other mirrors (e.g. ftp.jp) so probably not a network
issue here.

Any help appreciated.

Best,

On Wed, Nov 8, 2023 at 9:49 AM Philip Paeps <philip@freebsd.org> wrote:
>
> On 2023-11-08 05:44:22 (+0800), Junho Choi wrote:
> > On Tue, Nov 7, 2023 at 4:05 PM Philip Paeps <philip@freebsd.org>
> > wrote:
> >> On 2023-11-07 05:40:20 (+0800), Junho Choi wrote:
> >>> On Tue, Nov 7, 2023 at 1:58 AM Philip Paeps wrote:
> >>>> On 2023-11-06 19:34:59 (+0800), Junho Choi wrote:
> >>>>> I am running ftp/rsync.kr.freebsd.org and recently (not sure when
> >>>>> it
> >>>>> starts, but at least a few weeks) starting to see it's always
> >>>>> full:
> >>>>
> >>>> Could you let me know a src address I can grep for in the
> >>>> ftp-master
> >>>> logs?
> >>>
> >>> Source address should be 182.252.182.30 (ftp.kr.freebsd.org)
> >>
> >> I believe I've fixed your problem.  You were hitting the wrong
> >> section
> >> of the rsyncd.conf with tighter limits.  Please let me know if you
> >> still
> >> run into limits.
> >
> > Yes it started working again. Thanks!
>
> Hooray!
>
> > For a full mirror, what's the requirement? Last time I checked it's
> > too big for me. (I have one server running, not easy to expand)
>
> Still too big for you in that case, I think. ;-)
>
> We can do single-machine installations, but the machine has to be very
> beefy.  pkg alone currently needs about 10TB on disk on average, and a
> bit more during times we have to support three stable/X branches plus
> main.  The workload also wants 100+GB memory for ZFS to keep up.
>
> But no worries.  As I wrote earlier: you're well connected to Japan,
> where we have two full multi-machine installations (AS9597 and AS7530).
>
> Thanks again for keeping ftp.kr.freebsd.org up!
>
> Best wishes.
> Philip
>
> --
> Philip Paeps
> Senior Reality Engineer
> Alternative Enterprises



-- 
Junho Choi <junho dot choi at gmail.com> | https://saturnsoft.net