Sorry state of the rsync based CVS,replication

[Ken Smith]

On Wed, 2010-10-27 at 14:45 +0200, Patrick Bihan-Faou wrote:

> We use FreeBSD extensively and keep a local mirror of the CVS 
> repository. Up until recently things where working properly with the 
> various servers listed in 
> http://www.freebsd.org/doc/handbook/mirrors-rsync.html, but sometime 
> during the summer ftp13.freebsd.org did not respond anymore and since 
> then rsync replication is broken.
> 
> The main issue (besides the removal of ftp13.freebsd.org) is that most 
> rsync sources refuse to replicate the content of the .Attic directories 
> in the CVS tree. This means that performing a check-out on ports using a 
> tag usually won't work as some files will not be there anymore.
> 
> Here are the typical logs I get using most rsync servers:
> 
> rsync: opendir 
> "/3/freebsd-core/development/FreeBSD-CVS/ports/chinese/pcmanx/files/Attic" 
> (in vol) failed: Permission denied (13)
> 
> At this moment the only rsync server that provides an adequate 
> replication of the CVS repository is ftp2.tw.FreeBSD.org.

As others have reported this was caused by the permissions on the
Attic directories not including world read permission.  For sites
where it was working it's actually an indication they're not following
"best practices" for a mirror site.  It's typically a bad idea to have
the thing that allows access to the content of the mirror site running
with the same credentials as what keeps the mirror site up to date.  We
don't use the 'feature' that allow for (pre-staging content that the
world shouldn't have access to for a period of time, allowing the mirror
sites to get fully populated before the release date) but I know of
other projects that do.  The ftp-master machines don't have that in
place because they're not public and they need to allow the blessed
mirrors access to everything (for the purposes of pre-staging, if we
were actually using that feature...).

The development/ section of the FTP site is something I hadn't looked
at before so it took me a little time to find what populates it and
investigate a little.  I *think* the issue with the Attic directories
not including world-read permissions was either an issue with a badly
formed chmod(1) done a long time ago or an issue with the mechanism
that populates that portion of the FTP site missing a umask setting
in the script that does it some time back in history (it's there now).
Not all of the Attic directories had the wrong permissions, it seemed
to stop some time in 2007.

I adjusted the permissions on ftp-master so hopefully this issue is
fixed.  However ...

> We are moving to svn and svnsync for the freebsd source tree (and I am 
> happy with this), but the ports do not seem to be available using SVN 
> (or not in a documented way).
> 
> Can something be done to restore RSYNC mirroring of the CVS tree to a 
> working state ?

The FTP site desperately needs to go on a diet so we're poking around
to see if there is some stuff that can be dropped.  This section of
the site is a candidate for being removed.  As you say the ports are
not available in SVN but I'm curious why you use the content from the
FTP site instead of just using a CVSUP mirror.  Is there some benefit
to it?  We would sort of like to stop providing this as part of the
FTP site if there really isn't any benefit to it over using the
cvsup mirror infrastructure which won't be going away any time soon.

Thanks.

-- 
                                                Ken Smith
- From there to here, from here to      |       kensmith at buffalo.edu
  there, funny things are everywhere.   |
                      - Theodor Geisel  |
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 196 bytes
Desc: This is a digitally signed message part
Url : http://lists.freebsd.org/pipermail/freebsd-questions/attachments/20101111/e561a91b/attachment.pgp