HEADS UP: people running www.freebsd.org mirrors! Switch from cvsup to rsync please.

[Peter Wemm]

If you are running a www mirror that is being updated via cvsup, *please 
stop* and switch to rsync.

Right now, the data flow is like this:

build-web --rsync--> bit0.us-west.freebsd.org -+-rsync--> cvsup-master 
--cvsup--> mirrors
+-rsync--> www.freebsd.org

You can skip a whole lot of detouring of data by elinimating cvsup from 
your mirror feeds and fetching it directly from bit0.us-west.freebsd.org

# rsync rsync://bit0.us-west.freebsd.org/FreeBSD-www-data/
This is bit0.us-west.FreeBSD.org.

drwxrwxr-x          1,536 2014/05/24 23:21:03 .
-rw-rw-r--         10,290 2014/05/24 23:20:31 4xx.html
-rw-rw-r--         10,317 2014/05/24 23:20:31 5xx.html
-rw-rw-r--         14,015 2014/05/24 23:20:31 about.html
....

To switch, please use something like this:

$ rsync -aHz --delete rsync://bit0.us-west.freebsd.org/FreeBSD-www-data/ 
/home/mirror/www-data/

of note, -H for tracking hard links, and --delete to clean up old 
files.  Don't run rsync as root, etc etc.


However I would like to make a plea here. Please take a peek at your 
server's traffic logs to check that it makes sense to keep the mirror 
going and that it's not just bots.  I'm aware that people run them for a 
variety of reasons (anything from language variants to local content to 
improving latency to participating in the community etc), but please 
don't feel obligated to keep running one if there is no longer a 
reason.  A neglected mirror is far worse than no mirror.

Even worse, we risk being penalized in search engines for duplicated 
content because it looks like spam to certain algorithms.

If it's time to stop, contact the hostmaster for your zone and have the 
dns and mirror lists updated. Please don't let them go to dead air.

-Peter