package distribution crisis - CDN needed

[Robert Watson]

On Mon, 7 Apr 2008, Pav Lucistnik wrote:

> Okay the situation recently was that the mirrors had no chance keeping up 
> with all the package sets I've been uploading to ftp-master.
>
> We clearly need to move beyond rsync/cvsup synced ftp mirrors. This does not 
> scale.
>
> I do propose a creation of a CDN (Content Delivery Network), having these 
> features:

The success of a CDN is premised on the working set of active files being 
[significantly] smaller than the set of available files.  While I don't doubt 
that only a few packages are of interest to most users, for a CDN to provide a 
real benefit it needs to be the case that no users require most files.  The 
reason being that, if you have to transfer all or even most files every time, 
a CDN may well be a significantly less efficient way to do it than rsync. 
Has a working set analysis for our main mirrors been done that demonstrates 
that the actual working set is sufficiently smaller than the total set that a 
CDN would be more, rather than less, efficient?

Robert N M Watson
Computer Laboratory
University of Cambridge