package distribution crisis - CDN needed

[Brooks Davis]

On Tue, Apr 08, 2008 at 10:15:48AM +0200, Pav Lucistnik wrote:
> Eberhard Moenkeberg p??e v ?t 08. 04. 2008 v 01:54 +0200:
> 
> > > > I do propose a creation of a CDN (Content Delivery Network), having these
> > > > features:
> > > 
> > > The success of a CDN is premised on the working set of active files being
> > > [significantly] smaller than the set of available files.  While I don't doubt
> > > that only a few packages are of interest to most users, for a CDN to provide a
> > > real benefit it needs to be the case that no users require most files.  The
> > > reason being that, if you have to transfer all or even most files every time,
> > > a CDN may well be a significantly less efficient way to do it than rsync. Has
> > > a working set analysis for our main mirrors been done that demonstrates that
> > > the actual working set is sufficiently smaller than the total set that a CDN
> > > would be more, rather than less, efficient?
> 
> I don't have these statistics, but I'd like to see them very much.
> Can I ask some of the mirror admins to compile me statistics on
> downloads of files from /pub/FreeBSD/ports/*packages*, say for March
> 2008 ?
> 
> I'd also like to have these numbers from ftp.FreeBSD.org so I have a
> comparision of volume.
> 
> My suspicion is that 99% of users get their packages from
> ftp.FreeBSD.org, because that's what pkg_add -r uses without PACKAGEROOT
> specification.

I suspect that like virtually everything else on the internet package
access is heavy-tailed.  If so, no from of cooperative caching will be
of use.

For example, this paper killed all research on cooperative caching
web proxies by demostrating that its was fundamentally pointless:

http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.22.9583

-- Brooks
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 187 bytes
Desc: not available
Url : http://lists.freebsd.org/pipermail/freebsd-hubs/attachments/20080408/177d8078/attachment.pgp