Statistics for FreeBSD FTP and CVSup mirrors

[Edwin Groothuis]

Hello Jeremy,

On Sun, Aug 24, 2008 at 09:39:56PM -0700, Jeremy Chadwick wrote:
> On Mon, Aug 25, 2008 at 01:29:02PM +1000, Edwin Groothuis wrote:
> > About a year ago I made the FreeBSD FTP mirrors statistics for the
> > availability of ISO images. Recently I have added statistics for
> > the general up-to-dateness of the FTP server (via contents of the
> > file /pub/FreeBSD/TIMESTAMP) and the general up-to-dateness of CVSup
> > mirrors, via the file CVSROOT-ports/modules.
> > 
> > The statistics can be found at
> > http://www.mavetju.org/unix/freebsd-mirrors/
> > 
> > and are available per country or global (for CVSup mirrors and for
> > TIMESTAMPS).
> > 
> > And as a brain teaser:
> > 
> >     If you have an idea on how to identify the up-to-date
> >     /pub/FreeBSD/ports/*/packages*/All/ directory let me know and we
> >     can discuss it. For example, it has to overcome these kind of issues:
> > 
> > 	ftp> dir a*
> > 	229 Entering Extended Passive Mode (|||27149|)
> > 	150 Here comes the directory listing.
> > 	[... hundreds of lines deleted ...]
> > 	226 Directory send OK.
> > 	ftp> dir z*
> > 	229 Entering Extended Passive Mode (|||60927|)
> > 	150 Here comes the directory listing.
> > 	226 Directory send OK.
> 
> I don't understand what the "issue" is there; are you saying the
> methodology used for detecting out-of-date packages is difficult
> when using the base ftp(1) client?  If so, try using the ftp/omi or
> ftp/mirror ports, which support a form of directory listing diffs,
> including by timestamp.

The issue is that the FTP *servers* can limit the amount of returned
entries in a directory listing. Some return all 19K entries. Some
return the first 2K. Some only read the first 2K entries, apply the
filter and return that output. Using a different client will not
resolve it, this is a server issue.

Edwin

-- 
Edwin Groothuis      |            Personal website: http://www.mavetju.org
edwin at mavetju.org    |              Weblog: http://www.mavetju.org/weblog/