Requirements Final Draft Attempt #2 :-/

[João Carlos Mendes Luís]

Oliver Fromme wrote:
> João Carlos Mendes Luís wrote:
>  > - After discovering some "content directory", verify it's contents to 
>  > see if it's true.  This is a little intrusive, but could be done by 
>  > verifying the presence and size of some file, for example.
> 
> I was considering that, too.  But it would increase the time
> of the collector run by an order of magnitude.
> 
> Currently, it sends two LIST commands (for releases and for
> packages), then it proceeds sending LIST commands for every
> architecture it found, as well as for every ISO-IMAGES sub-
> directory.  The last collector run took 3 hours 40 minutes
> (148 mirror hosts).
> 
> If the collector was to check every directory it found, it
> would have to issue at least another LIST command for every
> release and every architecture on every host.  The last

     And have some hardcoded or file table to compare names and sizes.

> collector run found 6755 directories (that's the number of
> lines in the log file).  On average, a LIST command takes
> 2 - 3 seconds.  For some hosts it takes considerably longer,
> depending on reachability and load of that server.  To some
> countries, the RTTs from here are not very good.

     I do not program python, so please forgive me if this is stupid: 
Isn't it possible to thread the requests per IP?

>  > - If a problem is found, send an email to the CC.freebsd.br hostmaster, 
>  > and to postmaster at ftpX.CC.freebsd.org.
> 
> To be honest, I don't like the idea to spam people by using
> automated scripts.

     This is already done by the ports system.  Every port which cannot 
compile or fetched generates a message to it's maintainer, once a month, 
I think.  I have already received some, and gladly consider this not as 
spam, but part of my FreeBSD colaboration responsibility duties.

>  >      The discussion about ftp holes and links reminded me about other 
>  > possible problem: Some ftp DNS names may point to multiple IP Addresses 
> 
> If that's done on purpose, then the responsible admins must
> make sure that all hosts under the same IP _must_ have the
> same content.  Otherwise I would call it broken.
> 
>  > (I've seen this already, but do not know if it's valid),
> 
> It's valid and called "DNS round-robin".  It's a very simple
> way to provide load balancing among a set of hosts, but it
> has some limitations.  Especially Windows boxes don't work
> very well with it.  It's always preferable to use a "real"
> round-robin setup if possible, e.g. using L7 switches or a
> balancing proxy.

     Well, maybe you program is also the exact point to check for this 
problem, warn for multiple IPs, and error for multiple IPs with 
different contents.

>  > and some 
>  > multiple DNS names could point to the same IP (ftp3.de and ftp6.de 
>  > example).
> 
> Right.  The collector could be optimized to recognize that,
> so it doesn't check the same host twice.  I'll try to
> implement that when I have a few more minutes of time ...
> 
> However, I think it should list them twice, as if it were
> separate hosts.  After all, from a user's perspective, they
> are different hosts (which happen to have the same content).

     List them as aliases to the first host you found, so users can know 
they are valid, but not not waste time trying sites they have already gone.

     Let me remember you that my first suggestion was to leave a copy of 
this information in each site, to help sysinstall decide where could it 
get the needed files.

     Just my €0,02,

                                         Jonny

-- 
João Carlos Mendes Luís - Networking Engineer - jonny at jonny.eng.br
--
"the West won the world not by the superiority of its ideas or values
or religion but rather by its superiority in applying organized
violence. Westerners often forget this fact, non-Westerners never do."
-- Samuel P. Huntington