BSDstats v3.0 - The Security Rewrite

[Marc G. Fournier]

On Thu, 17 Aug 2006, Oliver Fromme wrote:

> (1)  When run for the first time, you get an error message:
> : not found
> That's because a few bogus spaces after the backslash in
> the line containing the chmod command.  Those trailing
> spaces should be removed.  I suppose I don't need to send
> a PR for that.  :-)

'k, that one is/was fixed ...

> (2)  Some people aborted the inital "sleep 900" (because
> of the above-mentioned error message, or other reasons),
> then restarted the script.  In this case there is no sleep,
> and the submission _seems_ to be successful (no negative
> feedback), but it isn't.
>
> One way to improve the situation would be to check the
> mtime on the /var/db/bsdstats file.  If it's younger than
> 900 seconds, a sleep is required.  For example, something
> like this piece of shell code (untested):
>
> FILETIME=$( stat -f %m $id_token_file )
> NOW=$( date +%s )
> if [ $(( $NOW - 900 )) -le $FILETIME ]; then
>        SLEEPTIME=$(( 900 - ($NOW - $FILETIME) ))
>        echo "Token key is younger than 15 minutes!"
>        echo "Sleeping $SLEEPTIME seconds, please wait."
>        sleep $SLEEPTIME
> fi

Code tested, and committed ... thanks ...

> (3)  Some sites require the use of a proxy for HTTP access.
> Such sites usually have an entry in /etc/make.conf, so the
> ports can fetch their distfiles:
>
> FETCH_ENV=  FTP_PROXY=http://proxy.my.site:3128 \
>            HTTP_PROXY=http://proxy.my.site:3128
>
> The bsdstats script could easily pick up that entry and set
> the environment variables appropriatly.  This line at the
> beginning of the script should be sufficient:
>
> export $( make -V FETCH_ENV 2>/dev/null )

'k, that one could be a problem, since adding that line produces:

# /usr/local/etc/periodic/monthly/300.statistics
MACHTYPE
SSH_CLIENT
USER
MAIL
SHLVL
VENDOR
HOME
SSH_TTY
PAGER
GROUP
LOGNAME
BLOCKSIZE
TERM
PATH
REMOTEHOST
HOST
SHELL
OSTYPE
PWD
SSH_CONNECTION
FTP_PASSIVE_MODE
HOSTTYPE
EDITOR
Token key is younger than 15 minutes!
Sleeping 361 seconds, please wait.

Is there a better way of doing it that it would be silent?

> (4)  Some sites have a proxy that requires authentication.
> It is possible to include the password in the FETCH_ENV
> entry in /etc/make.conf, but it's usually not a good idea
> to do that, because you shouldn't write passwords to files
> that are world-readable.
>
> That problem could be solved in different ways.  One way
> would be a periodic.conf setting that instructs the script
> not to try to submit the data, but instead just print a
> reminder to the admin that he should run the monthly script
> manually (or print that reminder automatically when the
> submission fails because the proxy denies access).
> When the admin runs the script manually (which could be
> detected by "test -t 0", i.e. stdin is a terminal), it
> could ask for the HTTP proxy password and then set the
> HTTP_PROXY_AUTH variable appropriately (see fetch(3)).

'k, this one has been brought up, and was something that I am hoping to 
address once I get back online properly this week ...

> (5)  Some machines might not be able to access the web at
> all.  For example, I'm right now working on a farm of 35
> machines which don't have internet access, not even via
> a proxy.  I can connect to them via ssh/scp (port 22) from
> a management machine, and that management machine only has
> web access via a proxy.
>
> It would be nice to be able to request token keys on behalf
> of those 35 servers from the management machine, transfer
> them to the servers, run the data gathering script on the
> servers (putting it into a file instead of submitting it
> directyl), copy the results to the management machine and
> finally submit them from there.  That's pretty complicated,
> but I'm afraid I haven't gotten a better idea so far.  :-(

Actually, this is one that we have discussed, and believe we have a 
solution for already, I just  have to sit down and code this one ...

And I think it might actually act as a way of dealing with (4) as well ...

We're goin ot have a 'use_email' setting ... what will happen is as 
follows:

Initial Install / Run:
   Email is sent to root containing IDTOKEN= as generated by host, root 
forwards that to rpt at bsdstats.org, rpt at bsdstats.org sends back KEY= value 
to be put in /var/db/bsdstats (manual cut-n-paste) ... script is re-run a 
second time, submits report values to root, root forwards that to 
rpt at bsdstats.org ...

Subsequent Runs:
   Email is sent to root containing report values, root forwards that to 
rpt at bsdstats.org

Now, in theory, root could have a filter on it that 'if subject = bsdstats 
report, auto-forward to rpt at bsdstats.org', but that would be totally up to 
the admin whether they wanted to do each report manually or not ...

> (6)  All of the statistics on the web page are sorted by percentages. 
> It would be nice to be able to click on a column header and have the 
> table sorted by that value. That would be especially useful for the 
> release statistics and the country statistics.

That is all major work in progress ... my first goal was to get data into 
the database, and reporting in place (and improve the script based on 
feedback like above) ... improving the stats is goal two :)

> (If the PHP sources and a database export were publicly
> available, I would have taken a shot at implementing it.)

Antony is working on site design and such, but if you are interested in 
helping out with building reports, and  have time, we'd be happy for the 
extra help :)