running newsyslog fiveminly

[Jeremy Chadwick]

On Wed, Sep 07, 2011 at 03:45:34PM +0700, Eugene Grosbein wrote:
> 01.08.2011 00:31, Jeremy Chadwick ?????:
> > On Sun, Jul 31, 2011 at 11:51:40PM +0700, Eugene Grosbein wrote:
> >> Hi!
> >>
> >> Suppose, there is a machine which writes two kinds of log files through syslogd:
> >> quickly-growing that need to be rotated based on their size (hourly is too seldom)
> >> and other that should be rotated once a day, at midnight only.
> >>
> >> For first kind of logs we have to run newsyslog once every 5 minutes using cron:
> >>
> >> */5     *       *       *       *       root    newsyslog
> >>
> >> For second kind of logs we have lines in newsyslog.conf such as following:
> >>
> >> /var/log/mpd.log 640 16 * @T0000  JC
> >>
> >> This must ensure that /var/log/mpd.log is rotated and compressed at midnigt only.
> >> Note, that compressing the file takes 8 minutes.
> >>
> >> However, every night at 00:05 I get an error:
> >>
> >> bzip2: I/O or other error, bailing out.  Possible reason follows.
> >> bzip2: No such file or directory
> >> 	Input file = /var/log/mpd.log.0, output file = /var/log/mpd.log.0.bz2
> >> newsyslog: `bzip2 -f /var/log/mpd.log.0' terminated with a non-zero status (1)
> >>
> >> It seems, newsyslog still wants to process my file at 00:05 despite @T0000
> >> time specification. Is it broken?
> > 
> > I have three things to say on the matter, all of which are somewhat
> > independent of one another so please keep that in mind.  I imagine #1
> > below is your problem.
> > 
> > 1) The newsyslog.conf(5) man page has this clause in it, for the "when"
> > field (in your case, @T0000):
> > 
> >      when    ...  If the when field contains an asterisk (`*'), log rotation
> >              will solely depend on the contents of the size field.  Otherwise,
> >              the when field consists of an optional interval in hours, usually
> >              followed by an `@'-sign and a time in restricted ISO 8601 format.
> > 
> >              If a time is specified, the log file will only be trimmed if
> >              newsyslog(8) is run within one hour of the specified time.  If an
> >              interval is specified, the log file will be trimmed if that many
> >              hours have passed since the last rotation. ...
> > 
> > You might think that "one hour of the specified time" value/clause
> > correlates with the interval that newsyslog is run at via cron, but that
> > would be wrong.  newsyslog REALLY DOES have hard-coded values for 3600
> > seconds (1 hour) in it (grep -r 3600 /usr/src/usr.sbin/newsyslog).  I
> > have not looked at the code, but the fact of the matter is, 1 hour
> > appears to be a "special" value.  I would heed that as a warning.
> 
> After reading newsyslog code, now it's obvious it just ignores minutes and seconds
> while making decision if a file should be rotated. It looks at hours only.
> That's sad.

I imagine this "design limitation" is due to the fact that newsyslog is
called from cron, which only supports minute-level granularity.

The newsyslog.conf man page even hints at this while describing the
"when" column:

    There is no provision for the specification of a timezone.  There
    is little point in specifying an explicit minutes or seconds com-
    ponent in the current implementation, since the only comparison
    is ``within the hour''.

Given this, I would say the "special" 3600-second value within the
source code makes sense.

I'm not sure what you could use for an alternate method of log rotation
for syslog-logged data.

Let's circle back to the reason you're rotating logs every 5 minutes:

>>> Most of my boxes are diskless NanoBSD installations having /var in
>>> memory and I need very detailed debug logs that grow quickly. These
>>> logs can easily overflow /var partition in case of network problems
>>> (storms etc.) so newsyslog have to check them often.
>>>
>>> And I have another router that has an HDD to keep daily log and I'd
>>> like to have their crontabs unified.

I think what the rest of the world might tell you is something to the
effect of "you can't have your cake and eat it too".  You've got
diskless systems that aren't syslogging via network (e.g. to a pool of
syslog servers, or a single syslog server) but instead to a
memory-backed filesystem, in addition to enabling debug-level logging in
mpd by default.

A memory-backed filesystem means you don't have much disk space, and you
know this based on the need to rotate logs every 5 minutes, right?  So
I'm confused why one would need debug logging.  I imagine that the
newsyslog.conf on these machines has a very small number for the "count"
column for /var/log/mpd.log.  So chances are, by the time you noticed a
problem, the logs would have been rotated and removed, no?  So why the
debug logging?

If debug logging really is something you absolutely need, no argument
about it, then honestly it sounds like you need some sort of
"centralised" logging infrastructure for all of these diskless machines.
Most diskless machines I've used utilise some form of centralised
"something" -- whether it be a centralised DHCP/PXE server (which you
obviously have in some form), or an NFS-mounted root or /home, etc...
You get the idea.  Could you deploy similar infrastructure for syslog
and simply use a remote syslog server in syslog.conf?

-- 
| Jeremy Chadwick                                jdc at parodius.com |
| Parodius Networking                       http://www.parodius.com/ |
| UNIX Systems Administrator                   Mountain View, CA, US |
| Making life hard for others since 1977.               PGP 4BD6C0CB |