Re: git: 2f036705f337 - main - Document the two recent newsyslog(8) change (-c option and <compress> configuration option).

Hi, Mike,

On Fri, Dec 29, 2023 at 7:25 AM Mike Karels <mike@karels.net> wrote:

> On 29 Dec 2023, at 2:46, Xin LI wrote:
>
> > The branch main has been updated by delphij:
> [...]

>
> > +61174ad88e33:
> > +     newsyslog(8) now supports specifying a global compression method
> directly
> > +     at the beginning of the newsyslog.conf file, which will make
> newsyslog(8)
> > +     to behave like the corresponding option was passed to the newly
> added
> > +     '-c' option. For example:
> > +
> > +     <compress> none
> > +
> > +906748d208d3:
> > +     newsyslog(8) now accepts a new option, '-c' which overrides all
> historical
> > +     compression flags by treating their meaning as "treat the file as
> compressible"
> > +     rather than "compress the file with that specific method."
> > +
> > +     The following choices are available:
> > +      * none: Do not compress, regardless of flag.
> > +      * legacy: Historical behavior (J=bzip2, X=xz, Y=zstd, Z=gzip).
> > +      * bzip2, xz, zstd, gzip: apply the specified compression method.
> > +
> > +     We plan to change the default to 'none' in FreeBSD 15.0.
>
> Sorry not to have noticed this in the review; it was only when I saw this
> message that it sunk in that we now have *three* ways to specify
> compression,
> and I'm not even sure what the precedence is.  I would have thought that
>

The current precedence is that command line takes highest priority; when
it's not present, <compress> overrides all lines with any compression
method.

The meaning of J/X/Y/Z is changed to "this file is compressible", unless
newsyslog is in legacy mode.  It's intentional to discourage the use of
different compression types for different files (because the current
newsyslog.conf(5) format uses one character to represent compression method
and without a change to e.g. UCL or some other configuration language,
supporting additional compression methods would be much more messier).


> <compress> would replace -c.  It's a mess if the config file has entries
> that specify J and X flags as well as none, the config file has
> <compress> zstd, and the -c option is given as well.  We now have a knob
>

newsyslog would issue a warning if both -c is given and the configuration
file has <compress>.

Part of the reason to introduce <compress> was that it would move the
configuration to one place (newsyslog.conf) and there were concerns that
having -c in newsyslog in /etc/crontab is counterintuitive (as raised in
some earlier discussion).


> to override the knob to override a knob. The only reason to keep -c that
> I can think of is to specify a different compression in a single
> invocation,
> but as noted, changing compression requires manual operations that make
> it unreasonable to change it invocation by invocation.
>

The intention of -c was to be added to /etc/crontab, but with the
introduction of <compress> it's probably a good idea to just remove the
command line option or at least leave it undocumented?


> Also, using the -c option or <compress> (except with legacy) loses the
> ability to specify different compression types for different files.
>

This (the ability of specifying different compression types for different
files) is what I deliberately wanted to get rid of.

Compression types only make a meaningful difference when files are
sufficiently large, and the purpose of newsyslog is to not allow log files
to grow to an unmanageable size.  Rotating files when they are large and
also requesting to compress them means the system would have prolonged
"bursts" of CPU workload at integral hours, which would be even worse for
busy systems where logs tend to have more contents.


> People have objected to the term "legacy" as well.
>

I don't agree.

Calling it "legacy" would send a clear signal that the historical way is
discouraged (for people who are using a different compression method,
merging upstream change would be a very painful process).  We already have
4 letters to represent 4 different compression methods, that's almost 1/3
of all valid configuration letters, and we should stop doing that because
the legacy way is causing problems and won't scale when new compression
methods are added.


> I still think it would be much better to add an option letter to select
> the default compression as specified by <compress>.  This would eliminate
> the need for "legacy", and it would add the ability to have both a global
> default and an exception.  I think the redefinition of the existing flags
> to have different meanings if <compress> is given is messy.
>

I insist the default should be anything except "legacy" -- legacy is for
backward compatibility and should serve that purpose only.

The entry for -c says that we plan to change the default to "none" in 15.0.
> Hopefully that would be done via <compress> and not -c.  However, there
>

Yes, my intention was to add a line to newsyslog.conf (
https://reviews.freebsd.org/D43169 ) instead of adding it to /etc/crontab.


> was significant pushback on "none" being the default.
>

I think we should provide a reasonable default.  Compression was useful
back in the 90's, but not so much nowadays and makes grep'ing the logs
harder (and for those who want compression, they could choose e.g. 'bzip2'
anyways).  This change would also make porter's lives a lot easier as they
could install their newsyslog.conf.d files with e.g. 'J' to indicate that
the file can be compressed, without messing up with the administrator's
decision to not compress or compress it with some other compressor.