newsyslog(8): Add option to globally override compression method.

Historically, newsyslog compressed rotated log files to save disk space.
This was useful in the early days. However, with modern file systems like
ZFS offering native compression, and with the availability of larger hard
drives, the benefits of additional compression have become less significant.
This is particularly true considering the inconvenience of decompressing
log files when searching for specific patterns.

Additionally, the original implementation of compression methods was not
future-proof. As a result, we have redefined the J, X, Y, Z flags to
signify "treat the file as compressible" rather than "compress the file
with that specific method."

A new command-line option, -c, has been introduced to allow overriding
these settings in a more future-proof way. The available choices are:

* none - do not compress, regardless of flag.
* legacy - historical behavior: J=bzip2, X=xz, Y=zstd, Z=gzip.
* bzip2, xz, zstd, gzip - apply the specified compression method.

Currently, the default is set to 'legacy' to preserve historical behavior.
However, our intention is to change this default to 'none' in FreeBSD 15.0.

Additionally, this update changes the default settings for zstd to use
multithreading and long-range options, better aligning with its intended
use.

Inspired by D42961 .

Reviewed by: debdrup (earlier version, mdoc(7))
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D43165

usr.sbin: Remove ancient SCCS tags.

Remove ancient SCCS tags from the tree, automated scripting, with two
minor fixup to keep things compiling. All the common forms in the tree
were removed with a perl script.

Sponsored by: Netflix

Remove $FreeBSD$: one-line nroff pattern

Remove /^\.\\"\s*\$FreeBSD\$$\n/

newsyslog.conf: Restrict included files in default config to [!.]*.conf

The new default config will only include files from the following
directories which end with '.conf' and do not beginning with a '.'
character:
- /etc/newsyslog.conf.d/
- /usr/local/etc/newsyslog.conf.d/

This matches the syslog.conf(5) functionality, and also prevents '.sample' or
'.pkgnew' files being included. This is important for ports which install files
in /usr/local/etc/newsyslog.conf.d/ and also for pkgbase.

Approved by: eadler
Approved by: bapt
Relnotes: yes
Differential Revision: https://reviews.freebsd.org/D17086

newsyslog.8: Remove cutesy nonsense

Sponsored by: Dell EMC Isilon

newsyslog: fix typeo for 'zstd'

Reported by: swildner@DragonFlyBSD.org
MFC After: 1 week

Add newsyslog capability to write RFC5424 compliant rotation message.

This modification adds the capability to newsyslog to write the
rotation message in a format that is compliant with RFC5424. This
capability is enabled on a per-log file basis through a new value
("T") in the flags field in newsyslog.conf. This is useful on systems
that use the RFC5424 format for log files so that the rotation message
format matches that of the other log messages. There has been recent
mention of adding an RFC5424 compliant mode to syslogd and at least
one alternative system log daemon (rsyslogd) that already has the
capability to use that format.

Reviewed by: vangyzen, ngie
Approved by: vangyzen (mentor)
MFC after: 2 months
Relnotes: yes
Sponsored by: Dell EMC
Differential Revision: https://reviews.freebsd.org/D10253

Add a new Y flag to newsyslog.conf

This makes newsyslog use zstandard to compress log files.

Given Z is already taken for gzip and zstandard compression level stands in
between gzip and xz (which has the X flag) chosing Y sounds ok :)

Make it more explicitly clear that -t will not change filename.

MFC after: 3 days

- Include /etc/newsyslog.conf.d/* and /usr/local/etc/newsyslog.conf.d/* by
default for newsyslog(8).

The /usr/local/etc/newsyslog.conf.d will give packages an opportunity to
install a default configuration to handle their own log files.

MFC after: 2 weeks
Relnotes: yes

Have -n imply -r, since dry-run mode obviously doesn't require root
privileges.

Approved by: rstone (co-mentor)
MFC after: 1 week

Minor spelling fixes.

Sync manpage's SYNOPSIS with program's usage.

Use correct alphabetical order for xz(1) in SEE ALSO section
of newsyslog manpages.

Reported by: maxim
MFC after: 3 weeks

Add xz(1) support to newsyslog.
Rewrite and simplify logfile compression code.

Approved by: gad
MFC after: 3 weeks

A new jail(8) with a configuration file, to replace the work currently done
by /etc/rc.d/jail.

Add a -S switch to override the default syslog pid file. This can be useful
if syslogd's -P switch or a syslogd alternative is being used.

MFC after: 3 weeks

Add support for creating the archived log filenames using a time-stamp
instead of the traditional simple counter.

Using the time-stamp based file-names, once a log file is archived, it
will not change name until it is deleted. This means that many backup
systems will only perform one backup of the archived log file, instead
for performing a new backup of the logfile upon each logfile rotation.

This implementation is separate from the patches in the mentioned PR,
as I wasn't aware of the existence of the PR until after I had
implemented the same functionality as the patches in the PR provide.
Unlike the PR, this new code does honor the 'log count' in
newsyslog.conf so old logfiles are deleted. This new code does not
currently support never deleting the archived logfiles.

PR: bin/29363
MFC after: 3 weeks

mdoc: move remaining sections into consistent order

This pertains mostly to FILES, HISTORY, EXIT STATUS and AUTHORS sections.

Found by: mdocml lint run
Reviewed by: ru

Create the altix project branch. The altix project will add support
for the SGI Altix 350 to FreeBSD/ia64. The hardware used for porting
is a two-module system, consisting of a base compute module and a
CPU expansion module. SGI's NUMAFlex architecture can be an excellent
platform to test CPU affinity and NUMA-aware features in FreeBSD.

MFC r202668+r200806:

Don't consider non-existence of a PID file an error, we should be able
to proceed anyway as this most likely mean that the process has been
terminated. [1]

Add a new option, -P, which reverts newsyslog(8) to the old behavior,
which stops to proceed further, as it is possible that processes which
fails to create PID file get screwed by rotation. [2]

PR: bin/140397
Submitted by: Dan Lukes [1]
Requested by: stas [2]

Add a new option, -P, which reverts newsyslog(8) to the old behavior,
which stops to proceed further, as it is possible that processes which
fails to create PID file get screwed by rotation.

Requested by: stas
MFC after: 2 weeks
X-MFC with: r200806

- Import the HEAD csup code which is the basis for the cvsmode work.

Markup fixes.

Approved by: re (blanket)

Add the -N option to not rotate any files. This is to be used in
cojunction with -C and is used by /etc/rc.d/newsyslog.

I forgot that this was in my perforce tree and not my running system and
thus committed a non-working newsyslog script.

Reported by: des
Pointy hat: brooks

Add a HISTORY section.

PR: 75282

Expand *n't contractions.

Fixed xrefs.

Sort sections.

Add a new -d argument which is used to specify an alternate root for log
files similar to DESTDIR in the BSD make process. This only affects log
file paths not config file (-f) or archive directory (-a) paths.

Mechanically kill hard sentence breaks.

Remove information about the configuration file.
Add an Xref to newsyslog.conf.5 and bzip2.1.

Add a -C (create) option for newsyslog, and a 'C' flag for entries in the
config file. If the -C option is specified once, then newsyslog will create
any entries which specify the 'C' option. If -C is given twice, then
newsyslog will create all missing log files. Some of this code comes
from NetBSD, although this implementation does not exactly match theirs.

Reviewed by: freebsd-arch
MFC after: 10 days

Document support for the new 'U' flag, which was implemented back in March
as revision 1.64 of newsyslog.c. I plan to MFC these changes next week.

MFC after: 1 week

Document support for the new -s and -R command-line options, and
the 'N' flag. These were coded in March as revisions 1.55 and 1.56
of newsyslog.c. I intend to MFC all the matching changes next week.
This also reorganizes the description of the 'flags' field to give
list of the valid flags, instead of a long paragraph explaining
each of the possible values.

Obtained from: NetBSD (in spirit at least, for -s and N)
MFC after: 1 week

Document support for the <default> entry, which was implemented in
revision 1.50 of newsyslog.c , back in February. I intend to MFC
those changes soon.

MFC after: 4 days

english(4) police.

Add a new `W' flag, that when used in conjunction with the `Z' flag or the
`J' flag means that newsyslog should wait for previously started compression
jobs to complete before starting a new one for this entry. When it is used
along with the `G' flag, in the case when several logfiles match the pattern
and should be compressed, the newsyslog will compress logs one by one,
ensuring that only one compression job is running at a time.

This prevents newsyslog(8) from overloading system by starting several
compression jobs on big files simultaneously.

Sponsored by: Porta Software Ltd
MFC after: 2 weeks

- Allow comments to be placed at the end of configuration line.

Sponsored by: Porta Software Ltd
Approved by: re
MFC after: 2 weeks

mdoc(7) police: markup fixes.

Approved by: re

Add a new newsyslog.conf flag - 'G', which if set tells newsyslog(8) that
the specified filename of the log to be rotated is in fact shell glob
pattern. In this case, all files matching this pattern will be rotated
using the same options. Useful in the case when there is no pre-defined
name for the logfiles (e.g. xtradius, samba etc).

Sponsored by: PortaOne Software Ltd
MFC after: 2 weeks

The .Nm utility

Add the ability to Bzip2 your logs.

PR: 27901
Submitted by: Anders Nordby <anders@fix.no> (stylistic changes by me)

Note the equivalence of certain time formats.

Instead of giving an example of rotating logs at the end of the month,
give an example of how to rotate logs at the beginning of the month.
Although they sound the same, since both of them rotate logs at the
beginning of the day, the former ended up taking place on, e.g., July
31 00:00 instead of the expected July 31 23:59. This is contraty to POLA.

Submitted by: Dan Langille <dan@langille.org>

Remove whitespace at EOL.

Fix a grammar bogon.

mdoc(7) police: split punctuation characters + misc fixes.

mdoc(7) police: use the new features of the Nm macro.

Allow user to specify which logs to rotate

- implement logfile archivation to a separate, user
configurable directory
- implement alternate and more flexible way to specify
logfile rotation time in addition to the ISO 8601
restricted format
- cleanup the source which was a mix of several styles
of persons who maintained it so far, ran through
knfom script got from bde.

Reviewed by: (in part) sheldonh and garyj

Remove single-space hard sentence breaks. These degrade the quality
of the typeset output, tend to make diffs harder to read and provide
bad examples for new-comers to mdoc.

.Nm corrections

$Id$ -> $FreeBSD$

Correctly refer to the companion program rather than as "chown".
Clean up wording a little (like I have any idea what real English looks like...)

Syntax for user/group is changed from "user.group" to "user:group" to be
consistant with chown(8).

Fix description of size, it's in kilobytes, not bytes.

jkoshy forgot up update the heading date on the man page.

Note that the 'owner.group' field is optional in the config file.

PR: docs/9602
Submitted by: Kazuo Horikawa <horikawa@jp.freebsd.org>

A slight bit of code and doco cleanup, but mostly:

Add the much-needed and long-desired ability to force the log rotation
to take place on a specific day or time, and document same. This works
by extending the syntax of the `when' field to allow a restricted
ISO 8601 time specification, and performing the log rotation if newsyslog
is run during the same hour as is specified.

Add optional config file field: signal number to send (defaulted to HUP)
F.e. apache restarts less agressively on USR1 then on HUP

Arg! Someone fixed the synopsis in -stable's newsyslog.8, but not
in -current. However, they fixed both the ordering and the missing -F
flag. Merge from 22 and fix ordering here, too.

1) Remove a line that somehow strayed from the one true path of
its sentence.
2) Fix usage (-F option).

.Sh AUTHOR -> .Sh AUTHORS. Use .An/.Aq.

Add "-F" (force) flag to newsyslog.
Submitted by: Terry Lambert <terry@freebsd.org>

Cosmetic in err() strings.

Show the real revision date and not the date that this
manpage is being viewed.

Allow a '-' flag as a placeholder so that path_to_pid_file
can be specified without the flags field.
Fix bogus "trail" left in parse routine where the code jumps
past the end of the line and wanders into oblivion.

1) Add yet one optional field: pid file to send SIGHUP to if log rotated
(good thing for apache f.e.) This change is backward compatible with old
newsyslog.conf files.

2) Do not compress log if SIGHUP sending failed for some reason
(f.e. pid file deleted). Newcoming messages will be lost
otherwise.

3) Misc cleanup while I am here.

Add a missing period so that the -n option is display properly
in the man page output. Closes PR# 2831.

Submitted by: Marc Slemko <marcs@znep.com>

Revert $FreeBSD$ to $Id$

Make the long-awaited change from $Id$ to $FreeBSD$

This will make a number of things easier in the future, as well as (finally!)
avoiding the Id-smashing problem which has plagued developers for so long.

Boy, I'm glad we're not using sup anymore. This update would have been
insane otherwise.

Convert to mdoc format.

Obtained from: NetBSD
imported the newsyslog command from NetBSD - it make the "rotation" of
the logfiles much simpler (it is currently done by "hand" in the
/etc/[daily,weekly,monthly] scripts) - now it will be done by invoking
newsyslog every hour which is very customizable via a
/etc/newsyslog.conf file