pwd_mkdb: Cosmetic nit.

Sponsored by: Klara, Inc.

Remove copyright strings ifdef'd out

We've ifdef'd out the copyright strings for some time now. Go ahead and
remove the ifdefs. Plus whatever other detritis was left over from other
recent removals. These copyright strings are present in the comments and
are largely from CSRG's attempt at adding their copyright to every
binary file (which modern interpretations of the license doesn't
require).

Sponsored by: Netflix

usr.sbin: Remove ancient SCCS tags.

Remove ancient SCCS tags from the tree, automated scripting, with two
minor fixup to keep things compiling. All the common forms in the tree
were removed with a perl script.

Sponsored by: Netflix

Remove $FreeBSD$: one-line .c pattern

Remove /^[\s*]*__FBSDID\("\$FreeBSD\$"\);?\s*\n/

pwd_mkdb: Sort options and update usage message.

Sponsored by: Klara, Inc.
Reviewed by: allanjude
Differential Revision: https://reviews.freebsd.org/D39266

pwd_mkdb(8): Don't copy comments from /etc/master.passwd to /etc/passwd.

The intention of /etc/passwd was to support legacy applications that are
not yet converted to use modern API like getpwent(3). Comments are not
defined in the legacy format, so copying them could break these
applications. Plus, it could leak sensitive information (e.g. encrypted
form of password of an user that was commented out instead of deleted
or disabled).

PR: bin/144652
MFC after: 1 month

pwd_mkdb: retire -B and -L endianness options

Legacy v3 db support was retired in r333133, and it was v3 support that
required the -B and -L options. The options were retained temporarily,
but now that stable/12 has branched they can be removed.

Sponsored by: The FreeBSD Foundation

pwd_mkdb: retire legacy v3 db support (-l option)

pwd_mkdb has emitted v4 password database records since 2003 (r113596)
in addition to v3, and as of r283981 by default it emitted only v4.
As described in r283981, retire the -l legacy option.

The -B and -L options were originally added to set the endianness of v3
records emitted by pwd_mkdb, but they also set the db hash endiannes and
so have been retained temporarily.

Announced on the FreeBSD-Current and FreeBSD-Stable lists. In stable/11
the man page contains a deprecation notice, and pwd_mkdb will emit a
deprecation notice if the -l option is specified.

Reviewed by: delphij, lidl, rgrimes
Relnotes: Yes
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D15144

pwd_mkdb: default to network (big) endian hash order

For cross-architecture reproducibility. The db(3) functions work with
hashes of either endianness, and the current (v4) version password db
entries already store integers in network order. Do so with the hash as
well so that identical password databases can be created on big- and
little-endian hosts.

The -B and -L flags exist to set the endianness for legacy (v3) entries
when the -l flag is used, and they will still control hash endianness
(at least until the backwards compatibility infrastructure is removed).

MFC after: 1 week
Sponsored by: The FreeBSD Foundation

pwd_mkdb: warn that legacy support is deprecated (if specified)

r283981 switched pwd_mkdb to emit only v4 database entries by default,
and introduced a -l (legacy) option emit v3 entries in addition. The
commit message claims that legacy support will be removed in 12.0, so
emit a warning now if it is used.

General further adoption of SPDX licensing ID tags.

Mainly focus on files that use BSD 3-Clause license.

The Software Package Data Exchange (SPDX) group provides a specification
to make it easier for automated tools to detect and summarize well known
opensource licenses. We are gradually adopting the specification, noting
that the tags are considered only advisory and do not, in any way,
superceed or replace the license texts.

Special thanks to Wind River for providing access to "The Duke of
Highlander" tool: an older (2014) run over FreeBSD tree was useful as a
starting point.

Renumber copyright clause 4

Renumber cluase 4 to 3, per what everybody else did when BSD granted
them permission to remove clause 3. My insistance on keeping the same
numbering for legal reasons is too pedantic, so give up on that point.

Submitted by: Jan Schaumann <jschauma@stevens.edu>
Pull Request: https://github.com/freebsd/freebsd/pull/96

Following revision r295924, the changes to a db file should be fsynced
before the file is closed. Consequently, it shouldn't be necessary to
open the file with O_SYNC any more.

This improves the performance of building large .db files for large
password files a lot and should resolve this problem:

https://forums.freebsd.org/threads/10-2-pre-pwd_mkdb-slow-on-larger-master-passwd.52700/

Differential Revision: https://reviews.freebsd.org/D5186
Reviewed by: garga, vangyzen, bapt, se
MFC after: 1 week

The sdp opens the database with PERM_SECURE mode and it is different than
dp that opens the database with PERM_INSECURE, so we need to check sdp->put
against sdp instead of use dp->put.

PR: bin/191720
Submitted by: Miles Ohlrich <turingsboy@yahoo.com>
Approved by: rodrigc (mentor)
Differential Revision: https://reviews.freebsd.org/D4255

When passwd or group information is changed (by pw, vipw, chpass, ...)
temporary file is created and then a rename() call move it to official file.
This operation didn't have any check to make sure data was written to disk
and if a power cycle happens system could end up with a 0 length passwd
or group database.

There is a pfSense bug with more infor about it:

https://redmine.pfsense.org/issues/4523

The following changes were made to protect passwd and group operations:

* lib/libutil/gr_util.c:
- Replace mkstemp() by mkostemp() with O_SYNC flag to create temp file
- After rename(), fsync() call on directory for faster result

* lib/libutil/pw_util.c
- Replace mkstemp() by mkostemp() with O_SYNC flag to create temp file

* usr.sbin/pwd_mkdb/pwd_mkdb.c
- Added O_SYNC flag on dbopen() calls
- After rename(), fsync() call on directory for faster result

* lib/libutil/pw_util.3
- pw_lock() returns a file descriptor to master password file on success

Differential Revision: https://reviews.freebsd.org/D2978
Approved by: bapt
Sponsored by: Netgate

Whitespace changes for r283981. No actual code changes.

In r113596, version 4 of entries have been added but pwd_mkdb have
been generating both new (machine independent) and legacy version
entries (endianness sensitive).

The base system have been using the new format for quite some time,
so disable the generation by default.

An interim option, -l, have been added to re-enable old behavior.
The -l, -B and -L options are considered deprecated and will be
removed in FreeBSD 12.0 release.

Mark global functions and/or variables in pwd_mkdb(8) static where possible.

This allows compilers and static analyzers to do more thorough analysis.

Only increment the ypcnt once per data record. The old implementation
was incrementing it twice making it impossible to iterate the table
since the records were 1, 3, 5, 7 (or 2, 4, 6, 8 for the v3 records).

MFC after: 10 days

A new jail(8) with a configuration file, to replace the work currently done
by /etc/rc.d/jail.

Create the altix project branch. The altix project will add support
for the SGI Altix 350 to FreeBSD/ia64. The hardware used for porting
is a two-module system, consisting of a base compute module and a
CPU expansion module. SGI's NUMAFlex architecture can be an excellent
platform to test CPU affinity and NUMA-aware features in FreeBSD.

Verify that the username length is smaller than MAXLOGNAME when
asked to verify a passwd file (pwd_mkdb -C).

Entries with oversized usernames are still permitted when building
the passwd database.

When entries are >= MAXLOGNAME in length, they are correctly stored
in passwd, pwd.db and spwd.db but are only correctly retrieved by
getpwent*() and getpwuid*(). getpwnam*() truncates to MAXLOGNAME - 1
when reading from a file (breaking at least sh, tcsh and bash)
and utilities such as su(1) check, complain and fail if the
passed name is >= MAXLOGNAME in length.

MFC after: 3 weeks

- Import the HEAD csup code which is the basis for the cvsmode work.

Correctly handle an input file without a newline on the last line (and
avoid the confusing error message about the line being too long). This
change uses fgetln to detect the right conditions, but the fixed-width
line buffer is kept because too many other places in the program make
assumptions about its maximum width.

Approved by: re (scottl)

Even if variable is never used uninitialized by the semantic, reduce compiler
warning by giving an initial value in all cases.

Add endianness support.

While version 4 entries are architecture-independent, we
also store old (version 3) entries in native byte order.
Also, the hash itself is created in a native byte order.

With this change, pwd_mkdb(8) can be used to cross-build
*pwd.db files for another architecture.

Tested on: i386, amd64, alpha, sparc64

Turn K&R functions into prototypes.

Per letter dated July 22, 1999 remove 3rd clause of Berkeley derived software
(with permission of addtional copyright holders where appropriate)

Add 'i' to the getopt string.

Noticed by: jhein

Add support for ignoring locking failures. This is only enabled when
you've specified a directory. It is intended to be used in building
custom releases over NFS where locking may be unreliable at best and
there is no contention that the locking is designed to arbitrate.
Other uses of this flag are discouraged. Document same in usage and
man page (including the warning about unwise).

Sponsored by: Timing Solutions

Two style related changes:
(1) use strlcpy instead of strncpy since the use here of the latter
was incorrect.
(2) Move 'N' case into proper sorted order (sorted the same way that
ls sorts its args).

Use __FBSDID over rcsid[]. Protect copyright[] where needed.

When updating a single entry (i.e. when the `-u' option is given),
do not add the "\xFF" "VERSION" key --- it should only be added once
ALL entries have been updated.

While I'm here, correct the logic that detects whether or not the
user-ID has changed so that it works even if all entries have not
yet been updated to the new format.

Users missing-in-action were
Reported by: tjr, Vallo Kallaste <vallo@estcard.ee>,
leafy <leafy@leafy.idv.tw>

Sponsored by: DARPA, Network Associates Laboratories

Revert the definitions of _PW_KEY* to their previous values. There is
at least one consumer outside of libc and pwd_mkdb.
Adjust the versioning in libc and pwd_mkdb accordingly.

named was the application affected, and that fact was first
Reported by: Zherdev Anatoly <tolyar@mx.ru>

Sponsored by: DARPA, Network Associates Laboratories

= Implement thread-safe versions of the getpwent(3) and getgrent(3)
family of functions using the new nsdispatch(3) core. Remove
arbitrary size limits when using the thread-safe versions.

= Re-implement the traditional getpwent(3)/getgrent(3) functions on
top of the thread-safe versions.

= Update the on-disk format of the hashed version of the passwd(5)
databases to allow for versioned entries. The legacy version is
`3'. (Don't ask.)

= Add support for version `4' entries in the passwd(5) database.
Entries in this format are identical to version 3 entries except
that all integers are stored as 32-bit integers in network byte
order (big endian).

= pwd_mkdb is updated to generate both version 3 and version 4
entries.

Sponsored by: DARPA, Network Associates Laboratories

WARNS=4 cleanup, de-__P()

When reporting that a line is too long, include the line number in the
error message. While I'm here, add a note that the "line too long"
message isn't always accurate.

PR: 35395
Submitted by: andrew@ugh.net.au

Explicitly use int32_t for on-disk records for pw_change and pw_expire,
since that is what we use now and this insulates us from any time_t
tweaks here. We can define a record format that uses 64 bit times if/when
we need to.

Add nsswitch support. By creating an /etc/nsswitch.conf file, you can
configure FreeBSD so that various databases such as passwd and group can be
looked up using flat files, NIS, or Hesiod.

= Hesiod has been added to libc (see hesiod(3)).

= A library routine for parsing nsswitch.conf and invoking callback
functions as specified has been added to libc (see nsdispatch(3)).

= The following C library functions have been modified to use nsdispatch:
. getgrent, getgrnam, getgrgid
. getpwent, getpwnam, getpwuid
. getusershell
. getaddrinfo
. gethostbyname, gethostbyname2, gethostbyaddr
. getnetbyname, getnetbyaddr
. getipnodebyname, getipnodebyaddr, getnodebyname, getnodebyaddr

= host.conf has been removed from src/etc. rc.network has been modified
to warn that host.conf is no longer used at boot time. In addition, if
there is a host.conf but no nsswitch.conf, the latter is created at boot
time from the former.

Obtained from: NetBSD

Fix various unsigned vs signed errors that caused problems with uids
and gids bigger than 16 bits. Added checks for uids and gids that are
bigger than 32 bits.

Approved by: jkh (partly, this fix is bigger than I first intended)

Replace the -q option to pwd_mkdb with a test for PW_SCAN_BIG_IDS in
the environment. This allows big ID warnings to be suppressed for
vipw and chpass as well.

Since the environment variable test is only performed for callers
of pw_scan() that do not set pw_big_ids_warning, the test can still
be overriden. Currently, chpass and pwd_mkdb are the only users
of pw_scan() and neither of them overrides the environment variable
test.

Add to pwd_mkdb a -q option to silence warnings about large IDs. Add a
suitably ominous warning in the manual page.

The diff applied is not the one provided in the attributed PR.

PR: 13344
Reviewed by: bde

Unifdef -DPASSWD_IGNORE_COMMENTS. This wasn't really optional and
we have enough pseudo-options already.

$Id$ -> $FreeBSD$

Enable source file locking in pwd_mkdb by including the proper letter
in getopt(). The code was there, the means to use it wasn't.

Also update the usage() statment to reflect reality.

PR: bin/9248
Submitted by: Jos Backus <jbackus@plex.nl>
Forgotten By: dillon

PR: bin/3478

Have pwd_mkdb lock the source file while rebuilding the database. When
called by programs such as vipw, the source file is a temporary file and
this does not conflict with the lock on /etc/master.passwd already held
by vipw. When run manually, however, master.passwd is typically specified
as the argument and the locking prevents other programs from messing with
master.passwd during the database rebuild.

Also pwd_mkdb uses a blocking exclusive lock as it may be called from
a script. The -N option was added to cause pwd_mkdb to get the lock
non-blocking and exit with an error if the attempt fails, again useful
for scripts.

Close PR bin/8753 pwd_mkdb problem when having comments in passwd file

Submitted by Chia-liang Kao clkao@CirX.ORG .

Rename a static variable, so it will not shadowed by a local variable.
Now comments will be ignored, rather than put junk in the password database.

Broken in: rev. 1.21

Fix usage string

Allow specification of cache-size to pwd_mkdb
PR: 5193
Reviewed by: phk
Submitted by: Nick Hilliard <nick@foobar.org>

Reenable building of /etc/passwd.
Pointed out by: "Julie M. Juracich" <julie@xaqti.com>

Apparently, if fprintf() fails, it can return any negative value
Pointed out by: Bruce

Check return values of fprintf, fclose (this one is overdone I guess)
and pw_db->close.
PR: 4202 (probably, I askd the submitter)
Obtained from: OpenBSD

Rename the pwd_mkdb(8) option '-c' to '-C' for better compatiblity
with BSD/OS.

Staticize usage(). Cosmetics.

db close routine actually writes data out to disk; return value was
not being checked. Change code to fail if db->close errors.

PR: 4202

compare return value from getopt against -1 rather than EOF, per the final
posix standard on the topic.

Fix possible (not stack) overflow from -d <dir> commandline option.

Allow comments in password database. The comments are copied from
the password file into /etc/master.passwd and optional (-p) into
/etc/passwd. Enable this feature with the compile
option -DPASSWD_IGNORE_COMMENTS.

The character `#' introduces a comment. Leading spaces and tabs are
ignored: '^[ \t]*#.*\n$'

Count an empty line - only spaces, tabs or newline - also as a comment.
An empty line at the bottom of /etc/master.passwd is a common
novice error and increased my mail load: '^[ \t]*\n$'

Begin closing out PR #1519 (this requires a change to chpass too,
and both changes need to be pulled into the stable branch). The
problem here is that when pwd_mkdb creates /etc/passwd, it turns
empty UID and GID fields into zeroes. To fix this, we check the
_PWF_UID and _PWF_GID bits in the pw_fields flag: if the bits
are not set, we print an empty field instead of a zero. This way,
you don't get zeroes in the UID or GID fields unless you explicit
want them.

New option: [-c]
Check if the password file is in the correct format. Do not
change, add, or remove any files.

Fixed brackets in usage message.

Reformatted some of recent changes to KNF.

Check if username is a NULL pointer before dereferencing it.

Implement incremental passwd database updates. This is done by ading a '-u'
option to pwd_mkdb and adding this option to utilities invoking it.
Further, the filling of both the secure and insecure databases has been
merged into one loop giving also a performance improvemnet.
Note that I did *not* change the adduser command. I don't read perl
(it is a write only language anyway).
The change will drastically improve performance for passwd and
friends with large passwd files. Vipw's performance won't change.
In order to do that some kind of diff should be made between the
old and new master.passwd and depending the amount of changes, an
incremental or complete update of the databases should be agreed
upon.

NIS cleanups and fixes, the next generation, continued.

pwd_mkdb.c:

- Don't save the PLUSCNT and MINUSCNT tokens: we don't need them anymore.

- Count the + and - entires for NIS together instead of counting + and -
entries seperately. Index all special NIS entries using new _PW_KEYYPBYNUM
token.

pwd.h:

- Remove the PLUSBYNUM, MINUSBYNUM, PLUSCNT and MINUSCNT tokens and replace
then with a single _PW_KEYYPBYNUM token.

Remove trailing whitespace.

Add what looks to be the right fix to this bogus code.

Re-enable +::::::::: wildcards that I had previously disallowed.

Note: if you put +::0:0:::::: in /etc/master.passwd as your only NIS
entry, it will cause all NIS uids and gids to be remapped to zero. This
is *intentional*. That's the way it's supposed to work. Enabling NIS with
no remapping at all is done with +:::::::::, not +::0:0::::::. Similarly,
+:::::::::/bin/csh will remap the shells of all NIS users to /bin/csh.
Or, you could do +wpaul:::::::::/bin/csh to remap NIS user wpaul's shell
to /bin/csh but leave everyone else alone.

Summarily enable YP as soon as any a '+' is detected instead of generating
special values when +user substitutions are in effect. The new getpwent
code can handle all the special cases itself now.

- Add support for embedding special entries in the password databases
for +@netgroup/-@netgroup entries. This saves the getpwent functions
from having to do all the work.

- Fix potential bug: when pwd_mkdb writes the YP-enabled flag to the secure
password database, it uses the wrong database descriptor. (It uses the
descriptor from the non-secure database, which is already closed by the time
things are being written into the secure dastabase).

Fix security hole in YP password support, which wouldmake +user entries
act like + entries in some cases. Requires support in libc from next
commit.

Maintain pw_fields, and output same to password database.

!!!!!!!!
NB
!!!!!!!!
You MUST pwd_mkdb /etc/master.passwd before attempting to use the new
libc, or things may go wrong. (I doubt anything actually /will/ go
wrong, but the actual behavior is undefined. YOU HAVE BEEN WARNED.)
The database format is, however, backwards-compatible, so old executables
will still work.

Added the -d <directory> functionality to allow construction of the
password databases somewhere other than /etc.

BSD 4.4 Lite usr.sbin Sources