libutil: move ftime to libutil

It seems that there are still some applications that use ftime(3)
(for example, science/siconos and sysutils/lcdproc). The issue
is that we don't build libcompat as a shared library anymore.
The easiest solution is to move it to libutil, until we
deprecate it for good.

This solution was proposed by kib@ in
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=257789.

PR: 257789
MFC after: 1 month
Reviewed by: kib (ages ago)
Differential Revision: https://reviews.freebsd.org/D39994

lib: Remove ancient SCCS tags.

Remove ancient SCCS tags from the tree, automated scripting, with two
minor fixup to keep things compiling. All the common forms in the tree
were removed with a perl script.

Sponsored by: Netflix

Remove $FreeBSD$: one-line sh pattern

Remove /^\s*#[#!]?\s*\$FreeBSD\$.*$\n/

cpuset(3): Move cpuset's parselist function into libutil

In order to allow to add cpuset(2) functionality to more utilities than just
cpuset(1) move the parselist code into libutil

While here, make the code a little more "library" friendly, by returning a range
of various errors so that the consumer can check for them and report appropriate
error message to the users

(One of the planed usage is the jail(8) utility)

Reviewed by: jilles
Differential Revision: https://reviews.freebsd.org/D12873

Add function getlocalbase() to libutil.

This function returns the path to the local software base directory, by
default "/usr/local" (or the value of _PATH_LOCALBASE in include/paths.h
when building the world).

The value returned can be overridden by 2 methods:

- the LOCALBASE environment variable (ignored by SUID programs)
- else a non-default user.localbase sysctl value

Reviewed by: hps (earlier version)
Relnotes: yes
Differential Revision: https://reviews.freebsd.org/D27236

Revert the whole getlocalbase() set of changes while a different design is
hashed out.

Add the library function getlocalbase and its manual page. This helps to
unify the retrieval of the various ways that the local software base directory,
typically "/usr/local", is expressed in the system.

Reviewed by: se
Differential Revision: https://reviews.freebsd.org/D27022

pkgbase: Put a lot of binaries and lib in FreeBSD-runtime

All of them are needed to be able to boot to single user and be able
to repair a existing FreeBSD installation so put them directly into
FreeBSD-runtime.

Reviewed by: bapt, gjb
Differential Revision: https://reviews.freebsd.org/D21503

Stop exporting __pw_scan and __pw_initpwd as freebsd-private libc functions
for use in libutil, and instead compile the small amount of common code
directly into libutil with a .PATH reachover.

Discussed with: kib@

Use __SCCSID() for SCCS IDs.

- Define NO__SCCSID in CFLAGS to preserve existing behavior of omitting
SCCS IDs by default.
- While here, fix the $FreeBSD$ in pw_util.c to use __FBSDID.

Introduce the flopenat(3) function.

Reviewed by: des, emaste
Differential Revision: https://reviews.freebsd.org/D11690

Add HAS_TESTS to all Makefiles that are currently using the
`SUBDIR.${MK_TESTS}+= tests` idiom.

This is a follow up to r321912.

Convert traditional ${MK_TESTS} conditional idiom for including test
directories to SUBDIR.${MK_TESTS} idiom

This is being done to pave the way for future work (and homogenity) in
^/projects/make-check-sandbox .

No functional change intended.

MFC after: 1 weeks

Correct MLINKS for sbuf_hexdump(9)

sbuf_hexdump(9) should be linked to sbuf(9), not hexdump(3). Another
review will be posted to deduplicate the sbuf_hexdump reference in
in hexdump(3) or at the very least make the information less duplicative.

MFC after: 1 week
X-MFC with: r313437
Sponsored by: Dell EMC Isilon

Clean up trailing and leading whitespace for variables to make it
consistent with the rest of the file and style.Makefile(9) a bit
more

MFC after: 3 weeks
Sponsored by: Dell EMC Isilon

Create link from hexdump(3) to sbuf_hexdump(9) as the manpage describes
sbuf_hexdump(9)'s behavior

MFC after: 3 weeks
Sponsored by: Dell EMC Isilon

Use SRCTOP-relative paths to other directories instead of .CURDIR-relative ones

This implifies pathing in make/displayed output

MFC after: 3 weeks
Sponsored by: Dell EMC Isilon

First pass through library packaging.

Sponsored by: The FreeBSD Foundation

Revert r284417 it is not necessary anymore

Enforce overwritting SHLIBDIR

Since METAMODE has been added, sys.mk loads bsd.mkopt.mk which ends load loading
bsd.own.mk which then defines SHLIBDIR before all the Makefile.inc everywhere.

This makes /lib being populated again.

Reported by: many

List kinfo_getfile.c only once.

MFC after: 3 days
Sponsored by: DARPA, AFRL

revert r283969,283970 not needed anymore after r283981

Add a pw_mkdb2(3) function which does the same thing as pw_mkdb(3) except
it takes a new argument allowing to specify the endianness of the database
to generate

Differential Revision: https://reviews.freebsd.org/D2730
Reviewed by: ian

Export a list of VM objects in the system via a sysctl. The list can be
examined via 'vmstat -o'. It can be used to determine which files are
using physical pages of memory and how much each is using.

Differential Revision: https://reviews.freebsd.org/D2277
Reviewed by: alc, kib
MFC after: 2 weeks
Sponsored by: Norse Corp, Inc. (forward porting to HEAD/10)

Check in first src/tests snapshot from NetBSD anoncvs

Sources were obtained like so:

% export CVSROOT="anoncvs@anoncvs.NetBSD.org:/cvsroot"
% cvs -z9 co -D "09/30/2014 20:45" -P src/tests
% mv src/tests/* tests/dist/.

'*CVS*' has been added to svn:ignore to ease updating periodically from
upstream

Some line ending issues had to be resolved with test outputs and scripts
via dos2unix and by deleting the eol-style property set in usr.bin/sort

Discussed with: rpaulo
Sponsored by: EMC / Isilon Storage Division

Integrate lib/libutil into the build/kyua

Remove the .t wrappers

Rename all of the TAP test applications from test-<test> to
<test>_test to match the convention described in the TestSuite
wiki page

humanize_number_test.c:

- Fix -Wformat warnings with counter variables
- Fix minor style(9) issues:
-- Header sorting
-- Variable declaration alignment/sorting in main(..)
-- Fit the lines in <80 columns
- Fix an off by one index error in the testcase output [*]
- Remove unnecessary `extern char * optarg;` (this is already provided by
unistd.h)

Phabric: D555
Approved by: jmmv (mentor)
MFC after: 2 weeks
Obtained from: EMC / Isilon Storage Division [*]
Submitted by: Casey Peel <cpeel@isilon.com> [*]
Sponsored by: EMC / Isilon Storage Division

Use src.opts.mk in preference to bsd.own.mk except where we need stuff
from the latter.

Document the pw_util(3) functions

Reviewed by: des, gjb

Finally nuke auth.conf, nine years after it was deprecated. The only
thing it was still used for was to set the "global default" password
hash. Since the stock auth.conf contained nothing but comments, the
global default was actually the first algorithm in crypt(3)'s list,
which happens to be DES; I take the fact that nobody noticed as proof
that it was not used outside of crypt(3).

The only other use in our tree was in the Kerberos support code in
in tinyware's passwd(1). I removed that code in an earlier commit;
it would not have compiled anyway, as it only supported Kerberos IV.

The auth_getval() function is now a stub that always returns NULL,
which has the same effect as a functional auth_getval() with an
empty auth.conf.

MFC after: 3 weeks

Fix disorder in MAN and MLINKS lists.

Requested by bde.

Add link for pidfile_fileno.3 to pidfile.3 - thanks again pjd!

- Commit work from libprocstat project. These patches add support for runtime
file and processes information retrieval from the running kernel via sysctl
in the form of new library, libprocstat. The library also supports KVM backend
for analyzing memory crash dumps. Both procstat(1) and fstat(1) utilities have
been modified to take advantage of the library (as the bonus point the fstat(1)
utility no longer need superuser privileges to operate), and the procstat(1)
utility is now able to display information from memory dumps as well.

The newly introduced fuser(1) utility also uses this library and able to operate
via sysctl and kvm backends.

The library is by no means complete (e.g. KVM backend is missing vnode name
resolution routines, and there're no manpages for the library itself) so I
plan to improve it further. I'm commiting it so it will get wider exposure
and review.

We won't be able to MFC this work as it relies on changes in HEAD, which
was introduced some time ago, that break kernel ABI. OTOH we may be able
to merge the library with KVM backend if we really need it there.

Discussed with: rwatson

A new jail(8) with a configuration file, to replace the work currently done
by /etc/rc.d/jail.

Create the altix project branch. The altix project will add support
for the SGI Altix 350 to FreeBSD/ia64. The hardware used for porting
is a two-module system, consisting of a base compute module and a
CPU expansion module. SGI's NUMAFlex architecture can be an excellent
platform to test CPU affinity and NUMA-aware features in FreeBSD.

Remove login(3), logout(3) and logwtmp(3) from libutil.

These functions only apply to utmp(5). They cannot be kept intact when
moving towards utmpx. The login(3) function would break, because its
argument is an utmp structure. The logout(3) and logwtmp(3) functions
cannot be used, since they provide a functionality which partially
overlaps.

Increment SHLIB_MAJOR to 9 to indicate the removal.

Build lib/ with WARNS=6 by default.

Similar to libexec/, do the same with lib/. Make WARNS=6 the norm and
lower it when needed.

I'm setting WARNS?=0 for secure/. It seems secure/ includes the
Makefile.inc provided by lib/. I'm not going to touch that directory.
Most of the code there is contributed anyway.

Further extend the quotafile API.

Bump the version of all non-symbol-versioned shared libraries in
preparation for 8.0-RELEASE. Add the previous version of those
libraries to ObsoleteFiles.inc and bump __FreeBSD_Version.

Reviewed by: kib
Approved by: re (rwatson)

WIP

Add filler man pages for the kinfo functions I added recently.
While here, hook up the hexdump(3) man page which wasn't being installed.

Merge user/peter/kinfo branch as of r185547 into head.

This changes struct kinfo_filedesc and kinfo_vmentry such that they are
same on both 32 and 64 bit platforms like i386/amd64 and won't require
sysctl wrapping.

Two new OIDs are assigned. The old ones are available under
COMPAT_FREEBSD7 - but it isn't that simple. The superceded interface
was never actually released on 7.x.

The other main change is to pack the data passed to userland via the
sysctl. kf_structsize and kve_structsize are reduced for the copyout.
If you have a process with 100,000+ sockets open, the unpacked records
require a 132MB+ copyout. With packing, it is "only" ~35MB. (Still
seriously unpleasant, but not quite as devastating). A similar problem
exists for the vmentry structure - have lots and lots of shared libraries
and small mmaps and its copyout gets expensive too.

My immediate problem is valgrind. It traditionally achieves this
functionality by parsing procfs output, in a packed format. Secondly, when
tracing 32 bit binaries on amd64 under valgrind, it uses a cross compiled
32 bit binary which ran directly into the differing data structures in 32
vs 64 bit mode. (valgrind uses this to track file descriptor operations
and this therefore affected every single 32 bit binary)

I've added two utility functions to libutil to unpack the structures into
a fixed record length and to make it a little more convenient to use.

Fixed style issues with variable ordering and naming, spacing and
parentheses.

Fixed alignment issue in gr_dup() in its assignment of gr_mem using a
struct to force alignment without performing alignment mathematics. This
was noticed recently with libutil was built with WARNS=6 on platform such
as sparc64.

Added checks to gr_dup(), gr_equal() and gr_make() to prevent segfaults
when examining struct group's with the struct members pointing to NULL's.

With fix of alignment issue, restore WARNS?=6.

Reviewed by: des
MFC after: 1 week

Comment out WARNS. There are too many alignment issues in libutil.

Disconnect gr_util.c from the build. It isn't documented or used anywhere
in the tree, and due to unsafe pointer arithmetic, it will most likely crash
on architectures with strict alignment requirements.

MFH (r184633, r184635): build at WARNS level 6

libutil now builds at WARNS level 6.

MFC after: 3 weeks

- Import the HEAD csup code which is the basis for the cvsmode work.

Merge hexdump(9) to userland as hexdump(3) in libutil. I'm tired of doing
this by hand in userland utilities.

MFC after: 1 month

Add four utility functions related to struct grp processing modeled in-part
after similar calls related to struct pwd in libutil/pw_util.c:
- gr_equal()
Perform a deep comparison of two struct grp's. It does a thorough, yet
unoptimized comparison of all the members regardless of order.

- gr_make()
Create a string (see group(5)) from a struct grp.

- gr_dup()
Duplicate a struct grp. Returns a value that is a single contiguous
block of memory.

- gr_scan()
Create a struct grp from a string (as produced by gr_make()).

MFC after: 3 weeks

Implement expand_number(3), which is the opposite of humanize_number(3), ie.
a number in human-readable form is converted to int64_t, for example:
123b -> 123
10k -> 10240
16G -> 17179869184

First version submitted by: Eric Anderson <anderson@freebsd.org>
Approved by: re (bmah)

Bump library versions in preparation for 7.0.

Ok'd by: kan

I'm tired of seeing this done incorrectly and non-portably, so add a
flopen(3) function which reliably opens and locks a file.

MFC after: 3 weeks

style.Makefile(5) is good for our eyes.

Specify default path for SHLIBDIR before bsd.own.mk does.
This fix shared library installed correct place.

Don't build IPv6 support if we have choosen not to have it.

Bump library majro version for gethostbyaddr(3).

Add utility functions for checking if a given kernel module is loaded,
and loading it.

Add a family of functions for reliable pidfiles handling.

Idea from: jmg
Discussed on: arch@

Bump the shared library version number of all libraries that have not
been bumped since RELENG_5.

Reviewed by: ru
Approved by: re (not needed for commit check but in principle...)

Add humanize_number(3) to libutil for formating numbers into a human
readable form.

Obtained from: NetBSD

Back out the "clean_environment()" function from libutil.

Further contemplation has convinced me that this was
not going to really solve the problem of environment-poisoning
without raising serious administrative headaches. There
must be a better way...

Add a clean_environment call to libutil.

This function removes all environment variables except
the ones listed on a "whitelist."

The function accepts two whitelist arguments.
If the first is NULL, a built-in default list will be
used. This allows callers to get a variety of behaviors:
* Default screening: provide NULL for both lists
* Custom screening: provide a custom list for the first argument
* Modified default screening: provide NULL for first arg,
list of additional variables to preserve in the second arg

Idea from: Jacques Vidrine

MFC after: 2 weeks

Bump the major version on libtuil. libutil now relies on the mac_*
symbols exported by newer versions of libc, and so we want applications
depending on the newer library code to be required to link against the
newer libc.

Discussed with: scottl, kris, imp

Remove a GCC specifig CFLAG. We should be using WARNS=? for this.
WARNS=? is not added here at this point, because I've not tested
it on enough platforms, and I don't want to break builds.

style.Makefile(5)

Stage 3 of dynamic root support. Make all the libraries needed to run
binaries in /bin and /sbin installed in /lib. Only the versioned files
reside in /lib, the .so symlink continues to live /usr/lib so the
toolchain doesn't need to be modified.

Zap now-unused SHLIB_MINOR

Backout previous delta (addition of -I${.CURDIR}/../../sys).

Submitted by: bde

Add -I${.CURDIR}/../../sys into CFLAGS, which should fix the world broken
by RLIMIT_VMEM addition.

Add passwd manipulation code based on parts of vipw and chpass.

Sponsored by: DARPA, NAI Labs

o Reflect repo-copy of extattr.[c3] from libutil to libc, moving
extattr namespace routines to the libc/posix1e directory. While
the extattr calls are not strictly POSIX.1e, POSIX.1e wasn't
strictly ever approved, so I think that's OK.

Obtained from: TrustedBSD Project

remove emalloc,ecalloc,erealloc,estrdup

add ecalloc, emalloc, erealloc, estrdup - versions of the e-less
functions that exit instead of failing

Add a manual page for extattr_string_to_namespace and
extattr_namespace_to_string.

Reviewed by: rwatson

Removed -I${.CURDIR}/.../sys from CFLAGS.

MAN[1-9] -> MAN.

o To support new EA interface with explicit namespaces, introduce two
utility functions which convert between string namespace names and
numeric constants used by the interface. Right now, two namespaces
are supported, EXTATTR_NAMESPACE_SYSTEM ("system") and
EXTATTR_NAMESPACE_USER ("user"). These functions are used by
various userland EA utilities, rather than hard coding the routines
all over the place.

Obtained from: TrustedBSD Project

Move trimdomain() into it's own source file and tidy things up a bit.
Fix disorder in the Makefile.

Reviewed (mostly) by: bde

Fix problems people were having with large -O levels with GCC and
getting libutil/libcrypt to work properly. I've determined that GCC
thinks it can inline all functions, including weak-symboled ones, if
it feels like it.

Create a new stub.c and move any stubs there to prevent inlining.
Thanks to jdp and William S. Duncanson for helping me finally find the
problem.

Move setproctitle() from libutil to libc (after a repo-copy)
and bump __FreeBSD_version to 500012 to mark the occasion.

setproctitle() is prototyped in unistd.h as opposed to stdlib.h
where OpenBSD and NetBSD have it.

Reviewed by: peter

Stick login_setcryptfmt() in its own file to make pulling in of
-lcrypt only happen if truly necessary.

Add working and easy crypt(3)-switching. Yes, we need a whole new API
for crypt(3) by now. In any case:

Add crypt_set_format(3) + documentation to -lcrypt.
Add login_setcryptfmt(3) + documentation to -lutil.
Support for switching crypt formats in passwd(8).
Support for switching crypt formats in pw(8).

The simple synopsis is:
edit login.conf; add a passwd_format field set to "des" or "md5"; go nuts :)

Reviewed by: peter

Update major version.

Now libutil depends on libc.so.4, so needs to update the major version.
Without this, old binaries which use libutil and build with libc.so.3
will coredump on recent 4.0.

Solicited comment for cvs-committers and there seems to be no objection.

Approved by: jkh

Historically file flags (schg, uschg, etc) have been converted from
string to u_long and back using two functions, flags_to_string and
string_to_flags, which co-existed with 'ls'. As time has progressed
more and more other tools have used these private functions to
manipulate the file flags.

Recently I moved these functions from /usr/src/bin/ls to libutil,
but after some discussion with bde it's been decided that they
really ought to go in libc.

There are two already existing libc functions for manipulating file
modes: setmode and getmode. In keeping with these flags_to_string
has been renamed getflags and string_to_flags to setflags.

The manual page could probably be improved upon ;)

several tcp apps IPv6 update
-inetd
-rshd
-rlogind
-telnetd
-rsh
-rlogin

Reviewed by: freebsd-arch, cvs-committers
Obtained from: KAME project

Replace beforeinstall target with new variables used by .mk system.

Reviewed by: marcel, and make world

Moved flags_to_string and string_to_flags into libutil. It's used in
many places nowadays.

Connect fparseln(3) for mailwrapper(8)

Link manual page for login_getpwclass(3) to login_cap(3).

PR: docs/14673
Submitted by: Andrew <andrew@ugh.net.au>

Axe LOGIN_CAP_AUTH.

PR: 10115
Reported by: Gene Skonicki <gene@cif.rochester.edu>
Requested by: jdp

Fix commented out CFLAGS addition for LOGIN_CAP_AUTH, which was missing
a make -D option.

PR: 12591
Submitted by: Craig Leres <leres@ee.lbl.gov>

add MLINKS for two functions used from login_auth.c
comment out unused functions from login_auth.3

Tidy up trimdomain() and document it.
Don't truncate one byte short of the passed length.

Add realhostname() - a function to correctly lookup
a name by address and ensure that the name resolves
back to the original address.

Add some rudimentary documentation for my new functions.

Add a simple mechanism for reading property lists from files (which
I'll convert sysinstall to use shortly) and a simple call which uses
this mechanism to implement an /etc/auth.conf file. I'll let Mark Murray
handle the format and checkin of the sample auth.conf file.
Reviewed by: markm

Add uu_lock_txfr() to transfer ownership of a successful
uu_lock() to another process.

MF22: add login_auth.3 to man page list.

Remove login_progok()
Suggested by: guido

Add full support for determining if a user
is restricted from running a given program.

-I${DESTDIR}/sys -> -I${.CURDIR}/../../sys.

Add appropriate ${DESTDIR} in front of absolute paths.

Back out previous revision. Shlib version numbers are supposed to be
bumped only 0.1 or 1.0 between releases. (See handbook.)

Note that if you have built world in -current in the last 48 hours or
so, you should manually remove /usr/lib/libutil.so.2.3 before
rebuilding world to cleanse your system.

Bump shared lib version to 2.3.

Suggested by: bde

Summary of login.conf support changes:

o Incorporated BSDI code and enhancements, better logging for error
checking (which has been shown to be a problem, and is therefore
justified, imho); also some minor things we were missing, including
better quad_t math, which checks for under/overflows.

o setusercontext() now allows user resource limit overrides, but
does this AFTER dropping root privs, to restrict the user to
droping hard limits and set soft limits within the kernel's
allowed user limits.

o umask() only set once, and only if requested.

o add _secure_path(), and use in login.conf to guard against
symlinks etc. and non-root owned or non-user owned files being
used. Derived from BSDI contributed code.

o revamped authentication code to BSDI's latest api, which
includes deleting authenticate() and adding auth_check()
and a few other functions. This is still marked as depecated
in BSDI, but is included for completeness. No other source
in the tree uses this anyway, so it is now bracketed with
#ifdef LOGIN_CAP_AUTH which is by default not defined. Only
auth_checknologin() and auth_cat() are actually used in
module login_auth.c.

o AUTH_NONE definition removed (collided with other includes
in the tree). [bde]

o BSDI's login_getclass() now accepts a char *classname
parameter rather than struct passwd *pwd. We now do likewise,
but added login_getpwclass() for (sort of) backwards
compatiblity, namely because we handle root as a special
case for the default class. This will require quite a few
changes elsewhere in the source tree.

o We no longer pretend to support rlim_t as a long type.

o Revised code formatting to be more bsd-ish style.

Move login_cap.h from src/include for easier maintenance with
related files.

MLINK uu_lockerr(3).

Move uucplock into libutil and create a manual page.

Added -Wall to CFLAGS, cleaned up (all avoidable) warnings.

Bump libutil.so version (2.1 -> 2.2) since a whole heap of new functions
were added with the login class stuff. This is needed since libutil.so.2.1
is what is used in RELENG_2_2 and well into the release cycle. We only
bump once per release cycle as needed.

Library functions relating to the login class capabilities database,
including manpages.
See also login_cap.h.

Finally document the interfaces found in libutil. While being here,
also add the missing declaration of forkpty() to libutil.h.

Btw., the calling interface for login(3) is crude. Some better
abstraction is needed, perhaps similar to logwtmp(3).

2.2 candidate, but i'll wait for the spelling police first. :)

Bump libutil revision after recent addition of setproctitle().

Install (optional) libutil.h with prototypes for the functions and
document this in the man page.

minor cleanups to the various routines, include the prototype file, declare
return codes etc.

Bring in an initial version of setproctitle().. This is intended to
replace the dozen other various hacks in the code that do all sorts
of crude things including spamming the envrionment strings with the new
argv string.

This version is mainly inspired by the sendmail version, with a couple of
ideas taken from the NetBSD implementation as well.

BSD 4.4 Lite Lib Sources