makefs: Make it possible to silence warnings about duplicate paths

When generating a VM image from an installworld mtree manifest, makefs
spits out several thousand warnings about duplicate paths in the
manifest. These are harmless and have been around for a long time (see
the phabricator revision for some more details), so let's at least have
a way to make makefs quieter.

Reviewed by: brooks, imp, emaste
MFC after: 2 weeks
Differential Revision: https://reviews.freebsd.org/D43513

usr.sbin: Automated cleanup of cdefs and other formatting

Apply the following automated changes to try to eliminate
no-longer-needed sys/cdefs.h includes as well as now-empty
blank lines in a row.

Remove /^#if.*\n#endif.*\n#include\s+<sys/cdefs.h>.*\n/
Remove /\n+#include\s+<sys/cdefs.h>.*\n+#if.*\n#endif.*\n+/
Remove /\n+#if.*\n#endif.*\n+/
Remove /^#if.*\n#endif.*\n/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/types.h>/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/param.h>/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/capsicum.h>/

Sponsored by: Netflix

makefs: Handle missing link keyword in mtree spec

PR: 241178
Reviewed by: emaste
MFC after: 3 days
Pull Request: https://github.com/freebsd/freebsd-src/pull/850

Remove $FreeBSD$: one-line .c pattern

Remove /^[\s*]*__FBSDID\("\$FreeBSD\$"\);?\s*\n/

makefs: Fix typos

Signed-off-by: Elyes Haouas <ehaouas@noos.fr>
Reviewed by: imp
Pull Request: https://github.com/freebsd/freebsd-src/pull/653

spdx: The BSD-2-Clause-FreeBSD identifier is obsolete, drop -FreeBSD

The SPDX folks have obsoleted the BSD-2-Clause-FreeBSD identifier. Catch
up to that fact and revert to their recommended match of BSD-2-Clause.

Discussed with: pfg
MFC After: 3 days
Sponsored by: Netflix

makefs: Ignore the "tags" keyword in mtree manifests

An install using -DNO_ROOT emits mtree entries containing tags used by
pkgbase. makefs(8) can safely ignore them, so do that rather than
emitting a warning for each entry.

Reviewed by: brooks, imp
MFC after: 2 weeks
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D29384

usr.sbin/makefs: fix use-after-free in read_mtree_keywords()

The st variable is used as a shortcut for &node->inode->st, but in one
branch just before the exit we update node->inode without changing st.

Reported by: AddressSanitizer
Reviewed By: emaste
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D28570

makefs: Fix "time" mtree attribute handling

When processing mtree(5) MANIFEST files, makefs(8) previously threw an
error if it encountered an entry whose "time" attribute contained a
non-zero subsecond component (e.g. time=1551620152.987220000).

Update the handling logic to properly assign the subsecond component if
built with nanosecond support, or silently discard it otherwise.

Also, re-enable the time attribute for the kyua tests.

PR: 194703
Submitted by: Mitchell Horne <mhorne063@gmail.com>
Differential Revision: https://reviews.freebsd.org/D19627

makefs: Use ENODATA instead of ENOMSG as a translation for missing ENOATTR.

This is consistent with what some linux filesystems do and has been
adopted in our linuxulator.

MFC after: 3 days

Allow xinstall and makefs to be crossbuilt on Linux and Mac

I need these tools in order to install the crossbuilt FreeBSD and create a
disk image. Linux does not have a st_flags in struct stat so unfortunately
I need a bunch of ugly ifdefs. The resulting binaries allow me to
sucessfully install a MIPS64 world and create a disk-image that boots.

Reviewed By: brooks, bdrewery, emaste
Approved By: jhb (mentor)
Differential Revision: https://reviews.freebsd.org/D13307

various: general adoption of SPDX licensing ID tags.

Mainly focus on files that use BSD 2-Clause license, however the tool I
was using misidentified many licenses so this was mostly a manual - error
prone - task.

The Software Package Data Exchange (SPDX) group provides a specification
to make it easier for automated tools to detect and summarize well known
opensource licenses. We are gradually adopting the specification, noting
that the tags are considered only advisory and do not, in any way,
superceed or replace the license texts.

No functional change intended.

Replace makefs' hand-rolled unescaping with strunvis

mtree path names and link attributes are encoded, generally using strvis. Newer
versions of mtree will use C-style escapes but previously the accepted form was
octal escapes. makefs' mtree code spots the C-style escapes but fails to deal
with octal escapes correctly.

Remove mtree's escape-decoding code (except for a few instances where it's
needed) and instead pass pathnames and link targets through strunvis prior to
use.

Reviewed by: marcel
MFC after: 2 weeks
Sponsored by: Dell EMC Isilon
Differential Revision: https://reviews.freebsd.org/D12104

makefs: use C standard memcpy/memset in userland

This file does not exist in NetBSD's makefs, but make the chance for
consistency with memcpy/memset used in the rest of makefs.

Sponsored by: The FreeBSD Foundation

makefs: clean up signedness warnings and bump WARNS to 3

Reviewed by: kib
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D10650

makefs: use emalloc and friends

The emalloc set of error-checking memory allocation routines were added
to libnetbsd in r316572. Use them in makefs to reduce differences with
NetBSD.

NetBSD revs:
cd9660.c 1.39
ffs.c 1.56
makefs.c 1.42
walk.c 1.27
cd9660/cd9660_archimedes.c 1.2
cd9660/cd9660_eltorito.c 1.20
cd9660/cd9660_write.c 1.16
cd9660/iso9660_rrip.c 1.12
ffs/buf.c 1.17
ffs/mkfs.c 1.26

Obtained from: NetBSD

Be more precise when including headers so that we're less likely to
depend on namespace pollution and as such become more portable. This
means including headers like <sys/types.h> or <stdlib.h>, but also
making sure we include system/host headers before local headers.

While here: define ENOATTR as ENOMSG in mtree.c. There is no ENOATTR
on Linux.

With this, makefs is ready for compilation on macOS and Linux.

Fix an off by one error to avoid overflowing rp[].

Reported by: Coverity
CID: 1007579

Don't leak 'var'.

Reported by: clang static analyzer

Timestamps may not have a decimal point.
While here, consistently use sbuf_new_auto().

Allow comments at end of line.

Reviewed by: marcel

Set st_nlink in the stat structure within the inode to 1 as well.
The cd9660 file system uses that field for the link count and it
was 0. This impacts pwd_mkdb(8) as it checks for st_nlink not being
0 as part of closing a race.

Support hardlinks in manifest files by the same logic as the treewalk
code.

Reviewed by: marcel
Sponsored by: DARPA, AFRL

Allow '.' components in manifest paths. They are always the first
component of mtree -C and install -M output and are easily skipped.

Reviewed by: marcel
Sponsored by: DARPA, AFRL

Fix the -N option in manifest mode by using pwcache(3). This also
speeds up image creation appreciably.

Reviewed by: marcel
Sponsored by: DARPA, AFRL

Add a -D flag that causes duplicate entries in an mtree manifest to be
treated as warnings rather than errors.

Reviewed by: marcel
Sponsored by: DARPA, AFRL

If no contents keyword is specified, the default for files is
the named file.

Approved by: marcel (mentor)

Add support for using mtree(5) manifest files to define the image
to be created. The support is based on mtree version 2.0, as used
in libarchive, but adds new features on top of it.

The current implementation is fully functional, but is envisioned
to grow at least the following additional features over time:
o Add support for the /include special command so that manifest
files can be constructed using includable fragments.
o Add support specifying a search path to locate content files.
o Content file filters: commands that provide file contents on
stdout.

The manifest file eliminates the need to first construct a tree
as root in order to create an image and allows images (releases)
to be created directly from object trees and/or source trees.

Reviewed by: deo
Sponsored by: Juniper Networks, Inc