makefs: Make it possible to silence warnings about duplicate paths

When generating a VM image from an installworld mtree manifest, makefs
spits out several thousand warnings about duplicate paths in the
manifest. These are harmless and have been around for a long time (see
the phabricator revision for some more details), so let's at least have
a way to make makefs quieter.

Reviewed by: brooks, imp, emaste
MFC after: 2 weeks
Differential Revision: https://reviews.freebsd.org/D43513

usr.sbin: Automated cleanup of cdefs and other formatting

Apply the following automated changes to try to eliminate
no-longer-needed sys/cdefs.h includes as well as now-empty
blank lines in a row.

Remove /^#if.*\n#endif.*\n#include\s+<sys/cdefs.h>.*\n/
Remove /\n+#include\s+<sys/cdefs.h>.*\n+#if.*\n#endif.*\n+/
Remove /\n+#if.*\n#endif.*\n+/
Remove /^#if.*\n#endif.*\n/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/types.h>/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/param.h>/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/capsicum.h>/

Sponsored by: Netflix

Remove $FreeBSD$: one-line .c pattern

Remove /^[\s*]*__FBSDID\("\$FreeBSD\$"\);?\s*\n/

makefs: remove vestigial '?' cases from top-level getopt(3) loop

getopt(3) returns '?' when it encounters a flag not present in the in
the optstring or if a flag is missing its option argument. We can
handle this case with the "default" failure case with no loss of
legibility.

Obtained from: OpenBSD makefs.c 1.22

makefs: whitespace cleanup (remove space before tab)

MFC after: 1 week

makefs: Add ZFS support

This allows one to take a staged directory tree and create a file
consisting of a ZFS pool with one or more datasets that contain the
contents of the directory tree. This is useful for creating virtual
machine images without using the kernel to create a pool; "zpool create"
requires root privileges and currently is not permitted in jails.
makefs -t zfs also provides reproducible images by using a fixed seed
for pseudo-random number generation, used for generating GUIDs and hash
salts. makefs -t zfs requires relatively little by way of machine
resources.

The "zpool_reguid" rc.conf setting can be used to ask a FreeBSD guest to
generate a unique pool GUID upon first boot.

A small number of pool and dataset properties are supported. The pool
is backed by a single disk vdev. Data is always checksummed using
Fletcher-4, no redundant copies are made, and no compression is used.
The manual page documents supported pool and filesystem properties.

The implementation uses a few pieces of ZFS support from with the boot
loader, especially definitions for various on-disk structures, but is
otherwise standalone and in particular doesn't depend on OpenZFS.

This feature should be treated as experimental for now, i.e., important
data shouldn't be trusted to a makefs-created pool, and the command-line
interface is subject to change.

Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D35248

makefs: add msdosfs (FAT) support

Add FAT support to makefs by copying some files from sys/fs/msdosfs/ and
updating others with changes from NetBSD.

The six files copied from sys/fs/msdosfs at r348251 and modified are:
denode.h direntry.h fat.h msdosfs_fat.c msdosfs_lookup.c msdosfsmount.h

I would prefer to avoid the duplication, but reluctance to doing so was
expressed in a previous review (D11197); for now copy the files and
revisit in the future.

Submitted by: Siva Mahadevan
Discussed with: cem, imp
MFC after: 1 month
Relnotes: Yes
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D16438

various: general adoption of SPDX licensing ID tags.

Mainly focus on files that use BSD 2-Clause license, however the tool I
was using misidentified many licenses so this was mostly a manual - error
prone - task.

The Software Package Data Exchange (SPDX) group provides a specification
to make it easier for automated tools to detect and summarize well known
opensource licenses. We are gradually adopting the specification, noting
that the tags are considered only advisory and do not, in any way,
superceed or replace the license texts.

No functional change intended.

makefs: add -O (offset) option

NetBSD revs:
ffs.c 1.60
makefs.8 1.44
makefs.c 1.48
makefs.h 1.33
ffs/buf.c 1.20
ffs/mkfs.c 1.27

Obtained from: NetBSD
Relnotes: Yes
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D10780

makefs: clean up warnings

- make functions and variables static where appropriate
- use const char * where appropriate
- remove unused variables

Sponsored by: The FreeBSD Foundation

makefs: use emalloc and friends

The emalloc set of error-checking memory allocation routines were added
to libnetbsd in r316572. Use them in makefs to reduce differences with
NetBSD.

NetBSD revs:
cd9660.c 1.39
ffs.c 1.56
makefs.c 1.42
walk.c 1.27
cd9660/cd9660_archimedes.c 1.2
cd9660/cd9660_eltorito.c 1.20
cd9660/cd9660_write.c 1.16
cd9660/iso9660_rrip.c 1.12
ffs/buf.c 1.17
ffs/mkfs.c 1.26

Obtained from: NetBSD

makefs: sync with NetBSD

This is a collection of minor changes as diff reduction against NetBSD.

NetBSD revs:
cd9660.c 1.39
cd9660.h 1.19
makefs.c 1.34

Obtained from: NetBSD
Sponsored by: The FreeBSD Foundation

makefs: sync option parsing with NetBSD

- add support for parsing different types; not just int
- homogenize option parsing
- fix single letter parsing
- remove duplicated code

NetBSD revisions:
cd9660.c 1.36 1.37 1.38 1.41 1.42 1.43
ffs.c 1.50 1.51 1.52 1.53 1.56 1.57
makefs.c 1.36 1.37 1.38 1.39 1.40 1.42 1.43 1.44 1.46
makefs.h 1.28 1.29 1.31 1.32

Obtained from: NetBSD
Sponsored by: The FreeBSD Foundation

makefs: sync -T timestamp with NetBSD

NetBSD revision:
makefs.c 1.51

Obtained from: NetBSD

makefs: reorder 'usage' alphabetically

From NetBSD, Mon Aug 15 14:45:01 2011 +0000 (wiz)

Re-order `usage' alphabetically;
rename option arguments in the manpage's `SYNOPSIS' section to
match those from `usage' (not the other way around; the `usage'-line
(and other parts of makefs.c) contain the correct names);
minor punctuation improvements.

From Snader_LB.

makefs.8 1.36
makefs.c 1.30

Obtained from: NetBSD

makefs: Provide a -T option to set timestamps to a consistent value

This is taken from the NetBSD versions listed below and adapted to the
makefs version in FreeBSD, along with a bug fix from cem@ that will be
sent to NetBSD.

Reviewed by: pfg
Approved by: re (gjb)
Obtained from: NetBSD
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D6835

Fix getopt(3) argument after r290180; I forgot to change -r to -R
by accident

MFC after: 3 days
Pointyhat to: ngie
Reported by: vangyzen
Sponsored by: EMC / Isilon Storage Division

makefs: use ENTRY macro for diff reduction with NetBSD

Sponsored by: The FreeBSD Foundation

Follow up to roundup feature addition in r289203

- Rename -r to -R to avoid the clash with makefs -r in NetBSD
- Note that -R is an FFS-specific option because it's not implemented
in cd9660 today
- Rename the roundup variable to "roundup-size" in the manpage and help
text for consistency with other variables.
- Bump .Dd (missed in r289203)

PR: 203707
MFC after: 1 week
X-MFC with: r289203
Differential Revision: https://reviews.freebsd.org/D3959
Reviewed by: adrian (earlier patch), emaste
Sponsored by: EMC / Isilon Storage Division

Revert r289694

I committed some other undesirable local changes by accident

Add some rudimentary [smoke] testcases for makefs

MFC after: 2 weeks
Sponsored by: EMC / Isilon Storage Division

makefs: introduce a new option to specify what to round the resulting
image up to.

From ticket:

While trying to run FreeBSD/mips on some device having very small flash media,
one is forced to compress file system with mkulzma(8) utility. It is desirable
to specify small UFS block/fragment sizes like 4096/512 bytes for makefs(8)
and big compression block size like 65535 bytes to mkulzma at the same time.
Then one obtains very good comression ratios (like 75% and more) but faces
the following problem.

geom_uncompress kernel module reports GEOM provider size rounded up to its
compression block size. Generally, this changes original media size and now
it fails to match the size of embedded UFS file system that leads to other
problems, f.e. geom_label kernel module does not like this and skips the
file system while tasting the GEOM and looking for UFS label.

This makes it impossible to refer to the file system using known UFS label
instead of something like /dev/map/rootfs.uncompress.

The following patch introduces new command line option "-r roundup" for makefs
that makes it round up the image to specified block size. Hence, geom_uncompress
does not change GEOM media size for images rounded that way and geom_label
accepts such GEOMs just fine.

With the patch applied, one can use following commands:

$ makefs -t ffs -r 65536 -o bsize=4096,fsize=512,label=flash optimization=space fs.img fs
$ mkulzma -s 65536 -o fs.img.ulzma fs.img

PR: bin/203707
Submitted by: <eugen@grosbein.net>

Mark the makefs(8) '-p' flag as deprecated in preference for
the '-Z' flag for compatibility with NetBSD.

Submitted by: Eric van Gyzen (via stable@)
MFC after: 3 days

Add a -D flag that causes duplicate entries in an mtree manifest to be
treated as warnings rather than errors.

Reviewed by: marcel
Sponsored by: DARPA, AFRL

Sort options.

Add -p flag to create the image as a sparse file.

Submitted by: Shesha Sreenivasamurthy
PR: bin/167779

Allow contents of multiple directories to be merged to the current image.
Note this patch was submitted to NetBSD and they already adopted it.

http://mail-index.netbsd.org/source-changes/2012/01/28/msg031078.html

MFC after: 1 week

Add support for using mtree(5) manifest files to define the image
to be created. The support is based on mtree version 2.0, as used
in libarchive, but adds new features on top of it.

The current implementation is fully functional, but is envisioned
to grow at least the following additional features over time:
o Add support for the /include special command so that manifest
files can be constructed using includable fragments.
o Add support specifying a search path to locate content files.
o Content file filters: commands that provide file contents on
stdout.

The manifest file eliminates the need to first construct a tree
as root in order to create an image and allows images (releases)
to be created directly from object trees and/or source trees.

Reviewed by: deo
Sponsored by: Juniper Networks, Inc

The dump, fsck_ffs, fsdb, fsirand, newfs, makefs, and quot utilities
include sys/time.h instead of time.h. This include is incorrect as
per the manpages for the APIs and the POSIX definitions. This commit
replaces sys/time.h where necessary with time.h.

The commit also includes some minor style(9) header fixup in newfs.

This commit is part of a larger effort by Garrett Cooper started in
//depot/user/gcooper/posix-conformance-work/ -- to make FreeBSD more
POSIX compliant.

Submitted by: Garrett Cooper yanegomi at gmail dot com

Sync with the latest version from NetBSD. It notably addds ISO9660 support.

Submitted by: bapt

A new jail(8) with a configuration file, to replace the work currently done
by /etc/rc.d/jail.

Create the altix project branch. The altix project will add support
for the SGI Altix 350 to FreeBSD/ia64. The hardware used for porting
is a two-module system, consisting of a base compute module and a
CPU expansion module. SGI's NUMAFlex architecture can be an excellent
platform to test CPU affinity and NUMA-aware features in FreeBSD.