makefs: warn that ffs sectorsize other than 512 may not work

newfs always sets sectorsize to DEV_BSIZE (512) and derives some other
values based on the number of 512-byte sectors per real sector. Similar
logic is required in makefs. Until that happens, emit a warning that
the image may be incorrect.

PR: 276571

usr.sbin: Remove ancient SCCS tags.

Remove ancient SCCS tags from the tree, automated scripting, with two
minor fixup to keep things compiling. All the common forms in the tree
were removed with a perl script.

Sponsored by: Netflix

Remove $FreeBSD$: one-line .c pattern

Remove /^[\s*]*__FBSDID\("\$FreeBSD\$"\);?\s*\n/

makefs: call brelse from bread

This matches NetBSD and rationalizes makefs with the kernel API.

This reverts commit 370e009188ba90c3290b1479aa06ec98b66e140a.

Reviewed by: mckusick
Sponsored by: The FreeBSD Foundation
Obtained from: NetBSD 0a62dad69f62, 0c4125e1a19f, cb6a5a3575fd
Differential Revision: https://reviews.freebsd.org/D39070

makefs: do not pass mode to open() call lacking O_CREAT

Obtained from: OpenBSD ffs.c 1.35

makefs: whitespace cleanup (remove space before tab)

MFC after: 1 week

makefs: fix calculation of file sizes

When a new FS image is created we need to calculate how much space each
file is going to consume.
Fix two bugs in that logic:

1) Count the space needed for indirect blocks for large files.
1) Normally the trailing data of a file is written to a block of frag
size, 4 kB by default.

However for files that use indirect blocks a full block is allocated,
32kB by default. Take that into account.

Adjust size calculations to match what is done in ffs_mkfs routine:

* Depending on the UFS version the superblock is stored at a different
offset. Take that into account.
* Add the cylinder group block size.
* All of the above has to be aligned to the block size.

Finally, Remove "ncg" variable. It's always 1 and it was used to
multiply stuff.

PR: 229929
Reviewed by: mckusick
MFC after: 2 weeks
Sponsored by: Semihalf
Submitted by: Kornel Dulęba <mindal@semihalf.com>
Differential Revision: https://reviews.freebsd.org/D35131
Differential Revision: https://reviews.freebsd.org/D35132

makefs: Fix warnings and reset WARNS to the default

Leave -Wcast-align disabled, at least for now, since there are numerous
instances of that warning in places where buffer pointers are cast to
pointers to various filesystem structures. Fixing this properly would
be too much work for too little gain.

MFC after: 2 weeks
Sponsored by: The FreeBSD Foundation

ufs: Rework shortlink handling to avoid subobject overflows

Shortlinks occupy the space of both di_db and di_ib when used. However,
everywhere that wants to read or write a shortlink takes a pointer do
di_db and promptly runs off the end of it into di_ib. This is fine on
most architectures, if a little dodgy. However, on CHERI, the compiler
can optionally restrict the bounds on pointers to subobjects to just
that subobject, in order to mitigate intra-object buffer overflows, and
this is enabled in CheriBSD's pure-capability kernels.

Instead, clean this up by inserting a union such that a new di_shortlink
can be added with the right size and element type, avoiding the need to
cast and allowing the use of the DIP macro to access the field. This
also mirrors how the ext2fs code implements extents support, with the
exact same structure other than having a uint32_t i_data[] instead of a
char di_shortlink[].

Reviewed by: mckusick, jhb
Differential Revision: https://reviews.freebsd.org/D33650

Allocate extra inodes in makefs when leaving free space in UFS images.

By default, makefs(8) has very few spare inodes in its output images,
which is fine for static filesystems, but not so great for VM images
where many more files will be added. Make makefs(8) use the same
default settings as newfs(8) when creating images with free space --
there isn't much point to leaving free space on the image if you
can't put files there. If no free space is requested, use current
behavior of a minimal number of available inodes.

Reviewed by: manu
MFC after: 3 weeks
Differential Revision: https://reviews.freebsd.org/D29492

Remove #define _KERNEL hacks from libprocstat

Make sys/buf.h, sys/pipe.h, sys/fs/devfs/devfs*.h headers usable in
userspace, assuming that the consumer has an idea what it is for.
Unhide more material from sys/mount.h and sys/ufs/ufs/inode.h,
sys/ufs/ufs/ufsmount.h for consumption of userspace tools, with the
same caveat.

Remove unacceptable hack from usr.sbin/makefs which relied on sys/buf.h
being unusable in userspace, where it override struct buf with its own
definition. Instead, provide struct m_buf and struct m_vnode and adapt
code to use local variants.

Reviewed by: mckusick
Tested by: pho
Sponsored by: The FreeBSD Foundation
Differential revision: https://reviews.freebsd.org/D28679

makefs: Also set UFS di_birthtime when building on Linux

Since st_birthtime doesn't exists on Linux (unless you use statx(2)), we
instead populate it with the st_ctime value.

Reviewed By: emaste
Differential Revision: https://reviews.freebsd.org/D22386

Improve error handling: bail out if one of the files scheduled
to go to the FS image we are making cannot be read (e.g. EPERM).
Current behaviour when we issue waring but still proceeed and
return success is definitely not correct: masking out error
condition as well as making a slighly inconsistent FS where
attempt to access the file in question ends up in EBADF. See
linked DR for details.

MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D18584

makefs: use FreeBSD brelse function signature

Although the ffs (and later msdosfs) implementation in makefs is
independent of the one in kernel, it makes sense to keep differences to
a minimum in order to ease comparison and porting changes across.

Submitted by: Siva Mahadevan
Sponsored by: The FreeBSD Foundation

makefs: sync fragment and block size with newfs

r222319 in newfs raised the default blocksize for UFS/FFS filesystems
from 16K to 32K and the default fragment size from 2K to 4K, with a
rationale that most disks were now running with 4K sectors.

MFC after: 2 weeks
Relnotes: Yes
Sponsored by: The FreeBSD Foundation

Allow xinstall and makefs to be crossbuilt on Linux and Mac

I need these tools in order to install the crossbuilt FreeBSD and create a
disk image. Linux does not have a st_flags in struct stat so unfortunately
I need a bunch of ugly ifdefs. The resulting binaries allow me to
sucessfully install a MIPS64 world and create a disk-image that boots.

Reviewed By: brooks, bdrewery, emaste
Approved By: jhb (mentor)
Differential Revision: https://reviews.freebsd.org/D13307

Fix a logic bug in makefs lazy inode initialization.

We may need to initialize multiple inode blocks before writing a given
inode. makefs(8) was only initializing a single block at a time, so
certain inode allocation patterns could lead to a situation where it
wrote an inode to an uninitialized block. That inode might be clobbered
by a later initialization, resulting in a filesystem image containing
directory entries that point to a seemingly unused inode.

Reviewed by: imp
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D13505

spdx: initial adoption of licensing ID tags.

The Software Package Data Exchange (SPDX) group provides a specification
to make it easier for automated tools to detect and summarize well known
opensource licenses. We are gradually adopting the specification, noting
that the tags are considered only advisory and do not, in any way,
superceed or replace the license texts.

Special thanks to Wind River for providing access to "The Duke of
Highlander" tool: an older (2014) run over FreeBSD tree was useful as a
starting point.

Initially, only tag files that use BSD 4-Clause "Original" license.

RelNotes: yes
Differential Revision: https://reviews.freebsd.org/D13133

makefs: free buf in case of error

CID: 270190
Submitted by: Siva Mahadevan <smahadevan@freebsdfoundation.org>
Reported by: Coverity
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D11011

makefs: add -O (offset) option

NetBSD revs:
ffs.c 1.60
makefs.8 1.44
makefs.c 1.48
makefs.h 1.33
ffs/buf.c 1.20
ffs/mkfs.c 1.27

Obtained from: NetBSD
Relnotes: Yes
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D10780

makefs: make buf generic

it has nothing to do with ffs and will eventually be moved.
gc sectorsize.

This is a corrected version of r317744.

NetBSD versions:
ffs.c 1.58
ffs/buf.c 1.14 1.18
ffs/buf.h 1.8

Submitted by: Siva Mahadevan <smahadevan@freebsdfoundation.org>
Obtained from: NetBSD
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D10803

makefs: Add soft-updates option

Add the ffs option to enable soft-updates.
The option is only processed is ufs2 has been selected.

Reviewed by: emaste, bapt (earlier version), allanjude (earlier version)
Sponsored by: Gandi.net
Differential Revision: https://reviews.freebsd.org/D10773

makefs: clean up signedness warnings and bump WARNS to 3

Reviewed by: kib
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D10650

Restore `sectorsize` global to unbreak makefs after r317744

This also unbreaks the fstyp tests.

Reported by: Alastair Hogge <agh@fastmail.fm>, Jenkins
Sponsored by: Dell EMC Isilon

makefs: cast snprintf return value to size_t to clear warning

Sponsored by: The FreeBSD Foundation

makefs: use size_t as appropriate to clean up warnings

Sponsored by: The FreeBSD Foundation

makefs: make buf generic

it has nothing to do with ffs and will eventually be moved.
gc sectorsize.

NetBSD versions:
ffs.c 1.58
ffs/buf.c 1.14 1.18
ffs/buf.h 1.8

Obtained from: NetBSD
Sponsored by: The FreeBSD Foundation

makefs: use emalloc and friends

The emalloc set of error-checking memory allocation routines were added
to libnetbsd in r316572. Use them in makefs to reduce differences with
NetBSD.

NetBSD revs:
cd9660.c 1.39
ffs.c 1.56
makefs.c 1.42
walk.c 1.27
cd9660/cd9660_archimedes.c 1.2
cd9660/cd9660_eltorito.c 1.20
cd9660/cd9660_write.c 1.16
cd9660/iso9660_rrip.c 1.12
ffs/buf.c 1.17
ffs/mkfs.c 1.26

Obtained from: NetBSD

makefs: improve error messages

- remove \n
- use __func__
- err adds the error string itself

NetBSD revs:
cd9660.c 1.48 1.49
ffs/buf.c 1.21
ffs/mkfs.c 1.27

Obtained from: NetBSD

makefs: sync option parsing with NetBSD

- add support for parsing different types; not just int
- homogenize option parsing
- fix single letter parsing
- remove duplicated code

NetBSD revisions:
cd9660.c 1.36 1.37 1.38 1.41 1.42 1.43
ffs.c 1.50 1.51 1.52 1.53 1.56 1.57
makefs.c 1.36 1.37 1.38 1.39 1.40 1.42 1.43 1.44 1.46
makefs.h 1.28 1.29 1.31 1.32

Obtained from: NetBSD
Sponsored by: The FreeBSD Foundation

prefix UFS symbols with UFS_ to reduce namespace pollution

Specifically:
ROOTINO -> UFS_ROOTINO
WINO -> UFS_WINO
NXADDR -> UFS_NXADDR
NDADDR -> UFS_NDADDR
NIADDR -> UFS_NIADDR
MAXSYMLINKLEN_UFS[12] -> UFS[12]_MAXSYMLINKLEN (for consistency)

Also prefix ext2's and nandfs's NDADDR and NIADDR with EXT2_ and NANDFS_

Reviewed by: kib, mckusick
Obtained from: NetBSD
MFC after: 1 month
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D9536

makefs: make the buffer functions look exactly like the kernel ones

From NetBSD christos Sat Jan 26 00:19:39 2013 +0000

make the buffer functions look exactly like the kernel ones and add other
cruft to make the kernel files compile.

ffs.c 1.54
ffs/buf.c 1.13
ffs/buf.h 1.3
ffs/ffs_alloc.c 1.21
ffs/ffs_balloc.c 1.15

Reviewed by: marcel, ngie
Obtained from: NetBSD
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D8404

Assign a random number to di_gen (for FFS), instead of extracting it
from struct stat. We don't necessarily have permissions to see the
generation number and the host OS may not have st_gen in struct stat
anyway. Since the kernel assigns random numbers, there's nothing
meaningful about the generation that requires us to preserve it when
the file system image is created. With this change, all generation
numbers come from random() and that makes it easier to add support
for reproducible builds at some time in the future (i.e. by adding
an argument to makefs that changes the behaviour of random() so that
it always returns 0 or some predictable sequence).

Differential Revision: https://reviews.freebsd.org/D8418

makefs: sync NetBSD IDs with upstream for changes that we already have

May 22 21:51:39 2011 +0000 (christos):

From Nathan Whitehorn (nwhitehorn at freebsd dot org):
Add code to generate bootable ISOs on Powermac and CHRP systems.
Synthesize some partition maps (APM and MBR, respectively) pointing
to (a) the whole disk, and (b) relevant El Torito boot images that
have been added by other code. These partition maps are a little
bit funny looking, but they seem to work. FreeBSD has been using
this successfully in their release generation on powerpc, as well
as generating all non-SPARC install media. SPARC support could
probably be added as an extension of this patch.

makefs.8 1.33

Tue Aug 23 17:09:11 2011 +0000 (christos):

PR/45285: Martin Matuska: makefs does not properly convert ISO level 1 and 2
filenames (buffer overflow)

makefs does not properly verify the maximum filename length in the
special "." case for both ISO level 1 and ISO level 2 filename
conversion. This creates broken images or causes a buffer overflow
(ISO level 2).

ISO level 1:
If a filename contains only dots or up to 8 characters followed by
dots the 8+3 limit check doesn't work.

ISO level 2:
If a filename contains a dot in the first 30 characters and a dot
on the 30th character, the length limit check doesn't work and the
buffer is overflowed.

$ mkdir level1
$ touch level1/12345............
$ makefs -t cd9660 -o isolevel=1 test.iso level1

$ mkdir level2
$ touch level2/1234567890.2345678901234567.....34567890123456789012345
$ makefs -t cd9660 -o isolevel=2 test.iso level2

cd9660.c 1.32

Sun Oct 9 21:33:43 2011 +0000 (christos):

add support for setting the ufs label. (Nathan Whitehorn)

ffs.c 1.45
ffs.h 1.2
mkfs.c 1.22
makefs.8 1.37

Obtained from: NetBSD

makefs: Provide a -T option to set timestamps to a consistent value

This is taken from the NetBSD versions listed below and adapted to the
makefs version in FreeBSD, along with a bug fix from cem@ that will be
sent to NetBSD.

Reviewed by: pfg
Approved by: re (gjb)
Obtained from: NetBSD
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D6835

makefs(8): Clarify the comment concerning seeding.

Avoid giving the impression makefs currently supports reproduceable
builds.

makefs(1): use all the random(3) range.

The generation number is uint32_t so we can fit the complete range
of random(3). We could have used arc4random() but the result would
be unpredictable and it would prohibit reproducible builds.

While here add a comment where seeding is done: this affects
reproducible builds and might have to be re-visited to use a
release dependent value.

MFC after: 2 weeks

makefs: introduce a new option to specify what to round the resulting
image up to.

From ticket:

While trying to run FreeBSD/mips on some device having very small flash media,
one is forced to compress file system with mkulzma(8) utility. It is desirable
to specify small UFS block/fragment sizes like 4096/512 bytes for makefs(8)
and big compression block size like 65535 bytes to mkulzma at the same time.
Then one obtains very good comression ratios (like 75% and more) but faces
the following problem.

geom_uncompress kernel module reports GEOM provider size rounded up to its
compression block size. Generally, this changes original media size and now
it fails to match the size of embedded UFS file system that leads to other
problems, f.e. geom_label kernel module does not like this and skips the
file system while tasting the GEOM and looking for UFS label.

This makes it impossible to refer to the file system using known UFS label
instead of something like /dev/map/rootfs.uncompress.

The following patch introduces new command line option "-r roundup" for makefs
that makes it round up the image to specified block size. Hence, geom_uncompress
does not change GEOM media size for images rounded that way and geom_label
accepts such GEOMs just fine.

With the patch applied, one can use following commands:

$ makefs -t ffs -r 65536 -o bsize=4096,fsize=512,label=flash optimization=space fs.img fs
$ mkulzma -s 65536 -o fs.img.ulzma fs.img

PR: bin/203707
Submitted by: <eugen@grosbein.net>

Fix a typo in the FFS maxbpg option, it was erroneously spelled maxbpf.

The error exists in the NetBSD upstream version as well and will be reported back.

PR: 196598
Submitted by: Dan McGregor
MFC after: 1 week

r258695 introduces a sanity check for makefs in order to verify that
the minimum image size specified is always less than the maximum
image size. If makefs(1) is invoked specifying minimum image size,
but not maximum one, the program exits with an error. Example:

# sudo -E makefs -M 538968064 -B be /home/davide/disk.img $DESTDIR
makefs: `/home/davide/tftproot/mips' minsize of 538968064 rounded up
to ffs bsize of 8192 exceeds maxsize 0. Lower bsize, or round the
minimum and maximum sizes to bsize.

Assert then that minsize < maxsize iff maxsize is specified.
This change allows me to build MIPS images using makefs(1) and following
what specified in the wiki again.

Reviewed by: jmallett, ngie

Provide a helpful diagnostic when the minimum size rounded to the block size
would exceed the maximum size. This can be a difficult problem to diagnose
if one is, for instance, using -s with a fixed size in a script and the bsize
calculated for a filesystem image changes, necessitating a re-rounding of the
image size or a hand-setting of the bsize. Previously one would get a
cryptic message about how the size exceeded the maximum size, which normally
only happens if the contents of the image are larger than specified.

Add -p flag to create the image as a sparse file.

Submitted by: Shesha Sreenivasamurthy
PR: bin/167779

Allow contents of multiple directories to be merged to the current image.
Note this patch was submitted to NetBSD and they already adopted it.

http://mail-index.netbsd.org/source-changes/2012/01/28/msg031078.html

MFC after: 1 week

Spelling fixes for usr.sbin/

Add support to makefs(8) to add UFS labels to images.

Submitted by: avg
MFC after: 3 days

Add support for using mtree(5) manifest files to define the image
to be created. The support is based on mtree version 2.0, as used
in libarchive, but adds new features on top of it.

The current implementation is fully functional, but is envisioned
to grow at least the following additional features over time:
o Add support for the /include special command so that manifest
files can be constructed using includable fragments.
o Add support specifying a search path to locate content files.
o Content file filters: commands that provide file contents on
stdout.

The manifest file eliminates the need to first construct a tree
as root in order to create an image and allows images (releases)
to be created directly from object trees and/or source trees.

Reviewed by: deo
Sponsored by: Juniper Networks, Inc

The dump, fsck_ffs, fsdb, fsirand, newfs, makefs, and quot utilities
include sys/time.h instead of time.h. This include is incorrect as
per the manpages for the APIs and the POSIX definitions. This commit
replaces sys/time.h where necessary with time.h.

The commit also includes some minor style(9) header fixup in newfs.

This commit is part of a larger effort by Garrett Cooper started in
//depot/user/gcooper/posix-conformance-work/ -- to make FreeBSD more
POSIX compliant.

Submitted by: Garrett Cooper yanegomi at gmail dot com

Sync with the latest version from NetBSD. It notably addds ISO9660 support.

Submitted by: bapt

A new jail(8) with a configuration file, to replace the work currently done
by /etc/rc.d/jail.

Create the altix project branch. The altix project will add support
for the SGI Altix 350 to FreeBSD/ia64. The hardware used for porting
is a two-module system, consisting of a base compute module and a
CPU expansion module. SGI's NUMAFlex architecture can be an excellent
platform to test CPU affinity and NUMA-aware features in FreeBSD.

fix 64-bit build