sbin: Automated cleanup of cdefs and other formatting

Apply the following automated changes to try to eliminate
no-longer-needed sys/cdefs.h includes as well as now-empty
blank lines in a row.

Remove /^#if.*\n#endif.*\n#include\s+<sys/cdefs.h>.*\n/
Remove /\n+#include\s+<sys/cdefs.h>.*\n+#if.*\n#endif.*\n+/
Remove /\n+#if.*\n#endif.*\n+/
Remove /^#if.*\n#endif.*\n/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/types.h>/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/param.h>/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/capsicum.h>/

Sponsored by: Netflix

Ensure I/O buffers in libufs(3) are 128-byte aligned.

Various disk controllers require their buffers to be aligned to a
cache-line size (128 bytes). For buffers allocated in structures,
ensure that they are 128-byte aligned. Use aligned_malloc to allocate
memory to ensure that the returned memory is 128-byte aligned.

While we are here, we replace the dynamically allocated inode buffer
with a buffer allocated in the uufsd structure just as the superblock
and cylinder group buffers do.

This can be removed if/when the kernel is fixed. Because this problem
has existed on one I/O subsystem or another since the 1990's, we
are probably stuck with dealing with it forever.

The problem most recent showed up in Azure, see:
https://reviews.freebsd.org/D41728
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=267654
Before these fixes were applied, it was confirmed that the changes
in this commit also fixed the issue in Azure.

Reviewed-by: Warner Losh, kib
Tested-by: Souradeep Chakrabarti of Microsoft (earlier version)
PR: 267654
Differential Revision: https://reviews.freebsd.org/D41724

Remove $FreeBSD$: one-line .c pattern

Remove /^[\s*]*__FBSDID\("\$FreeBSD\$"\);?\s*\n/

Support background fsck_ffs(8) on filesystems using journaled soft updates

An earlier addition of code to fsck_ffs(8) allowed it to support
snapshots when running with journalled soft updates. Further
functionality has now been added to fsck_ffs(8) to allow it to use
snapshots to run in background on live filesystems running with
journaled soft updates. This commit enables the use of this functionality.

Tested-by: Peter Holm
Sponsored-by: The FreeBSD Foundation
MFC-after: 2 weeks

Add diagnostics to fsck_ffs(8) for journaled soft-updates debugging.

MFC-after: 1 week
Sponsored-by: The FreeBSD Foundation

Text format cleanups. No functional changes intended.

MFC-after: 1 week
Sponsored-by: The FreeBSD Foundation

fsck_ffs: Fix typos

Signed-off-by: Elyes Haouas <ehaouas@noos.fr>
Reviewed by: imp
Pull Request: https://github.com/freebsd/freebsd-src/pull/653

Fix a bug in fsck_ffs(8) triggered by corrupted filesystems.

When loading the root directory ensure that it is a directory
and has a size greater than the minimum directory size. If an
invalid root directory is found, fall back to full fsck.

Reported-by: Robert Morris
PR: 271414
MFC-after: 1 week
Sponsored-by: The FreeBSD Foundation

Fix a bug in fsck_ffs(8) triggered by corrupted filesystems.

Check for valid block numbers while loading journal entries that
contain block numbers. If an invalid block number is found, fall
back to full fsck.

Reported-by: Robert Morris
PR: 271383
MFC-after: 1 week
Sponsored-by: The FreeBSD Foundation

Fix a bug in fsck_ffs(8) triggered by corrupted filesystems.

Check for valid file size before processing journal entries for it.
Done by extracting the file size check from pass1.c into chkfilesize()
then using it in the journal code in suj.c

Reported-by: Robert Morris
PR: 271378
MFC-after: 1 week
Sponsored-by: The FreeBSD Foundation

spdx: The BSD-2-Clause-FreeBSD identifier is obsolete, drop -FreeBSD

The SPDX folks have obsoleted the BSD-2-Clause-FreeBSD identifier. Catch
up to that fact and revert to their recommended match of BSD-2-Clause.

Discussed with: pfg
MFC After: 3 days
Sponsored by: Netflix

fsck_ffs: fix the previous change that skipped pass 5 in some cases

The previous change involved calling check_cgmagic() twice in a row
for the same CG in order to differentiate when the CG was already ok vs.
when the CG was rebuilt, but that doesn't work because the second call
(which was supposed to rebuild the CG) returns 0 (indicating that
the CG was not rebuilt) due to the prevfailcg check causing an early
failure return. Fix this by moving the rebuild part of check_cgmagic()
out into a separate function which is called by pass1() when it wants to
rebuild a CG.

Fixes: da86e7a20dc4a4b17e8d9e7630ed9b675cf71702
Reported by: pho
Discussed with: mckusick
Sponsored by: Netflix

Rewrite function definitions with identifier lists.

A few functions snuck in with K&R style definitions.

Also add some missing memory frees.

MFC after: 1 week

Correct several bugs in fsck_ffs(8) triggered by corrupted filesystems.

If a directory entry has an illegal inode number (less than zero
or greater than the last inode in the filesystem) the entry is removed.
If a directory '.' or '..' entry had an illegal inode number they
were being removed. Since fsck_ffs knows what the correct value is
for these two entries fix them rather deleting them.

Add much more extensive cylinder group checks and use them to be
more careful about rebuilding a cylinder group.

Check for out-of-range block numbers before trying to free them.

When a directory is deleted also remove its cache entry created
in pass1 so that later passes do not try to operate on a deleted
directory.

Check for ctime(3) returning NULL before trying to use its return.

When freeing a directory inode, do not try to interpret it as a
directory.

Reserve space in the inostatlist to have room to allocate a
lost+found directory.

If an invalid block number is found past the end of an inode simply
remove it rather than clearing and removing the inode.

Modernize the inoinfo structure to use queue(3) LIST rather than a
handrolled linked list implementation.

Reported by: Bob Prohaska, John-Mark Gurney, and Mark Millard
Tested by: Peter Holm
Reviewed by: Peter Holm
MFC after: 2 weeks
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D38668

Add support for managing UFS/FFS snapshots to fsck_ffs(8).

The kernel handles the managment of UFS/FFS snapshots. Since UFS/FFS
updates filesystem data (rather than always writing changes to new
locations like ZFS), the kernel must check every filesystem write
to see if the block being written is part of a snapshot. If it is
part of a snapshot, then the kernel must make a copy of the old
block value into a newly allocated block for the snapshot before
allowing the write to be done. Similarly, if a block is being freed,
the kernel must check to see if it is part of a snapshot and let
the snapshot claim the block rather than freeing it for future use.
When a snapshot is freed, its blocks need to be offered to older
snapshots and freed only if no older snapshots wish to claim them.

When snapshots were added to UFS/FFS they were integrated into soft
updates and just a small part of the management of snapshots needed
to be added to fsck_ffs(8) as soft updates minimized the set of
snapshot changes that might need correction. When journaling was
added to soft updates a much more complete knowledge of snapshots
needed to be added to fsck_ffs(8) for it to be able to properly
handle the filesystem changes that a journal rollback needs to do
(specifically the freeing and allocation of blocks). Since this
functionality was unavailable, the use of snapshots was disabled
when running with journaled soft updates.

This set of changes imports the kernel code for the management of
snapshots to fsck_ffs(8). With this code in place it will become
possible to enable snapshots when running with journalled soft
updates. The most immediate benefit will be the ability to use
snapshots to take consistent filesystem dumps on live filesystems.
Future work will be done to update fsck_ffs(8) to be able to use
snapshots to run in background on live filesystems running with
journaled soft updates.

Reviewed by: kib
Tested by: Peter Holm
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D36491

Remove #define _KERNEL hacks from libprocstat

Make sys/buf.h, sys/pipe.h, sys/fs/devfs/devfs*.h headers usable in
userspace, assuming that the consumer has an idea what it is for.
Unhide more material from sys/mount.h and sys/ufs/ufs/inode.h,
sys/ufs/ufs/ufsmount.h for consumption of userspace tools, with the
same caveat.

Remove unacceptable hack from usr.sbin/makefs which relied on sys/buf.h
being unusable in userspace, where it override struct buf with its own
definition. Instead, provide struct m_buf and struct m_vnode and adapt
code to use local variants.

Reviewed by: mckusick
Tested by: pho
Sponsored by: The FreeBSD Foundation
Differential revision: https://reviews.freebsd.org/D28679

Rewrite the disk I/O management system in fsck_ffs(8). Other than
making fsck_ffs(8) run faster, there should be no functional change.

The original fsck_ffs(8) had its own disk I/O management system.
When gjournal(8) was added to FreeBSD 7, code was added to fsck_ffs(8)
to do the necessary gjournal rollback. Rather than use the existing
fsck_ffs(8) disk I/O system, it wrote its own from scratch. Similarly
when journalled soft updates were added in FreeBSD 9, code was added
to fsck_ffs(8) to do the necessary journal rollback. And once again,
rather than using either of the existing fsck_ffs(8) disk I/O
systems, it wrote its own from scratch. Lastly the fsdb(8) utility
uses the fsck_ffs(8) disk I/O management system. In preparation for
making the changes necessary to enable snapshots to be taken when
using journalled soft updates, it was necessary to have a single
disk I/O system used by all the various subsystems in fsck_ffs(8).

This commit merges the functionality required by all the different
subsystems into a single disk I/O system that supports all of their
needs. In so doing it picks up optimizations from each of them
with the results that each of the subsystems does fewer reads and
writes than it did with its own customized I/O system. It also
greatly simplifies making changes to fsck_ffs(8) since everything
goes through a single place. For example the ginode() function
fetches an inode from the disk. When inode check hashes were added,
they previously had to be checked in the code implementing inode
fetch in each of the three different disk I/O systems. Now they
need only be checked in ginode().

Tested by: Peter Holm
Sponsored by: Netflix

Update the libufs cgget() and cgput() interfaces to have a similar
API to the sbget() and sbput() interfaces. Specifically they take
a file descriptor pointer rather than the struct uufsd *disk pointer
used by the libufs cgread() and cgwrite() interfaces. Update fsck_ffs
to use these revised interfaces.

No functional changes intended.

Sponsored by: Netflix

Inode check-hash errors were being reported after system crashes.
Trace the cause down to journalled soft updates recovery code in
fsck failing to recompute the check-hash after updating an inode.

As inode check-hash was first introduced to UFS in FreeBSD 13,
there is no need to MFC this commit.

Reported by: Chuck Silvers
Sponsored by: Netflix

Add an inode check-hash verification when running the journalled
soft update recovery code with the debugging (-d) option.

As inode check-hash was first introduced to UFS in FreeBSD 13,
there is no need to MFC this commit.

Reported by: Chuck Silvers
Sponsored by: Netflix

When running fsck_ffs manually, do not ask:

USE JOURNAL? [yn]

when the journal timestamp does not match the filesystem mount time
as we are just going to print an error and fall through to a full fsck.
Instead, just run a full fsck.

Requested by: Bjoern A. Zeeb (bz)
MFC after: 7 days

Replace an uninitialized variable with the correct element from the
superblock when doing recovery with journalled soft updates.

Reported by: Chuck Silvers
MFC after: 3 days
Sponsored by: Netflix

When running with journaled soft updates, some updated inodes were not
having their check hashes recomputed which resulted in spurious inode
check-hash errors when the system came back up after a crash.

Reported by: Alan Somers
Sponsored by: Netflix

Revert r313780 (UFS_ prefix)

Prefix UFS symbols with UFS_ to reduce namespace pollution

Followup to r313780. Also prefix ext2's and nandfs's versions with
EXT2_ and NANDFS_.

Reported by: kib
Reviewed by: kib, mckusick
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D9623

Use sbput(3) rather than sbwrite(3) to ensure that the updated copy of
the superblock gets written.

Reported by: Mark Johnston <markj@FreeBSD.org>

More throughly integrate libufs into fsck_ffs by using its cgput()
routine to write out the cylinder groups rather than recreating the
calculation of the cylinder-group check hash in fsck_ffs.

No functional change intended.

Correct fsck journal-recovery code to update a cylinder-group
check-hash after making changes to the cylinder group. The problem
was that the journal-recovery code was calling the libufs bwrite()
function instead of the cgput() function. The cgput() function updates
the cylinder-group check-hash before writing the cylinder group.

This change required the additions of the cgget() and cgput() functions
to the libufs API to avoid a gratuitous bcopy of every cylinder group
to be read or written. These new functions have been added to the
libufs manual pages. This was the first opportunity that I have had
to use and document the use of the EDOOFUS error code.

Reviewed by: kib
Reported by: emaste and others

various: general adoption of SPDX licensing ID tags.

Mainly focus on files that use BSD 2-Clause license, however the tool I
was using misidentified many licenses so this was mostly a manual - error
prone - task.

The Software Package Data Exchange (SPDX) group provides a specification
to make it easier for automated tools to detect and summarize well known
opensource licenses. We are gradually adopting the specification, noting
that the tags are considered only advisory and do not, in any way,
superceed or replace the license texts.

No functional change intended.

Use UFS_LINK_MAX instead of LINK_MAX.

Submitted by: bde
Sponsored by: Chelsio Communications

Commit the 64-bit inode project.

Extend the ino_t, dev_t, nlink_t types to 64-bit ints. Modify
struct dirent layout to add d_off, increase the size of d_fileno
to 64-bits, increase the size of d_namlen to 16-bits, and change
the required alignment. Increase struct statfs f_mntfromname[] and
f_mntonname[] array length MNAMELEN to 1024.

ABI breakage is mitigated by providing compatibility using versioned
symbols, ingenious use of the existing padding in structures, and
by employing other tricks. Unfortunately, not everything can be
fixed, especially outside the base system. For instance, third-party
APIs which pass struct stat around are broken in backward and
forward incompatible ways.

Kinfo sysctl MIBs ABI is changed in backward-compatible way, but
there is no general mechanism to handle other sysctl MIBS which
return structures where the layout has changed. It was considered
that the breakage is either in the management interfaces, where we
usually allow ABI slip, or is not important.

Struct xvnode changed layout, no compat shims are provided.

For struct xtty, dev_t tty device member was reduced to uint32_t.
It was decided that keeping ABI compat in this case is more useful
than reporting 64-bit dev_t, for the sake of pstat.

Update note: strictly follow the instructions in UPDATING. Build
and install the new kernel with COMPAT_FREEBSD11 option enabled,
then reboot, and only then install new world.

Credits: The 64-bit inode project, also known as ino64, started life
many years ago as a project by Gleb Kurtsou (gleb). Kirk McKusick
(mckusick) then picked up and updated the patch, and acted as a
flag-waver. Feedback, suggestions, and discussions were carried
by Ed Maste (emaste), John Baldwin (jhb), Jilles Tjoelker (jilles),
and Rick Macklem (rmacklem). Kris Moore (kris) performed an initial
ports investigation followed by an exp-run by Antoine Brodin (antoine).
Essential and all-embracing testing was done by Peter Holm (pho).
The heavy lifting of coordinating all these efforts and bringing the
project to completion were done by Konstantin Belousov (kib).

Sponsored by: The FreeBSD Foundation (emaste, kib)
Differential revision: https://reviews.freebsd.org/D10439

prefix UFS symbols with UFS_ to reduce namespace pollution

Specifically:
ROOTINO -> UFS_ROOTINO
WINO -> UFS_WINO
NXADDR -> UFS_NXADDR
NDADDR -> UFS_NDADDR
NIADDR -> UFS_NIADDR
MAXSYMLINKLEN_UFS[12] -> UFS[12]_MAXSYMLINKLEN (for consistency)

Also prefix ext2's and nandfs's NDADDR and NIADDR with EXT2_ and NANDFS_

Reviewed by: kib, mckusick
Obtained from: NetBSD
MFC after: 1 month
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D9536

Use type-independent formats for printing nlink_t and ino_t.

Extracted from: ino64 work by gleb, mckusick
Discussed with: mckusick
Sponsored by: The FreeBSD Foundation
MFC after: 1 week

sbin: ake use of our rounddown() macro when sys/param.h is available.

No functional change.

fsck_ffs: Revert partially the unsigned changes.

Any value of uint16_t will be internally promoted to int so
changing them to an unsigned value doesn't help.

Missing revert value in suj_read().

X-MFC with: r298551

fsck_ffs: Revert partially the unsigned changes.

Any value of uint16_t will be internally promoted to int so
changing them to an unsigned value doesn't help.

Make clear we want to use uint32_t for closedisk()

X-MFC with: r298551

fsck_ffs: Adopt some type safety for the journalling checks.

fs_ncg is of type uint32, and we were indexing it with an int.
Fixed this using an unsigned type and adopt some other unsigned
indexes to remind us when we are dealing with unsigned numbers.

Reviewed by: mckusick
MFC after: 5 days

Add the -R option to allow fsck_ffs to restart itself when too many critical
errors have been detected in a particular run.

Clean up the global state variables so that a restart can happen correctly.

Separate the global variables in fsck_ffs and fsdb to their own file. This
fixes header sharing with fscd.

Correctly initialize, static-ize, and remove global variables as needed in
dir.c. This fixes a problem with lost+found directories that was causing
a segfault.

Correctly initialize, static-ize, and remove global variables as needed in
suj.c.

Initialize the suj globals before allocating the disk object, not after.
Also ensure that 'preen' mode doesn't conflict with 'restart' mode

Submitted by: scottl, max
Reviewed by: max, mckusick (earlier version)
Obtained from: Netflix
MFC after: 3 days

Revert 248634 and 248643 (e.g., restoring 248625 and 248639).

Build verified by: Glen Barber (gjb@)

Revert svn r248625

Clang errors around printf could be trivially fixed, but the breakage in
sbin/fsdb were to significant for this type of change.

Submitter of this changeset has been notified and hopefully this can be
restored soon.

Speed up fsck by caching the cylinder group maps in pass1 so
that they do not need to be read again in pass5. As this nearly
doubles the memory requirement for fsck, the cache is thrown away
if other memory needs in fsck would otherwise fail. Thus, the
memory footprint of fsck remains unchanged in memory constrained
environments.

This work was inspired by a paper presented at Usenix's FAST '13:
www.usenix.org/conference/fast13/ffsck-fast-file-system-checker

Details of this implementation appears in the April 2013 of ;login:
www.usenix.org/publications/login/april-2013-volume-38-number-2.
A copy of the April 2013 ;login: paper can also be downloaded
from: www.mckusick.com/publications/faster_fsck.pdf.

Reviewed by: kib
Tested by: Peter Holm
MFC after: 4 weeks

- blk_equals() is too strict. If the journal entry defines more frags
than we're claiming it should still be considered an exact match. This
would previously leak frags that had been extended.
- If there is a sequence number problem in the journal print the sequence
numbers we've seen so far for debugging.
- Clean up the block mask related debuging printfs. Some are redundant.

MFC after: 1 week

Fix some nearby type and style errors.

Pointed out by: bde

Fix sbin/ build with a 64-bit ino_t.

Original code by: Gleb Kurtsou

Fix fsck_ffs build with a 64-bit ino_t.

Original code by: Gleb Kurtsou

- Fix a typo in debug message.

Approved by: kib (mentor)
MFC after: 3 days

fsck_ffs shall accept the configured journal size, and not refuse to
operate on it if journal size is greater then SUJ_MAX. The later
constant is only to select maximal journal size when user did not
specified size explicitely.

Submitted by: Andrey Zonov <andrey@zonov.org>
Reviewed by: mckusick
MFC after: 1 week

For incompleted block allocations or frees, the inode block count usage
must be recalculated. The blk_check pass of suj checker explicitely marks
inodes which owned such blocks as needing block count adjustment. But
ino_adjblks() is only called by cg_trunc pass, which is performed before
blk_check. As result, the block use count for such inodes is left wrong.
This causes full fsck run after journaled run to still find inconsistencies
like 'INCORRECT BLOCK COUNT I=14557 (328 should be 0)' in phase 1.

Fix this issue by running additional adj_blk pass after blk_check, which
updates the field.

Reviewed by: jeff, mckusick
MFC after: 1 week

Change the type of real_dev_bsize variable from long to u_int.
The DIOCGSECTORSIZE takes u_int * as an argument, using long *
causes failures on big-endian targets.

Diagnosed by: Michiel Boland <boland37 xs4all nl>
PR: sparc64/163460
Tested by: pho (x86), flo (sparc64)
MFC after: 1 week

- Handle the JOP_SYNC case as appropriate.

Reported by: pho

Implement fully asynchronous partial truncation with softupdates journaling
to resolve errors which can cause corruption on recovery with the old
synchronous mechanism.

- Append partial truncation freework structures to indirdeps while
truncation is proceeding. These prevent new block pointers from
becoming valid until truncation completes and serialize truncations.
- On completion of a partial truncate journal work waits for zeroed
pointers to hit indirects.
- softdep_journal_freeblocks() handles last frag allocation and last
block zeroing.
- vtruncbuf/ffs_page_remove moved into softdep_*_freeblocks() so it
is only implemented in one place.
- Block allocation failure handling moved up one level so it does not
proceed with buf locks held. This permits us to do more extensive
reclaims when filesystem space is exhausted.
- softdep_sync_metadata() is broken into two parts, the first executes
once at the start of ffs_syncvnode() and flushes truncations and
inode dependencies. The second is called on each locked buf. This
eliminates excessive looping and rollbacks.
- Improve the mechanism in process_worklist_item() that handles
acquiring vnode locks for handle_workitem_remove() so that it works
more generally and does not loop excessively over the same worklist
items on each call.
- Don't corrupt directories by zeroing the tail in fsck. This is only
done for regular files.
- Push a fsync complete record for files that need it so the checker
knows a truncation in the journal is no longer valid.

Discussed with: mckusick, kib (ffs_pages_remove and ffs_truncate parts)
Tested by: pho

Mechanical whitespace cleanup.

MFC after: 3 weeks

In checker, read journal by sectors.

Due to UFS insistence to pretend that device sector size is 512 bytes,
sector size is obtained from ioctl(DIOCGSECTORSIZE) for real devices,
and from the label otherwise. The file images without label have to
be made with 512 sector size.

In collaboration with: pho
Reviewed by: jeff
Tested by: bz, pho

The dump, fsck_ffs, fsdb, fsirand, newfs, makefs, and quot utilities
include sys/time.h instead of time.h. This include is incorrect as
per the manpages for the APIs and the POSIX definitions. This commit
replaces sys/time.h where necessary with time.h.

The commit also includes some minor style(9) header fixup in newfs.

This commit is part of a larger effort by Garrett Cooper started in
//depot/user/gcooper/posix-conformance-work/ -- to make FreeBSD more
POSIX compliant.

Submitted by: Garrett Cooper yanegomi at gmail dot com

A new jail(8) with a configuration file, to replace the work currently done
by /etc/rc.d/jail.

- Permit zero length directories as a handled inconsistency. This allows
directory truncation to proceed before the link has been cleared. This
is accomplished by detecting a directory with no . or .. links and
clearing the named directory entry in the parent.
- Add a new function ino_remref() which handles the details of removing
a reference to an inode as a result of a lost directory. There were
some minor errors in various subcases of this routine.

Improve fsck robustness for SU+J cases:

- Use err/errx only when the case is really fatal. For other
cases, fall back to full fsck instead of quiting fsck.
- Plug a memory leak.
- Avoid divide by zero when printing summary.
- Output "FILE SYSTEM IS MARKED CLEAN" when a successful
journal recovering is done.
- When -f is specified, do full fsck instead of journal recovery.

suj.c seems to contain two versions of the code.
Remove the one that doesn't compile.

- Merge soft-updates journaling from projects/suj/head into head. This
brings in support for an optional intent log which eliminates the need
for background fsck on unclean shutdown.

Sponsored by: iXsystems, Yahoo!, and Juniper.
With help from: McKusick and Peter Holm