Enable soft updates by default for UFS2 filesystems.

Soft updates dramatically improve the performance of UFS filesystems.
The newfs(8) utility currently does not enable them by default. The
FreeBSD installer enables soft updates by default. However custom
built installations that do not specify the -U option to newfs(8)
and the prebuilt UFS system images get filesystems without soft
updates enabled.

There are several testing sites that run benchmarks comparing the
performance of Linux distributions versus BSD distributions. When
they run filesystem comparison benchmarks they use newfs(8) to
create the UFS filesystem. Because it does not have soft updates
enabled it runs poorly versus the Linux ext4 filesystem. When I
have suggested to them that they should enable soft updates on the
UFS filesystem in their testing their response is that they expect
the utility that creates the filesystem to use optimal defaults and
that they cannot be expected to fiddle with various option settings.

The purpose of this change is to give a filesystem created with
newfs(8) reasonably optimal settings. For UFS2 this means enabling
soft updates. For UFS1 which tends to be used on small systems with
minimal memory and CPU speed, the lower memory footprint of running
without soft updates is a more sensible default.

This change adds a note in the section of the newfs(8) manual page
that describes the -U option for enabling soft updates that they
are enabled by default for UFS2 filesystems and that they can be
disabled by using tunefs(8).

Reviewed-by: Warner Losh, kib
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D45201

Remove copyright strings ifdef'd out

We've ifdef'd out the copyright strings for some time now. Go ahead and
remove the ifdefs. Plus whatever other detritis was left over from other
recent removals. These copyright strings are present in the comments and
are largely from CSRG's attempt at adding their copyright to every
binary file (which modern interpretations of the license doesn't
require).

Sponsored by: Netflix

sbin: Remove ancient SCCS tags.

Remove ancient SCCS tags from the tree, automated scripting, with two
minor fixup to keep things compiling. All the common forms in the tree
were removed with a perl script.

Sponsored by: Netflix

Remove $FreeBSD$: one-line .c pattern

Remove /^[\s*]*__FBSDID\("\$FreeBSD\$"\);?\s*\n/

Mark usage function as __dead2 in programs where it does not return

In most cases, usage does not return, so mark them as __dead2. For the
cases where they do return, they have not been marked __dead2.

Reviewed by: imp
Pull Request: https://github.com/freebsd/freebsd-src/pull/735

newfs: prefer unsigned index over signed

We can just use a for loop starting at 0 instead of a while loop
starting at -1.

Reviewed by: imp, mckusick
Pull Request: https://github.com/freebsd/freebsd-src/pull/733

Remove #define _KERNEL hacks from libprocstat

Make sys/buf.h, sys/pipe.h, sys/fs/devfs/devfs*.h headers usable in
userspace, assuming that the consumer has an idea what it is for.
Unhide more material from sys/mount.h and sys/ufs/ufs/inode.h,
sys/ufs/ufs/ufsmount.h for consumption of userspace tools, with the
same caveat.

Remove unacceptable hack from usr.sbin/makefs which relied on sys/buf.h
being unusable in userspace, where it override struct buf with its own
definition. Instead, provide struct m_buf and struct m_vnode and adapt
code to use local variants.

Reviewed by: mckusick
Tested by: pho
Sponsored by: The FreeBSD Foundation
Differential revision: https://reviews.freebsd.org/D28679

Allow dashes as a valid character in UFS labels.

Reviewed by: mckusick, imp, 0mp
MFC after: 2 weeks
Differential Revision: D18991

Update tunefs and newfs error messages for the -L (volume label) option
to note that underscores are valid.

PR: 235182
Reported by: Rodney W. Grimes (rgrimes@)
Sponsored by: Netflix

newfs: clean up warnings

- remove param: unused since r95357.
- correct definition of usage
- add explicit fallthrough notice. The existing one doesn't work with
our selection of "implicit-fallthrough" strictness.

This results in WARNS=6 building on amd64, but not other arches

General further adoption of SPDX licensing ID tags.

Mainly focus on files that use BSD 3-Clause license.

The Software Package Data Exchange (SPDX) group provides a specification
to make it easier for automated tools to detect and summarize well known
opensource licenses. We are gradually adopting the specification, noting
that the tags are considered only advisory and do not, in any way,
superceed or replace the license texts.

Special thanks to Wind River for providing access to "The Duke of
Highlander" tool: an older (2014) run over FreeBSD tree was useful as a
starting point.

Only try to enable CK_CLYGRP if we're running on kernel newer than
1200046, the first version that supports this feature. If we set it,
then use an old kernel, we'll break the 'contract' of having
checksummed cylinder groups this flag signifies. To avoid creating
something with an inconsistent state, don't turn the flag on in these
cases. The first full fsck with a new kernel will turn this on.

Spnsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D13114

newfs: warn if newer than kernel

Creating a UFS filesystem with a newfs newer than the running kernel,
and then mounting that filesystem, can lead to interesting failures.

Add a safety belt to explicitly warn when newfs is newer than the
running kernel.

Reviewed by: gjb, jhb, mckusick
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D12765

Allow '_' in labels when specifying -L to newfs.

Reported by: Keve Nagy
Reviewed by: kib
PR: 220163
MFC after: 5 days

Renumber copyright clause 4

Renumber cluase 4 to 3, per what everybody else did when BSD granted
them permission to remove clause 3. My insistance on keeping the same
numbering for legal reasons is too pedantic, so give up on that point.

Submitted by: Jan Schaumann <jschauma@stevens.edu>
Pull Request: https://github.com/freebsd/freebsd/pull/96

Use NULL instead of 0 for pointers.

strchr(3) will return NULL if the character does not appear in the
string.

MFC after: 2 weeks.

Code cleanup unused-but-set-variable spotted by gcc.

Approved by: bapt (mentor)
Differential Revision: D3475

Remove old ioctl use and support, once and for all.

The purpose of this change to the FFS layout policy is to reduce the
running time for a full fsck. It also reduces the random access time
for large files and speeds the traversal time for directory tree walks.

The key idea is to reserve a small area in each cylinder group
immediately following the inode blocks for the use of metadata,
specifically indirect blocks and directory contents. The new policy
is to preferentially place metadata in the metadata area and
everything else in the blocks that follow the metadata area.

The size of this area can be set when creating a filesystem using
newfs(8) or changed in an existing filesystem using tunefs(8).
Both utilities use the `-k held-for-metadata-blocks' option to
specify the amount of space to be held for metadata blocks in each
cylinder group. By default, newfs(8) sets this area to half of
minfree (typically 4% of the data area).

This work was inspired by a paper presented at Usenix's FAST '13:
www.usenix.org/conference/fast13/ffsck-fast-file-system-checker

Details of this implementation appears in the April 2013 of ;login:
www.usenix.org/publications/login/april-2013-volume-38-number-2.
A copy of the April 2013 ;login: paper can also be downloaded
from: www.mckusick.com/publications/faster_fsck.pdf.

Reviewed by: kib
Tested by: Peter Holm
MFC after: 4 weeks

Fix problem with geom_label(4) not recognizing UFS labels on filesystems
extended using growfs(8). The problem here is that geom_label checks if
the filesystem size recorded in UFS superblock is equal to the provider
(i.e. device) size. This check cannot be removed due to backward
compatibility. On the other hand, in most cases growfs(8) cannot set
fs_size in the superblock to match the provider size, because, differently
from newfs(8), it cannot recompute cylinder group sizes.

To fix this problem, add another superblock field, fs_providersize, used
only for this purpose. The geom_label(4) will attach if either fs_size
(filesystem created with newfs(8)) or fs_providersize (filesystem expanded
using growfs(8)) matches the device size.

PR: kern/165962
Reviewed by: mckusick
Sponsored by: FreeBSD Foundation

Add the -j option to enable soft updates journaling when creating
a new file system.

Reviewed by: Kostik Belousov <kostikbel@gmail.com>

Add support for FS_TRIM to user-mode UFS utilities.

Reviewed by: mckusick, pjd, pho
Tested by: pho
MFC after: 1 month

Add the missed 'p' flag to getopt() optstring argument.

MFC after: 1 week

A new jail(8) with a configuration file, to replace the work currently done
by /etc/rc.d/jail.

Create the altix project branch. The altix project will add support
for the SGI Altix 350 to FreeBSD/ia64. The hardware used for porting
is a two-module system, consisting of a base compute module and a
CPU expansion module. SGI's NUMAFlex architecture can be an excellent
platform to test CPU affinity and NUMA-aware features in FreeBSD.

o bdeficize expand_number_int() function;

o revert most of the recent changes (int -> int64_t conversion) by using
this functon for parsing all options.

Change secrorsize back to int, since that's the data type expected by the
ioctl(DIOCGSECTORSIZE). It creates issues on some architectures.

MFC after: 1 week
Reported by: Jayachandran C.

Use expand_number(3) from libutil instead of home-grown function to parse
human-friendly power-of-two numbers (i.e. 2k, 5M etc).

Suggested by: many
MFC after: 1 week

Teach newfs(8) to understand size modifiers for all options taking
size or size-like argument. I.e. "-s 32k" instead of "-s 32768".
Size parsing function has been shamelessly stolen from the truncate(1).
I'm sure many sysadmins out there will appreciate this small
improvement.

MFC after: 1 week

MFC of 203763, 203764, 203768, 203769, 203770, 203782, and 203784.

These fixes correct a problem in the file system that treats large
inode numbers as negative rather than unsigned. For a default
(16K block) file system, this bug began to show up at a file system
size above about 16Tb.

These fixes also update newfs to ensure that it will never create a
filesystem with more than 2^32 inodes.

They also update libufs, tunefs, and growfs so that they properly
handle inode numbers as unsigned.

Reported by: Scott Burns, John Kilburg, and Bruce Evans
Followup by: Jeff Roberson
PR: 133980

MFC r203534: Correct two typos.

Reported by: Brandon Falk <falkman gamozo org>

Ensure that newfs will never create a filesystem with more than 2^32
inodes by cutting back on the number of inodes per cylinder group if
necessary to stay under the limit. For a default (16K block) file
system, this limit begins to take effect for file systems above 32Tb.

This fix is in addition to -r203763 which corrected a problem in the
kernel that treated large inode numbers as negative rather than unsigned.
For a default (16K block) file system, this bug began to show up at a
file system size above about 16Tb.

Reported by: Scott Burns, John Kilburg, Bruce Evans
Followup by: Jeff Roberson
PR: 133980
MFC after: 2 weeks

Correct two typos.

Reported by: Brandon Falk <falkman gamozo org>
MFC after: 1 week

Don't add a bwrite() symbol, it breaks the build when building newfs
statically.
Instead, bring in a stripped down version of sbwrite(), and add the offset
to every bwrite() calls.

Move the check for the ending char in the partition name where
it was before -- the check is only made when getdisklabel()
returns valid info.
On passing, use MAXPARTITIONS to identify the max partition number,
instead of the hardwired 'h'

MFC after: 4 weeks

Enable operation of newfs on plain files, which is useful when you
want to prepare disk images for emulators (though 'makefs' in port
can do something similar).

This relies on:
+ minor changes to pass the consistency checks even when working on a file;

+ an additional option, '-p partition' , to specify the disk partition to
initialize;

+ some changes on the I/O routines to deal with partition offsets.

The latter was a bit tricky to implement, see the details in newfs.h:
in newfs, I/O is done through libufs which assumes that the file
descriptor refers to the whole partition. Introducing support for
the offset in libufs would require a non-backward compatible change
in the library, to be dealt with a version bump or with symbol
versioning.

I felt both approaches to be overkill for this specific application,
especially because there might be other changes to libufs that might
become necessary in the near future.

So I used the following trick:
- read access is always done by calling bread() directly, so we just add
the offset in the (few) places that call bread();
- write access is done through bwrite() and sbwrite(), which in turn
calls bwrite(). To avoid rewriting sbwrite(), we supply our own version
of bwrite() here, which takes precedence over the version in libufs.

MFC after: 4 weeks

- Import the HEAD csup code which is the basis for the cvsmode work.

Rename the undocumented -E option to -X.

Implement -E option which will erase the filesystem sectors before
making the new filesystem. Reserved space in front of the superblock
(bootcode) is not erased.

NB: Erasing can take as long time as writing every sector sequentially.

This is relevant for all flash based disks which use wearlevelling.

- Pay attention to the fact that ioctl(2) is only known to
return -1 on error while any other return value from it can
indicate success. (See RETURN VALUE in our ioctl(2) manpage
and the POSIX spec.)

- Avoid assumptions about the state of the data buffer after
ioctl(2) failure.

MFp4:

Add a new option to newfs(8), -r, to specify reserved space at the
end of the device. It can be useful, e.g., when the device is to
become a member of a gmirror array later w/o losing the file system
on it.

Document the new option in the manpage.

While I'm here, improve error handling for -s option, which is
syntactically similar to -r; and document the fact that -s0 selects
the default fs size explicitly, which can be useful, e.g., in a
menu-based wrapper around newfs(8) requiring some value be entered
for the fs size.

Also fix a small typo in the help line for -s (missing space).

Idea and initial implementation by: marck
Discussed on: -fs
Critical review by: bde
Tested with: cmp(1)

Document -J in usage.

Submitted by: Eric Anderson <anderson@freebsd.org>

Add -J flag to both newfs(8) and tunefs(8) which allows to enable gjournal
support.
I left -j flag for UFS journal implementation which we may gain at some
point.

Sponsored by: home.pl

Document -l and -n options in usage().

Add an option to suppress the creation of the .snap directory in
the new filesystem. This is intended for memory and vnode filesystems
that will never be fsck'ed or dumped.

Obtained from: St. Bernard Software RAPID
MFC after: 2 weeks

Cast to intmax_t when using %jd format.

MFC after: 3 days

Fix '-s' option for large disks and fix printing maximum file system size.

Remove advertising clause from University of California Regent's license,
per letter dated July 22, 1999.

Approved by: core, imp

Add a "-l" flag to newfs, which sets the FS_MULTILABEL flag. This
permits users of newfs to set the multilabel flag on UFS1 and UFS2
file systems from inception without using tunefs.

Obtained from: TrustedBSD Project
Sponsored by: DARPA, McAfee Research

Add the -E command line option to force error conditions for testing.

Sponsord by: St. Bernard Software

Use __FBSDID() to quiet GCC 3.3 warnings.

Throw the switch--change to UFS2 as our default file system format for
FreeBSD 5.1-RELEASE and later:

- newfs(8) will now create UFS2 file systems unless UFS1 is specifically
requested (-O1). To do this, I just twiddled the Oflag default.

- sysinstall(8) will now select UFS2 as the default layout for new
file systems unless specifically requested (use '1' and '2' to change
the file system layout in the disk labeler). To do this, I inverted
the ufs2 flag into a ufs1 flag, since ufs2 is now the default and
ufs1 is the edge case. There's a slight semantic change in the
key behavior: '2' no longer toggles, it changes the selection to UFS2.

This is very similar to a patch David O'Brien sent me at one point, and
that I couldn't find.

Approved by: re (telecon)
Reviewed by: mckusick, phk, bmah

Correct lines incorrectly added to the copyright message. Add missing period.

Submitted by: Bruce Evans <bde@zeta.org.au>
Sponsored by: DARPA & NAI Labs.

Convert newfs to libufs (really). Solves one real issue with previous
version of such. Differences in filesystems generated were found to be
from 1) sbwrite with the "all" parameter 2) removal of writecache. The
sbwrite call was made to perform as the original version, and otherwise
this was checked against a version of newfs with the write cache removed.

Bring in support for volume labels to the filesystem utilities.

Reviewed by: mckusick

Back out conversion to libufs, for now. It seems to cause problems.

Reported by: phk

Convert newfs to use libufs. I've tested this on md filesystems, as has
keramida, and all seems well.

Add some more checks to newfs so that it will not build filesystems
that the kernel will refuse to mount. Specifically it now enforces
the MAXBSIZE blocksize limit. This update also fixes a problem where
newfs could segment fault if the selected fragment size was too large.

PR: bin/30959
Submitted by: Ceri Davies <setantae@submonkey.net>
Sponsored by: DARPA & NAI Labs.

Remove a comma trailing an if clause.

According to Kirk: "Luckily, the statement is usually true".

Spotted by: FlexeLint

Failure to rewrite the disklabel should not be fatal.

Sponsored by: DARPA & NAI Labs.

s/filesystem/file system/g as discussed on -developers

This commit adds basic support for the UFS2 filesystem. The UFS2
filesystem expands the inode to 256 bytes to make space for 64-bit
block pointers. It also adds a file-creation time field, an ability
to use jumbo blocks per inode to allow extent like pointer density,
and space for extended attributes (up to twice the filesystem block
size worth of attributes, e.g., on a 16K filesystem, there is space
for 32K of attributes). UFS2 fully supports and runs existing UFS1
filesystems. New filesystems built using newfs can be built in either
UFS1 or UFS2 format using the -O option. In this commit UFS1 is
the default format, so if you want to build UFS2 format filesystems,
you must specify -O 2. This default will be changed to UFS2 when
UFS2 proves itself to be stable. In this commit the boot code for
reading UFS2 filesystems is not compiled (see /sys/boot/common/ufsread.c)
as there is insufficient space in the boot block. Once the size of the
boot block is increased, this code can be defined.

Things to note: the definition of SBSIZE has changed to SBLOCKSIZE.
The header file <ufs/ufs/dinode.h> must be included before
<ufs/ffs/fs.h> so as to get the definitions of ufs2_daddr_t and
ufs_lbn_t.

Still TODO:
Verify that the first level bootstraps work for all the architectures.
Convert the utility ffsinfo to understand UFS2 and test growfs.
Add support for the extended attribute storage. Update soft updates
to ensure integrity of extended attribute storage. Switch the
current extended attribute interfaces to use the extended attribute
storage. Add the extent like functionality (framework is there,
but is currently never used).

Sponsored by: DARPA & NAI Labs.
Reviewed by: Poul-Henning Kamp <phk@freebsd.org>

more file system > filesystem

Remove the -v option, it is now default behaviour.

Sponsored by: DARPA & NAI Labs

Continue the cleanup preparations for UFS2 (& GEOM):

Use only one filedescriptor. Open in R/O or R/W based in the '-N' option.
Make the filedescriptor a global variable instead of passing it around
as semi-global variable(s).

Remove the undocumented ability to specify type without '-T' option.

Replace fatal() with straight err(3)/errx(3). Save calls to strerror()
where applicable. Loose the progname variable.

Get the sense of the cpgflag test correct so we only issue warnings if
people specify cpg and can't get that. It can be argued that this
should be an error.

Remove the check to see if the disk is mounted: Open for writing
would fail if it were mounted.

Attempt to get the sectorsize and mediasize with the generic disk
ioctls, fall back to disklabel and /etc/disktab as we can.

Notice that on-disk labels still take precedence over /etc/disktab,
this is probably wrong, but not as wrong as the entire concept of
/etc/disktab is.

Sponsored by: DARPA & NAI Labs.

bbsize and sbsize cannot ever be trusted from the disklabel, in
particular as there may not be one. Remove #if 0'ed code which might
mislead people to think otherwise.

unifdef -ULOSTDIR, fsck can make lost+found on the fly.

Sponsored by: DARPA & NAI Labs

Fixed some style bugs in axings. Whitespace before __P was not axed when
__P was axed. The ordering of several things was bogotified by axing
ifdefs.

Unifdef -DCOMPAT

Swing the axe and remove some archaic features from newfs which modern
diskdrives do neither need nor want:

-O create a 4.3BSD format filesystem
-d rotational delay between contiguous blocks
-k sector 0 skew, per track
-l hardware sector interleave
-n number of distinguished rotational positions
-p spare sectors per track
-r revolutions/minute
-t tracks/cylinder
-x spare sectors per cylinder

No change in the produced filesystem image unless one or more of
these options were used.

Approved by: mckusick

Add the undocumented -R option to disable randomness for regression-testing.

Add a couple of simple regression tests accessible with "make test", they
depend on the md(4) driver.

FYI I have also tried running the test against a week old newfs and it
passed.

Further cleanups.

Complete the ANSIfication of newfs by converting function declarations
to C89 style.

Remove the ancient STANDALONE code.

Approved by: phk

Remove yet more vestiges of mount_mfs.

Fixed some style bugs (mainly ones not fixed or made worse by rev.1.44).
Don't use ISO string concatentation to obfuscate long single-line
messages...

Removed vestiges of mount_mfs. Sorted the Makefile a bit.

Remove __P() and register.
Set WARNS=2

This is the beginning of a pre-UFS2 cleanup of newfs.

Sponsored by: DARPA, NAI Labs

Update the default newfs block and fragment sizes from 8192/1024 to
16384/2048.

Following recent discussions on the -arch mailing list, involving dillon
and mckusick, this change parallels the one made over a decade ago when
the default was bumped up from 4096/512.

This should provide significant performance improvements for most
folks, less significant performance losses for a few folks and
wasted space lost to large fragments for many folks.

For discussion, please see the following thread in the -arch archive:

Subject: Using a larger block size on large filesystems

The discussion ceases to be relevant when the issue of partitioning
schemes is raised.

Remove support for FreeBSD/tahoe

Submitted by: phk

style(9) cleanup.

Submitted by: j mckitrick <jcm@freebsd-uk.eu.org>
Reviewed by: phk, /sbin/md5

Fix diskless clients by removing the code for calculating the minimum
value for cpg. The change was bogus.

Submitted by: bde
MFC after: 2 days

Following the discussion in -arch and the submission of a patch by bde, here
it is. I added the manpage change.

Submitted by: bde
MFC after: 1 week

Silence non-constant format string warnings by marking functions
as __printflike()/__printf0like(), adding const, or adding missing "%s"
format strings, as appropriate.

MFC after: 2 weeks

A more complete removal of MFS related code.

XXX: This program badly needs a style(9) + BDECFLAGS treatment.

Initial cleanout of MFS from newfs. More complete wash needed.

sprintf() -> snprintf()

Partially submitted by: "Andrew R. Reiter" <arr@watson.org>
Obtained from: OpenBSD

Add a missing argument to an error message format string.

Directory layout preference improvements from Grigoriy Orlov <gluk@ptci.ru>.
His description of the problem and solution follow. My own tests show
speedups on typical filesystem intensive workloads of 5% to 12% which
is very impressive considering the small amount of code change involved.

------

One day I noticed that some file operations run much faster on
small file systems then on big ones. I've looked at the ffs
algorithms, thought about them, and redesigned the dirpref algorithm.

First I want to describe the results of my tests. These results are old
and I have improved the algorithm after these tests were done. Nevertheless
they show how big the perfomance speedup may be. I have done two file/directory
intensive tests on a two OpenBSD systems with old and new dirpref algorithm.
The first test is "tar -xzf ports.tar.gz", the second is "rm -rf ports".
The ports.tar.gz file is the ports collection from the OpenBSD 2.8 release.
It contains 6596 directories and 13868 files. The test systems are:

1. Celeron-450, 128Mb, two IDE drives, the system at wd0, file system for
test is at wd1. Size of test file system is 8 Gb, number of cg=991,
size of cg is 8m, block size = 8k, fragment size = 1k OpenBSD-current
from Dec 2000 with BUFCACHEPERCENT=35

2. PIII-600, 128Mb, two IBM DTLA-307045 IDE drives at i815e, the system
at wd0, file system for test is at wd1. Size of test file system is 40 Gb,
number of cg=5324, size of cg is 8m, block size = 8k, fragment size = 1k
OpenBSD-current from Dec 2000 with BUFCACHEPERCENT=50

You can get more info about the test systems and methods at:
http://www.ptci.ru/gluk/dirpref/old/dirpref.html

Test Results

tar -xzf ports.tar.gz rm -rf ports
mode old dirpref new dirpref speedup old dirprefnew dirpref speedup
First system
normal 667 472 1.41 477 331 1.44
async 285 144 1.98 130 14 9.29
sync 768 616 1.25 477 334 1.43
softdep 413 252 1.64 241 38 6.34
Second system
normal 329 81 4.06 263.5 93.5 2.81
async 302 25.7 11.75 112 2.26 49.56
sync 281 57.0 4.93 263 90.5 2.9
softdep 341 40.6 8.4 284 4.76 59.66

"old dirpref" and "new dirpref" columns give a test time in seconds.
speedup - speed increasement in times, ie. old dirpref / new dirpref.

------

Algorithm description

The old dirpref algorithm is described in comments:

/*
* Find a cylinder to place a directory.
*
* The policy implemented by this algorithm is to select from
* among those cylinder groups with above the average number of
* free inodes, the one with the smallest number of directories.
*/

A new directory is allocated in a different cylinder groups than its
parent directory resulting in a directory tree that is spreaded across
all the cylinder groups. This spreading out results in a non-optimal
access to the directories and files. When we have a small filesystem
it is not a problem but when the filesystem is big then perfomance
degradation becomes very apparent.

What I mean by a big file system ?

1. A big filesystem is a filesystem which occupy 20-30 or more percent
of total drive space, i.e. first and last cylinder are physically
located relatively far from each other.
2. It has a relatively large number of cylinder groups, for example
more cylinder groups than 50% of the buffers in the buffer cache.

The first results in long access times, while the second results in
many buffers being used by metadata operations. Such operations use
cylinder group blocks and on-disk inode blocks. The cylinder group
block (fs->fs_cblkno) contains struct cg, inode and block bit maps.
It is 2k in size for the default filesystem parameters. If new and
parent directories are located in different cylinder groups then the
system performs more input/output operations and uses more buffers.
On filesystems with many cylinder groups, lots of cache buffers are
used for metadata operations.

My solution for this problem is very simple. I allocate many directories
in one cylinder group. I also do some things, so that the new allocation
method does not cause excessive fragmentation and all directory inodes
will not be located at a location far from its file's inodes and data.
The algorithm is:
/*
* Find a cylinder group to place a directory.
*
* The policy implemented by this algorithm is to allocate a
* directory inode in the same cylinder group as its parent
* directory, but also to reserve space for its files inodes
* and data. Restrict the number of directories which may be
* allocated one after another in the same cylinder group
* without intervening allocation of files.
*
* If we allocate a first level directory then force allocation
* in another cylinder group.
*/

My early versions of dirpref give me a good results for a wide range of
file operations and different filesystem capacities except one case:
those applications that create their entire directory structure first
and only later fill this structure with files.

My solution for such and similar cases is to limit a number of
directories which may be created one after another in the same cylinder
group without intervening file creations. For this purpose, I allocate
an array of counters at mount time. This array is linked to the superblock
fs->fs_contigdirs[cg]. Each time a directory is created the counter
increases and each time a file is created the counter decreases. A 60Gb
filesystem with 8mb/cg requires 10kb of memory for the counters array.

The maxcontigdirs is a maximum number of directories which may be created
without an intervening file creation. I found in my tests that the best
performance occurs when I restrict the number of directories in one cylinder
group such that all its files may be located in the same cylinder group.
There may be some deterioration in performance if all the file inodes
are in the same cylinder group as its containing directory, but their
data partially resides in a different cylinder group. The maxcontigdirs
value is calculated to try to prevent this condition. Since there is
no way to know how many files and directories will be allocated later
I added two optimization parameters in superblock/tunefs. They are:

int32_t fs_avgfilesize; /* expected average file size */
int32_t fs_avgfpdir; /* expected # of files per directory */

These parameters have reasonable defaults but may be tweeked for special
uses of a filesystem. They are only necessary in rare cases like better
tuning a filesystem being used to store a squid cache.

I have been using this algorithm for about 3 months. I have done
a lot of testing on filesystems with different capacities, average
filesize, average number of files per directory, and so on. I think
this algorithm has no negative impact on filesystem perfomance. It
works better than the default one in all cases. The new dirpref
will greatly improve untarring/removing/coping of big directories,
decrease load on cvs servers and much more. The new dirpref doesn't
speedup a compilation process, but also doesn't slow it down.

Obtained from: Grigoriy Orlov <gluk@ptci.ru>

Allow enabling soft updates (with -U) on a new filesystem.

[I first added this functionality, and thought to check prior art. Seeing
OpenBSD had already done this, I changed my addition to reduce the diffs
between the two and went with their option letter.]
Obtained from: OpenBSD

The common wisdom is to use the largest number of cylinders per group.
So bump the default from `16' to `22', which is the largest value allowed
with the current default block size. This change increases the the
group size from 32MB/g to 44MB/g on a 4GB SCSI disk.

Make mount_mfs annoy users for 15 seconds and point them at mdconfig(8).

Don't try to do anything with the /dev/rXXX device.

$Id$ -> $FreeBSD$

Add again the ':' after the x option in th eargument list to getopt.

It disappeared in rev. 1.23 newfs.c

PR: 12292
Submitted by: Cy Schubert <cy@cschuber.net.gov.bc.ca>

Fix bug in mount_mfs whereby mount_mfs would sometimes return before
the mount is completely active, causing the next few commands attempting
to manipulate data on the mount to fail. mount_mfs's parent now tries
to wait for the mount point st_dev to change before returning, indicating
that the mount has gone active.

Backed out previous commit. It broke fsck again. See rev.1.22 and the
references there, and rev.1.38 of sys/ufs/ufs/ufs_disksubr.c.

Don't rewrite the disk label. The type field is already set correctly
and we don't use the frags info, so why bother? More to the point, it
seems to result in an EXDEV error when the label is written out and we
lose because of it (don't know why though). This is a work-around and
is marked as such.

Correct source file corruption in last checkin

Observed by: jkh

Don't require an argument for -v flag
Correct checks for null special file names
Add Usage entry for -v flag
Get terminology straight in man page
Reviewed by: bde

Reviewed by: bde,jkh

Add -v flag to newfs:

-v Specify that the partition does not contain any slices, and that
newfs should treat the whole partition as the file system. This
option is useful for synthetic disks such as ccd and vinum.

Backed out rev.1.9 (except don't bring back the vax code deleted
in rev.1.9). fsck uses the per-partition ffs-related information
in the label to find alternate superblocks when the main superblock
is hosed. Rev.1.9 broke this by deleting the code that wrote the
label.

PR: 2537
xref: fsck/setup.c rev.1.8

Make it compile again in the !__STDC__ case.
Found by: Bruce.

Add prototypes. Check malloc() return value. Use err(). Remove unused #includes
Do not \n nor dot terminate syslog()/err() messages. -Wall.

Fixed printf format errors.

Fixed some spelling errors.

Allow use of the name "swap" instead of an actual swap device.
This makes configuration of mfs /tmp on diskless clients more intuitive
for people like me, that have used this feature on NetBSD and SunOS.
Using the -T option and /dev/null, while already supported,
is neither intuitive nor documented in the handbook.
Obtained from: NetBSD

compare return value from getopt against -1 rather than EOF, per the final
posix standard on the topic.

Merge from Lite2:
- use new getvfsbyname() and mount(2) interface (mount_mfs)
- use new fs include files
- updated inode / cg layout calculations (?)

This update adds the support for != 512 byte sector SCSI devices to
the sd & od drivers. There is also slight changes to fdisk & newfs
in order to comply with different sectorsizes.
Currently sectors of size 512, 1024 & 2048 are supported, the only
restriction beeing in fdisk, which hunts for the sectorsize of
the device.
This is based on patches to od.c and the other system files by
John Gumb & Barry Scott, minor changes and the sd.c patches by
me.
There also exist some patches for the msdos filesys code, but I
havn't been able to test those (yet).

John Gumb (john@talisker.demon.co.uk)
Barry Scott (barry@scottb.demon.co.uk)

Second try: attempt to import Lite2's newfs.

Add hooks into the mount_mfs code in newfs to do the FreeBSD-style
LKM loading if it was not configured into the system.

Note that the LKM for MFS is not enabled by default, but I got it working on
my machine.. I'll see what I did..

Shorten a variable name.

Avoid the "calculated sectors per cylinder disagrees with disklabel"
warning for the default case where the user hasn't specified either -t
or -u on the command line. It's been confusing our users.

Fixed error in maxcontig calculation that caused it to default to "1".

Yank out the rewriting of disklabels. This code can and will get confused
in a couple of cases, and it doesn't do much anyway. It used to save only
the newfs params (block/frag/cgroup.. and nothing more. Something that
don't belong in a disklabel in the first place.

Allow zero as value for certain arguments to indicate "take from disklabel".

Change the defaults for newfs to disregard the geometry in the disklabel.

We pretend we have one head with two megabyte worth of sectors per cylinder.

The code try to access another head in what it belives to the same
physical cylinder, because it belives that it would be faster than
waiting for the next free sector under this head to come around.

Most modern drives doesn't have a "classical" geometry, and thus
we end up fooling ourselves doing the above optimization. With this
change we will fill a cylinder sequentially if we can, and thus get
much more mileage from the track-buffer/cache built into the drives.

As a result a lot of seeks to the next or previous track should be
avoided by this.

(My disk is a lot less noisy actually...)

You can still get the old behaviour, by specifying zero for the
numbers.

This will also solve the problem with newfs barfing at really big
drives.

Obtained from: adult advice from Kirk.

Add support for filesystem-specific `-o' options, and re-implement the
most common cd9660 and nfs options like God intended them. (It is now
possible to say

mount -o ro,soft,bg,intr there:/foo/bar /foo/bar

again.) This whole getmntopt() business is an incredible botch;
it never should have been anything more than a wrapper around
getsubopt(3). Because if the way the current hackaround is implemented,
options which take arguments (like the old `rsize' and `wsize') are still
unavailable, and must be accessed the new, broken way.

(It's unimaginable how Berkeley managed to screw up one of the few things
about NFS that Sun actually got right to begin with!)

Put back the `:' in the trinary ?: so this can actually compile again! :)

Added '-F file' option of mount_mfs. This allows me to make floppy images
without waiting for my floppy-drive all the time :-) Might have other
interesting uses too.

Backed out part of the last change that prevents the rpos table from
being output if <= 1 rpos; there is a bug in the kernel which doesn't
quite get along with this. Changed default #rpos to 1, and fixed up
manual page. Converted nrpos to 1 if user specifies 0.

1) If nrpos <= 1, don't output rpos table (and set fs_cpc to 0) - disabling
the use of the rotational position table.
2) Allow specification of 0 rotational positions (disables function).
3) Make rotdelay=0 and nrpos=0 by default.

The purpose of the above is to optimize for modern SCSI (and IDE) drives
that do read-ahead/write-behind.

BSD 4.4 Lite sbin Sources

Note: XNSrouted and routed NOT imported here, they shall be imported with
usr.sbin.