Enable soft updates by default for UFS2 filesystems.

Soft updates dramatically improve the performance of UFS filesystems.
The newfs(8) utility currently does not enable them by default. The
FreeBSD installer enables soft updates by default. However custom
built installations that do not specify the -U option to newfs(8)
and the prebuilt UFS system images get filesystems without soft
updates enabled.

There are several testing sites that run benchmarks comparing the
performance of Linux distributions versus BSD distributions. When
they run filesystem comparison benchmarks they use newfs(8) to
create the UFS filesystem. Because it does not have soft updates
enabled it runs poorly versus the Linux ext4 filesystem. When I
have suggested to them that they should enable soft updates on the
UFS filesystem in their testing their response is that they expect
the utility that creates the filesystem to use optimal defaults and
that they cannot be expected to fiddle with various option settings.

The purpose of this change is to give a filesystem created with
newfs(8) reasonably optimal settings. For UFS2 this means enabling
soft updates. For UFS1 which tends to be used on small systems with
minimal memory and CPU speed, the lower memory footprint of running
without soft updates is a more sensible default.

This change adds a note in the section of the newfs(8) manual page
that describes the -U option for enabling soft updates that they
are enabled by default for UFS2 filesystems and that they can be
disabled by using tunefs(8).

Reviewed-by: Warner Losh, kib
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D45201

sbin: Remove ancient SCCS tags.

Remove ancient SCCS tags from the tree, automated scripting, with two
minor fixup to keep things compiling. All the common forms in the tree
were removed with a perl script.

Sponsored by: Netflix

Remove $FreeBSD$: one-line nroff pattern

Remove /^\.\\"\s*\$FreeBSD\$$\n/

Update newfs(8) and tunefs(8) manual pages.

Delete note that UFS/FFS filesystems running with journaled soft updates
cannot run background fsck as 344b5bf made it possible to do so.

MFC-with: 344b5bf
Sponsored-by: The FreeBSD Foundation

Enable taking snapshots on UFS/FFS filesystems using journaled soft updates.

All the needed infrastructure updates have been made to allow
snapshots to be taken on UFS/FFS filesystems that are using journaled
soft updates. The most immediate benefit is the ability to use a
snapshot to take a consistent filesystem dump on a live filesystem
using the -L option to dump(8).

Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D36491

Add a description of soft updates journaling to newfs(8).

Add a descrition to the newfs(8) -j (journal enablement) flag
that explains what soft updates journaling does, the tradeoffs
to using it, and the limitations that it imposes. Copied from
the description in tunefs(8).

PR: 261944
Sponsored by: The FreeBSD Foundation

The fdformat man page is in section 8 (not 1).

Explain the newfs naming convention

It might be unclear why newfs and newfs_msdos should cross-reference
each other. Add a note explaining it.

This is a follow-up to 74bd20769706041108a573601cf0b61c755bdc56.

Reported by: kib
Reviewed by: imp, kib, rpokala
MFC after: 3 days

Reference newfs_msdos(8) from the newfs(8) manual

PR: 252484
Reported by: Graham Perrin <grahamperrin@gmail.com>
MFC after: 3 days

newfs(8): Fix unusual Xr order

- unusual Xr order: gjournal after gpart

MFC after: 3 days

Correct the location of the first backup superblock in fsck_ffs.8.
Make a note in the newfs.8 manual page to update the first backup
superblock location when changing the default fragment size for
the filesystem.

Reported by: O. Hartmann

Allow dashes as a valid character in UFS labels.

Reviewed by: mckusick, imp, 0mp
MFC after: 2 weeks
Differential Revision: D18991

Allow tunefs to include '_' as a legal character in label names
to make it consistent with newfs. Document the legality of '_'
in label names in both tunefs(8) and newfs(8).

PR: 235182
Submitted by: darius@dons.net.au
Reviewed by: Conrad Meyer
MFC after: 3 days
Sponsored by: Netflix

Bump date for today's commit.

Improve wording for -E and -t flags. -E never writes the entire disk,
so don't imply that. Note that if BIO_DELETE isn't supported, the
operation will fail (as opposed to writing the entire disk with
zeros). Thin storage also benefits from trim. List more accurate
reason why trim helps flash-memory.

Renumber copyright clause 4

Renumber cluase 4 to 3, per what everybody else did when BSD granted
them permission to remove clause 3. My insistance on keeping the same
numbering for legal reasons is too pedantic, so give up on that point.

Submitted by: Jan Schaumann <jschauma@stevens.edu>
Pull Request: https://github.com/freebsd/freebsd/pull/96

It's 2015, and some people are still trying to use fdisk and then
go asking what debug flags to set for GEOM to make it work. Advice
them to use gpart(8) instead.

Something similar should probably done with disklabel,
but I need to rewrite the disklabel examples first.

Reviewed by: wblock@
MFC after: 1 month
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D3315

fix the docs, the number of frags per inode (NFPI) changed in r228794
to 2 from 4, but the man page didn't get updated...

other minor changes to make igor happy...

MFC after: 3 days

mdoc: improvements to SEE ALSO.

Sweep man pages replacing ad -> ada.

Approved by: re (blackend)
MFC after: 1 week
X-MFC note: stable/9 only

Remove contractions.

The purpose of this change to the FFS layout policy is to reduce the
running time for a full fsck. It also reduces the random access time
for large files and speeds the traversal time for directory tree walks.

The key idea is to reserve a small area in each cylinder group
immediately following the inode blocks for the use of metadata,
specifically indirect blocks and directory contents. The new policy
is to preferentially place metadata in the metadata area and
everything else in the blocks that follow the metadata area.

The size of this area can be set when creating a filesystem using
newfs(8) or changed in an existing filesystem using tunefs(8).
Both utilities use the `-k held-for-metadata-blocks' option to
specify the amount of space to be held for metadata blocks in each
cylinder group. By default, newfs(8) sets this area to half of
minfree (typically 4% of the data area).

This work was inspired by a paper presented at Usenix's FAST '13:
www.usenix.org/conference/fast13/ffsck-fast-file-system-checker

Details of this implementation appears in the April 2013 of ;login:
www.usenix.org/publications/login/april-2013-volume-38-number-2.
A copy of the April 2013 ;login: paper can also be downloaded
from: www.mckusick.com/publications/faster_fsck.pdf.

Reviewed by: kib
Tested by: Peter Holm
MFC after: 4 weeks

Advertise growfs(8) a little better.

Update the manual page to reflect the new 32K/4K defaults.

Reminded by: Ivan Voras

Xref makefs(8).

PR: 154708
Submitted by: jhs

Add the -j option to enable soft updates journaling when creating
a new file system.

Reviewed by: Kostik Belousov <kostikbel@gmail.com>

Add support for FS_TRIM to user-mode UFS utilities.

Reviewed by: mckusick, pjd, pho
Tested by: pho
MFC after: 1 month

A new jail(8) with a configuration file, to replace the work currently done
by /etc/rc.d/jail.

Synchronize newfs(8) manual with code.

PR: 61716
Submitted by: Radim Kolar <hsn at netmag cz>
Patch by: arundel
Approved by: keramida (mentor)
MFC after: 1 week

Rewording and typo fixes in newfs(8).

PR: 150490
Submitted by: Eitan Adler <foreignuser at eitanadler com>
Additional fixes by: Warren Block <wblock at wonkity com>, keramida
Approved by: keramida (mentor)
MFC after: 1 week

Create the altix project branch. The altix project will add support
for the SGI Altix 350 to FreeBSD/ia64. The hardware used for porting
is a two-module system, consisting of a base compute module and a
CPU expansion module. SGI's NUMAFlex architecture can be an excellent
platform to test CPU affinity and NUMA-aware features in FreeBSD.

Fix "Empty input line" mdoc warning.

Submitted by: Alexander Best

Slightly improve gjournal documentation.

Reviewed by: pjd

Enable operation of newfs on plain files, which is useful when you
want to prepare disk images for emulators (though 'makefs' in port
can do something similar).

This relies on:
+ minor changes to pass the consistency checks even when working on a file;

+ an additional option, '-p partition' , to specify the disk partition to
initialize;

+ some changes on the I/O routines to deal with partition offsets.

The latter was a bit tricky to implement, see the details in newfs.h:
in newfs, I/O is done through libufs which assumes that the file
descriptor refers to the whole partition. Introducing support for
the offset in libufs would require a non-backward compatible change
in the library, to be dealt with a version bump or with symbol
versioning.

I felt both approaches to be overkill for this specific application,
especially because there might be other changes to libufs that might
become necessary in the near future.

So I used the following trick:
- read access is always done by calling bread() directly, so we just add
the offset in the (few) places that call bread();
- write access is done through bwrite() and sbwrite(), which in turn
calls bwrite(). To avoid rewriting sbwrite(), we supply our own version
of bwrite() here, which takes precedence over the version in libufs.

MFC after: 4 weeks

- Import the HEAD csup code which is the basis for the cvsmode work.

Replace reference from vinum.8 to gvinum.8, it was advised in the PR to
replace this with vinum.4, but that's the kernel interface manual, which
is not appropriate in my understanding. I think that gvinum is a suitable
replacement for this.

PR: docs/121938
Submitted by: "Federico" <federicogalvezdurand at yahoo dot com>
MFC after: 3 days

Rename the undocumented -E option to -X.

Implement -E option which will erase the filesystem sectors before
making the new filesystem. Reserved space in front of the superblock
(bootcode) is not erased.

NB: Erasing can take as long time as writing every sector sequentially.

This is relevant for all flash based disks which use wearlevelling.

MFp4:

Add a new option to newfs(8), -r, to specify reserved space at the
end of the device. It can be useful, e.g., when the device is to
become a member of a gmirror array later w/o losing the file system
on it.

Document the new option in the manpage.

While I'm here, improve error handling for -s option, which is
syntactically similar to -r; and document the fact that -s0 selects
the default fs size explicitly, which can be useful, e.g., in a
menu-based wrapper around newfs(8) requiring some value be entered
for the fs size.

Also fix a small typo in the help line for -s (missing space).

Idea and initial implementation by: marck
Discussed on: -fs
Critical review by: bde
Tested with: cmp(1)

Add -J flag to both newfs(8) and tunefs(8) which allows to enable gjournal
support.
I left -j flag for UFS journal implementation which we may gain at some
point.

Sponsored by: home.pl

Polish previous revision:

- Bump document date.
- Spell "file system" properly.
- Add missing markup bits.

Add an option to suppress the creation of the .snap directory in
the new filesystem. This is intended for memory and vnode filesystems
that will never be fsck'ed or dumped.

Obtained from: St. Bernard Software RAPID
MFC after: 2 weeks

Assorted markup, grammar, and spelling fixes.

Remove advertising clause from University of California Regent's license,
per letter dated July 22, 1999.

Approved by: core, imp

Add a "-l" flag to newfs, which sets the FS_MULTILABEL flag. This
permits users of newfs to set the multilabel flag on UFS1 and UFS2
file systems from inception without using tunefs.

Obtained from: TrustedBSD Project
Sponsored by: DARPA, McAfee Research

s/disklabel/bsdlabel where needed.

Remove an unneccessary comma.

mdoc(7): Use the new feature of the .In macro.

Remove reference to diskpart(8)

PR: 51193
Submitted by: Yonatan@xpert.com

Throw the switch--change to UFS2 as our default file system format for
FreeBSD 5.1-RELEASE and later:

- newfs(8) will now create UFS2 file systems unless UFS1 is specifically
requested (-O1). To do this, I just twiddled the Oflag default.

- sysinstall(8) will now select UFS2 as the default layout for new
file systems unless specifically requested (use '1' and '2' to change
the file system layout in the disk labeler). To do this, I inverted
the ufs2 flag into a ufs1 flag, since ufs2 is now the default and
ufs1 is the edge case. There's a slight semantic change in the
key behavior: '2' no longer toggles, it changes the selection to UFS2.

This is very similar to a patch David O'Brien sent me at one point, and
that I couldn't find.

Approved by: re (telecon)
Reviewed by: mckusick, phk, bmah

mdoc(7) police: markup laundry.

Our first keyword hit for apropos ufs2.

Bring in support for volume labels to the filesystem utilities.

Reviewed by: mckusick

Fix some 'SYNOPSIS' and 'usage' messages.

s/filesystem/file system/g as discussed on -developers

mdoc(7) police: Consistently name options' arguments in the SYNOPSIS and DESCRIPTION sections..

The .Nm utility

This commit adds basic support for the UFS2 filesystem. The UFS2
filesystem expands the inode to 256 bytes to make space for 64-bit
block pointers. It also adds a file-creation time field, an ability
to use jumbo blocks per inode to allow extent like pointer density,
and space for extended attributes (up to twice the filesystem block
size worth of attributes, e.g., on a 16K filesystem, there is space
for 32K of attributes). UFS2 fully supports and runs existing UFS1
filesystems. New filesystems built using newfs can be built in either
UFS1 or UFS2 format using the -O option. In this commit UFS1 is
the default format, so if you want to build UFS2 format filesystems,
you must specify -O 2. This default will be changed to UFS2 when
UFS2 proves itself to be stable. In this commit the boot code for
reading UFS2 filesystems is not compiled (see /sys/boot/common/ufsread.c)
as there is insufficient space in the boot block. Once the size of the
boot block is increased, this code can be defined.

Things to note: the definition of SBSIZE has changed to SBLOCKSIZE.
The header file <ufs/ufs/dinode.h> must be included before
<ufs/ffs/fs.h> so as to get the definitions of ufs2_daddr_t and
ufs_lbn_t.

Still TODO:
Verify that the first level bootstraps work for all the architectures.
Convert the utility ffsinfo to understand UFS2 and test growfs.
Add support for the extended attribute storage. Update soft updates
to ensure integrity of extended attribute storage. Switch the
current extended attribute interfaces to use the extended attribute
storage. Add the extent like functionality (framework is there,
but is currently never used).

Sponsored by: DARPA & NAI Labs.
Reviewed by: Poul-Henning Kamp <phk@freebsd.org>

more file system > filesystem

Remove the -v option, it is now default behaviour.

Sponsored by: DARPA & NAI Labs

Swing the axe and remove some archaic features from newfs which modern
diskdrives do neither need nor want:

-O create a 4.3BSD format filesystem
-d rotational delay between contiguous blocks
-k sector 0 skew, per track
-l hardware sector interleave
-n number of distinguished rotational positions
-p spare sectors per track
-r revolutions/minute
-t tracks/cylinder
-x spare sectors per cylinder

No change in the produced filesystem image unless one or more of
these options were used.

Approved by: mckusick

Update the default newfs block and fragment sizes from 8192/1024 to
16384/2048.

Following recent discussions on the -arch mailing list, involving dillon
and mckusick, this change parallels the one made over a decade ago when
the default was bumped up from 4096/512.

This should provide significant performance improvements for most
folks, less significant performance losses for a few folks and
wasted space lost to large fragments for many folks.

For discussion, please see the following thread in the -arch archive:

Subject: Using a larger block size on large filesystems

The discussion ceases to be relevant when the issue of partitioning
schemes is raised.

Fix typo: 'fragement' -> 'fragment'

Fix the example of suggested default settings. It stated that settings
were only of benefit to large filesystems, which recent research
suggests is not the case, and which the original author of the text
no longer endorses.

Correct the example introduced in rev 1.29, which suggested a block:frag
size ratio other than 8:1. Currently, we only recommend an 8:1
ratio, because the impact of others ratios has not been adequately
investigated.

Also, do not recommend the use of the -c option in the example, since
newfs now automatically calculates the best cyl:cylgrp ratio.

This change was discussed with the author of rev 1.29.

Document the optimal block:fragment ratio, per discussion
on -arch and cvs-all.

Reviewed by: dillon

Following the discussion in -arch and the submission of a patch by bde, here
it is. I added the manpage change.

Submitted by: bde
MFC after: 1 week

Remove whitespace at EOL.

mdoc(7) police: removed HISTORY info from the .Os call.

mdoc(7) police: remove extraneous .Pp before and/or after .Sh.

Remove all references to MFS.

Add information about the new options to newfs and tunefs which set the
expected average file size and number of files per directory. Could do
with some fleshing out.

Document the newfs.c rev 1.33 changing the default c/g from 16 to 22.

Fix patch merge braino.

Allow enabling soft updates (with -U) on a new filesystem.

[I first added this functionality, and thought to check prior art. Seeing
OpenBSD had already done this, I changed my addition to reduce the diffs
between the two and went with their option letter.]
Obtained from: OpenBSD

Make mount_mfs annoy users for 15 seconds and point them at mdconfig(8).

mdoc(7) police: fixed broken references.

Replace reference to replacing mkfs(8) with a paragraph actually
describing what newfs *does*.

Prepare for mdoc(7)NG.

o Add an example for a large file system.
o Remove bug about boot blocks hating non-8k file systems. This hasn't been
the case for a long time.

Not Objected to by: hackers, doc

mdoc(7) police: use the new features of the Nm macro.

Avoid use of direct troff requests in mdoc(7) manual pages.

Remove single-space hard sentence breaks. These degrade the quality
of the typeset output, tend to make diffs harder to read and provide
bad examples for new-comers to mdoc.

Add Xref to camcontrol(8) (replacing previously-removed scsiformat(8)).

Submitted by: joerg

Fix various man pages to stop abusing the .Bx macro to generate
the strings "FreeBSD" and "NetBSD". Use the .Fx or .Nx macro
instead.

Remove dead xref to scsiformat(8)

Obtained from: OpenBSD (kind of)

$Id$ -> $FreeBSD$

Describe the default values for useful options.

Clarify which options are no longer useful.

Straightened the terminology straightening in 1.17-1.18. Fixed hard
line breaks in rev.1.16-1.18.

Refer to "da" rather than "sd" device.

Don't require an argument for -v flag
Correct checks for null special file names
Add Usage entry for -v flag
Get terminology straight in man page
Reviewed by: bde

Reviewed by: bde (again)

Correct terminology (partitions are in slices, not the other way around)

Reviewed by: bde,jkh

Add -v flag to newfs:

-v Specify that the partition does not contain any slices, and that
newfs should treat the whole partition as the file system. This
option is useful for synthetic disks such as ccd and vinum.

Add prototypes. Check malloc() return value. Use err(). Remove unused #includes
Do not \n nor dot terminate syslog()/err() messages. -Wall.

Add warning about root-fs blocksize expectations.
PR: 4485
Reviewed by: phk
Submitted by: Kees Jan Koster <kjk1@ukc.ac.uk>

Fix syntax error for the mount_mfs example.
Also update to a more realistic example.

Fix typo.
PR: Closes PR docs/3488
Submitted by: k-horik@yk.rim.or.jp

Merge from Lite2:
- use new getvfsbyname() and mount(2) interface (mount_mfs)
- use new fs include files
- updated inode / cg layout calculations (?)

Replace "mfs" with "mount_mfs" in the NAME section.

Add example for mount_mfs(8).

Update the newfs(8) man page to reflect current
default values for some options better. Closes PR# 1374.

Second try: attempt to import Lite2's newfs.

Fix a bunch of spelling errors.

Change the reference to format(8) (which we don't have) to fdformat(1) and
scsiformat(8) (which we do have). Closes PR# 663.

Change the defaults for newfs to disregard the geometry in the disklabel.

We pretend we have one head with two megabyte worth of sectors per cylinder.

The code try to access another head in what it belives to the same
physical cylinder, because it belives that it would be faster than
waiting for the next free sector under this head to come around.

Most modern drives doesn't have a "classical" geometry, and thus
we end up fooling ourselves doing the above optimization. With this
change we will fill a cylinder sequentially if we can, and thus get
much more mileage from the track-buffer/cache built into the drives.

As a result a lot of seeks to the next or previous track should be
avoided by this.

(My disk is a lot less noisy actually...)

You can still get the old behaviour, by specifying zero for the
numbers.

This will also solve the problem with newfs barfing at really big
drives.

Obtained from: adult advice from Kirk.

Added '-F file' option of mount_mfs. This allows me to make floppy images
without waiting for my floppy-drive all the time :-) Might have other
interesting uses too.

Backed out part of the last change that prevents the rpos table from
being output if <= 1 rpos; there is a bug in the kernel which doesn't
quite get along with this. Changed default #rpos to 1, and fixed up
manual page. Converted nrpos to 1 if user specifies 0.

Fixed manpage to conform to current reality.

Submitted by: Rod Grimes, with additional sentence by me.

BSD 4.4 Lite sbin Sources

Note: XNSrouted and routed NOT imported here, they shall be imported with
usr.sbin.