mdconfig: Print options when listing devices

PR: 278624
Reviewed by: imp
Pull Request: https://github.com/freebsd/freebsd-src/pull/1210

Remove $FreeBSD$: two-line .h pattern

Remove /^\s*\*\n \*\s+\$FreeBSD\$$\n/

Mark usage function as __dead2 in programs where it does not return

In most cases, usage does not return, so mark them as __dead2. For the
cases where they do return, they have not been marked __dead2.

Reviewed by: imp
Pull Request: https://github.com/freebsd/freebsd-src/pull/735

spdx: The BSD-2-Clause-FreeBSD identifier is obsolete, drop -FreeBSD

The SPDX folks have obsoleted the BSD-2-Clause-FreeBSD identifier. Catch
up to that fact and revert to their recommended match of BSD-2-Clause.

Discussed with: pfg
MFC After: 3 days
Sponsored by: Netflix

md: Add MD_MUSTDEALLOC support

This adds an option to detect if hole-punching is implemented by the
underlying file system. If this flag is set, and if the underlying file
system does not support hole-punching, md(4) fails BIO_DELETE requests
with EOPNOTSUPP.

Sponsored by: The FreeBSD Foundation
Reviewed by: markj
Differential Revision: https://reviews.freebsd.org/D31883

[skip ci] fix a typo in a comment in mdconfig.c

Sponsored by: Axcient
Reviewed by: mav, imp
MFC after: 2 weeks
Differential Revision: https://reviews.freebsd.org/D28968

Use VOP_ADVISE() with POSIX_FADV_DONTNEED instead of IO_DIRECT to
implement not double-caching for reads from vnode-backed md devices.
Use VOP_ADVISE() similarly instead of !IO_DIRECT unsimilarly for writes.
Add a "cache" option to mdconfig to allow changing the default of not
caching.

This depends on a recent commit to fix VOP_ADVISE(). A previous version
had optimizations for sequential i/o's (merge the i/o's and only uncache
for discontiguous i/o's and for full blocks), but optimizations and
knowledge of block boundaries belong in VOP_ADVISE(). Read-ahead should
also be handled better, by supporting it in md and discarding it in
VOP_ADVISE().

POSIX_FADV_DONTNEED is ignored by zfs, but so is IO_DIRECT.

POSIX_FADV_DONTNEED works better than IO_DIRECT if it is not ignored,
since it only discards from the buffer cache immediately, while
IO_DIRECT also discards from the page cache immediately.

IO_DIRECT was not used for writes since it was claimed to be too slow,
but most of the slowness for writes is from doing them synchronously by
default. Non-synchronous writes still deadlock in many cases.

IO_DIRECT only has a special implementation for ffs reads with DIRECTIO
configured. Otherwise, if it is not ignored than it uses the buffer and
page caches normally except for discarding everything after each i/o,
and then it has much the same overheads as POSIX_FADV_DONTNEED. The
overheads for reading with ffs and DIRECTIO were similar in tests of md.

Reviewed by: kib

Fix missing (sub)options in usage message to prepare for adding a new one.

Reviewed by: kib

Make (no)ro an alias for (no)readonly

various: general adoption of SPDX licensing ID tags.

Mainly focus on files that use BSD 2-Clause license, however the tool I
was using misidentified many licenses so this was mostly a manual - error
prone - task.

The Software Package Data Exchange (SPDX) group provides a specification
to make it easier for automated tools to detect and summarize well known
opensource licenses. We are gradually adopting the specification, noting
that the tags are considered only advisory and do not, in any way,
superceed or replace the license texts.

No functional change intended.

Add ability to label md(4) devices.

This feature comes from the fact that we rely memory-backed md(4)
in our build process heavily. However, if the build goes haywire
the allocated resources (i.e. swap and memory-backed md(4)'s) need
to be purged. It is extremely useful to have ability to attach
arbitrary labels to each of the virtual disks so that they can
be identified and GC'ed if neecessary.

MFC after: 4 weeks
Differential Revision: https://reviews.freebsd.org/D10457

Add MD_VERIFY option to enable O_VERIFY in open for vnode type.
Add -o [no]verify option to mdconfig (and document in man page.)
Implement GEOM attribute MNT::verified to ask md if the backing vnode is
verified.
Check for MNT::verified in cd9660 mount to flag the mount as MNT_VERIFIED if
the underlying device has been verified.

Reviewed by: rwatson
Approved by: sjg (mentor)
Obtained from: Juniper Networks, Inc.
Differential Revision: https://reviews.freebsd.org/D2902

Make mdconfig(8) not segfault when the kernel returns garbage data.

MFC after: 2 weeks
Sponsored by: DARPA, AFRL

For MD_PRELOAD type md(4) devices, if there is a file name in the preloaded
meta-data, copy it into the softc structure.

When returning md(4) device details to the caller, include the file name in
any MD_PRELOAD type devices if it is set (first character is not NUL.)

In mdconfig, for "preload" type md(4) devices, if there is file config
available, print it in the file column of the output.

Reviewed by: brooks
Approved by: sjg (mentor)
MFC after: 1 month
Sponsored by: Juniper Networks, Inc.
Differential Revision: https://reviews.freebsd.org/D9529

Add 'p' postfix to mdconfig(8).

MFC after: 1 month
Sponsored by: The FreeBSD Foundation

Add "null" backend to mdconfig(8). This does exactly what the name
suggests, and is somewhat useful for benchmarking.

MFC after: 1 month
No objections from: kib
Sponsored by: The FreeBSD Foundation

Return 0 if:

1. "-u N" specified, no -f, and mdN found,
2. no -u, "-f /pathname" specified, and mdN associated with /pathname found,
3. "-u N" specified, "-f /pathname" specified, and both of them found,
4. "-l" specified and no -f,
5. "-l" specified, "-f /pathname" specified, and /pathname found.

otherwise return -1.

Spotted by: Julian H. Stacey

Resolve fflag with realpath().

MFC after: 2 weeks

When listing with -f, skip all memory disks that are not vnode-backed.

Noticed by: kevlo
MFC after: 3 days

- Use length of _PATH_DEV instead of a hardcoded value[*].
- Use MD_NAME for "md".

Pointed out by: trasz [*]
MFC after: 1 week

eturn -1 when the specified backing store file is not found in the md
device list.

MFC after: 1 week

- Add "-f file" support to listing mode (-l). When a -f option is
specified, only md(4) devices which have the specified file as backing
store are displayed.
- Use MD_NAME instead of "md".
- Use _PATH_DEV instead of "/dev/".

MFC after: 1 week

Disallow attaching preloaded memory disks via ioctl.

- The feature is dangerous because the kernel code didn't check
validity of the memory address provided from user space.
- It seems that mdconfig(8) never really supported attaching preloaded
memory disks.
- Preloaded memory disks are automatically attached during md(4)
initialization. Thus there shouldn't be much use for the feature.

PR: kern/169683
Discussed on: freebsd-hackers

Make it possible to resize md(4) devices.

Reviewed by: kib
Sponsored by: FreeBSD Foundation

Add missing "static".

Rewrite option parsing in mdconfig(8). This makes it more user-friendly
by removing the ordering requirements and adding more descriptive error
messages; it also makes it more readable and maintainable.

Sponsored by: The FreeBSD Foundation

Replace the beerware license on mdconfig(8) with standard 2-clause BSD.

Approved by: phk@

Somewhere around the 473rd time I mistyped "mdconfig file" instead of
"mdconfig -f file", I decided that it would be easier to make mdconfig
DWIM than to teach my fingers to type the correct command line.

MFC after: 3 weeks

whitespace nit - sorry for the churn

whitespace nit

Add a check to make sure the provider name is "mdN" before printing the
unit number.

Suggested by: jh
MFC after: 3 days

Only print the unit number when invoked with the -n flag.

PR: bin/144300
Submitted by: arundel
MFC after: 3 days

Use "cylinder" instead of "cyl" for consistency.

Reviewed by: phk

A new jail(8) with a configuration file, to replace the work currently done
by /etc/rc.d/jail.

Create the altix project branch. The altix project will add support
for the SGI Altix 350 to FreeBSD/ia64. The hardware used for porting
is a two-module system, consisting of a base compute module and a
CPU expansion module. SGI's NUMAFlex architecture can be an excellent
platform to test CPU affinity and NUMA-aware features in FreeBSD.

MFC r202573:

Print sizes up to INT64_MAX in md_prthumanval().

PR: bin/125365
Approved by: trasz (mentor)

Make mdconfig(8) WARNS=6 clean:

- Constify geom_config_get() name argument.
- Add void keyword for usage().
- Initialize mdunit to NULL.
- Don't call md_prthumanval() at all if length is NULL.

Approved by: trasz (mentor)

Print sizes up to INT64_MAX in md_prthumanval().

PR: bin/125365
Approved by: trasz (mentor)
MFC after: 2 weeks

Add the possibility to specify "-o force" with "mdconfig -du".

Reviewed by: scottl
Approved by: rwatson (mentor)
Sponsored by: FreeBSD Foundation

- Import the HEAD csup code which is the basis for the cvsmode work.

Add -v (verbose) option to -l command, to show size and backing store
of all md devices at one time.

Approved by: phk
MFC after: 2 weeks

Fix exit code when using -l on a non-existent md(4) device.

PR: conf/116177
Submitted by: Remi Guyomarch <rguyom@pobox.com>
MFC after: 3 days

Kris suggested that swap is a better choice as a default than malloc.

MFC: 1 week

[Found the original diff I made, see previous commit for other part]
Assume '-a' and '-t malloc' flags for '-s <size>' (malloc ramdisk) if not
specified.

Reviewed by: phk (some time ago)
MFC: 1 week

Make attach the default for -f. That way

mdconfig -f image

works like a charm.

Reviewed by: phk (some time ago)
MFC: 1 week

Unbreak the build.

For now, bring back some of the old bits as a fix for specifying md(4)
device number at creation time with -u option. Together with XMLizing
mdconfig(8), I broke this functionality.

This change is temporary. Complete fix will be commited soon.

Approved by: cognet (mentor)

Teach md(4) and mdconfig(8) how to understand XML. Right now there won't be
a problem with listing large number of md(4) devices. Either 'list' or
'query' mode uses XML.

Additionally, new functionality was introduced. It's possible to pass
multiple devices to -u:

# ./mdconfig -l -u md0,md1

Approved by: cognet (mentor)

Keep proper order of includes. Additionally, sort them. Make functions used
in that file static.

Approved by: cognet (mentor)

Use kld(3) for loading geom_md.ko.

Sort the list results by the unit number. The list returned by the
kernel is in the order the devices were made, which is not useful to
the user. Also, remove the "%d more" test since the kernel does not
return the complete count in md_pad[0] (maybe it should?).

Submitted by: Wojciech A. Koszek

Rename GEOM class kernel module g_md.ko to geom_md.ko for consistency
with the rest.

mdconfig.c: Simplify mdmaybeload() function.
mdioctl.h: Removed (now unused) #define.
loader.conf: Sort GEOM classes properly.

OK'ed by: phk

When using files as backing stores for devices, and the user has requested the
device be created read+write, check to see if the backing store is read only
through the use of the access(2) system call. If this check fails returning
EACCES, EPERM or EROFS then gracefully downgrade the access to read only. Also
print a warning message to stderr, informing the user that the access mode
they requested is not available.

This behavior used to be handled by md(4) but was changed in revision 1.154

Discussed with: pjd, phk, Dario Freni <saturnero at freesbie dot org>
Reviewed by: phk

We can specify device size in bytes. Document this in usage.

Document the terabyte "-s" parameter in the usage string.
Split the usage line to not exceed 80 chars.

Sync program's usage() with manpage's SYNOPSIS.

Allow to specify device size in bytes.

MFC after: 1 week

Rewrite piece of code which I committed some time ago that allows to
show file name for 'mdconfig -l -u <x>' command.
This allows to preserve API/ABI compatibility with version 0 (that's why
I changed version number back to 0) and will allow to merge this change
to RELENG_5.

MFC after: 5 days

Sync usage() with manpage SYNOPSIS and code.

For file backed md(4) devices output their source file via
'mdconfig -l -u <unit>'.
Bump version number, as this change breaks ABI/API.

If there aren't any devices to list, output nothing instead of an
empty line. This is consistent with other utilities.

While I'm here, remove artifacts of the previous list implementation.

- Make md(4) 64-bit clean.
After this change it should be possible to use very big md(4) devices.
- Clean up and simplify the code a bit.
- Use humanize_number(3) to print size of md(4) devices.
- Add 't' suffix which stands for terabyte.
- Make '-S' to really work with all types of devices.
- Other minor changes.

add support for documented readonly option...
also print out the option that is unknow so that the user knows what (s)he
did wrong..

MFC after: 3 days

change the name of the md module, to g_md, introduce a define with the
name MD_MODNAME, and make mdconfig use this new define...

Give better diagnostic for problems with backing files.

Fix a long-standing deadlock issue with vnode backed md(4) devices:

On vnode backed md(4) devices over a certain, currently undetermined
size relative to the buffer cache our "lemming-syncer" can provoke
a buffer starvation which puts the md thread to sleep on wdrain.

This generally tends to grind the entire system to a stop because the
event that is supposed to wake up the thread will not happen until a fair
bit of the piled up I/O requests in the system finish, and since a lot
of those are on a md(4) vnode backed device which is currently waiting
on wdrain until a fair amount of the piled up ... you get the picture.

The cure is to issue all VOP_WRITES on the vnode backing the device
with IO_SYNC.

In addition to more closely emulating a real disk device with a
non-lying write-cache, this makes the writes exempt from rate-limited
(there to avoid starving the buffer cache) and consequently prevents
the deadlock.

Unfortunately performance takes a hit.

Add "async" option to give people who know what they are doing the
old behaviour.

Do not compare unsigned int values with ULONG_MAX. The comparison is
always false on 64bit platforms and GCC 3.3.1 issues warning there.

Add "-n" argument, which causes mdconfig to simply print the unit
number X, rather than mdX, making it easier to script tests that
use md devices but don't want to make assumptions about any existing
md use (such as in diskless environments).

Add a couple of undocumented test options to MD(4) to aid in regression
testting of GEOM.

Give clearer diagnostics on some cases of invalid
combinations of mdconfig(8) command-line arguments.

Make mdconfig(8) accept "-a -f file -o options"
equally with "-a -f file" (assuming "-t vnode".)

Add a "-S sectorsize" option to enable Kirk to find a bug :-)

Don't use the kern.disks sysctl to find a list of md devices, use the
MDIOCLIST ioctl instead.

Spotted by: keramida

Make sure strsep() gets a nul-terminated string.

Reviewed by: phk

Don't use NULL where you really mean 0 to sysctlbyname.

Usage style sweep: spell "usage" with a small 'u'.
Also change one case of blatant __progname abuse (several more remain)
This commit does not touch anything in src/{contrib,crypto,gnu}/.

Revert 1.20:
Memory disks may be used for other purposes besides newfs(8), so it
isn't helpful to require the minimum size meet newfs(8)'s criteria.

Provide a proper error message in mdconfig(8) when a filesystem is too
small, instead of a less meaningful error in newfs(8).

Actually make use of the md_version field of 'struct mdio'. In order
not to needlessly break compatibility, decrement MDIOVERSION to 0.

Approved by: phk

Move all the prototypes to one place.

Introduce a force option, MD_FORCE, that instructs the driver to
bypass some extra anti-foot-shooting measures. Currently, its only
effect is to allow detaching a device while it's still open (e.g.,
mounted). This is useful for testing how the system reacts to a disk
suddenly going away, which can happen with some removeable media.

At this point, the force option is only checked on detach, so it
would've been possible to allow the option to be passed with the
MDIOCDETACH operation. This was not done to allow the possibility of
having the force flag influence other tests in the future, which may
not necessarily deal with detaching the device.

Reviewed by: sobomax
Approved by: phk

Use MD_NAME and MDCTL_NAME constants where appropriate.

Constify mdmaybeload(), add a prototype for usage(), and get rid of a
sizeof(int) == sizeof(long) assumption; clamp down with WARNS=2.

Don't assume the length of MD_NAME is 2.

Fail if -s isn't specified for an MD_MALLOC or MD_SWAP disk; the
driver itself obviously won't configure such a disk, but the error
returned (EDOM) is more cryptic to the average user than it should be.

Also assert that the argument to -u is in fact a valid unit; don't
just accept any string to mean 0.

Approved by: phk

In the absence of explicit ``-t type'' option assume that ``-f file''
implies ``-t vnode''.

Approved by: phk

1) If mdconfig isn't given an action argument, it assumes detach.
2) Running `mdconfig -l` without any attached devices results in
mdconfig printing "md0" to standard output.

Submitted by: dd [1]

[1] no, not "dd(1)" but "Dima Dorfman"

Remove "autounit" from settable options, it's the default unless you
specify -u.

Spotted by: dcs

Forgot to remove unneeded "intcmp" function.

Make "md" and "mdctl" macroized parameters.

Implement "-l" option to mdconfig which can list one or all md devices.

Submitted by: Dima Dorfman <dima@unixfreak.org>

Make md/mdconfig do kld.

Submitted by: dcs

Strip optional prefixes "/dev/" and "md" from the -u argument.

Duh, the version of mdconfig.c I committed came from the wrong machine.

Commit the right version, but without the -b option which is too evil
for the present.

Remove -b and preload from the manpage as well.

Add padding space and a version number to the md_ioctl structre for future
extension.

Add ability to create a preload disk giving an address and a length
(suggested by imp)

Fix bug relating to very small md(4) devices.

Update md.c copyright to reflect the status of code copied from vn.c.
(noticed by dillon)

This is the first snapshot of the new all-singing-and-dancing md(4).

Using the mdconfig(8) program you can now configure memory disks
on malloc(9), swap or a file/vnode. preloaded md disks also work
as usual.

Improve cmdline parsing and messages.

Add a rudimentary man-page. (I'm no mdoc wizard, feel free to attack my
mistakes and blunders)

Preliminary scaffolding for the new integrated vn+md device driver.

I decided to work on the md(4) driver and integrate the vn(4)
functionality into it mainly based on the name being more suitable.
Ideally 'vd' as in "virtual disk" would probably be the most logical
but our sound-master pointed out that this would cause uncontrollable
fits of giggles in the brits. Another complication would the needed
changes to the ramdisk boot/root functionality.

The vn driver will stay around for some time after I complete this
merge for transition reasons, and I'll make it whine to people that
they should migrate to the md(4) driver for some time before it
dies.

The kernel part of the new md(4) driver will be committed after more
testing.