Copy stable/11@r303970 to releng/11.0 as part of the 11.0-RELEASE
cycle.

Prune svn:mergeinfo from the new branch, and rename it to RC1.

Update __FreeBSD_version.

Use the quarterly branch for the default FreeBSD.conf pkg(8) repo and
the dvd1.iso packages population.

Approved by: re (implicit)
Sponsored by: The FreeBSD Foundation

Copy head@r302406 to stable/11 as part of the 11.0-RELEASE cycle.
Prune svn:mergeinfo from the new branch, as nothing has been merged
here.

Additional commits post-branch will follow.

Approved by: re (implicit)
Sponsored by: The FreeBSD Foundation

DIRDEPS_BUILD: Update dependencies.

Sponsored by: EMC / Isilon Storage Division

Try to make gcc builds happy again by removing a redundant declaration.

GC duplicate define.

Improve performance in a few key areas:

o Split the compression across several worker threads. By default, "several"
matches number of CPUs, capped at 24 for sanity when running on a very big
hardwares. Provide option to set that number manually;

o Fix bug inherited from the mkulzma (R.I.P) which degraded already slow LZMA
compression even further by calling function to release compression state
after processing each block.

It is neither documented as required nor actually required by the LZMA
library. This caused spree of system calls to release memory and then map
it again for every block. LZMA compression is more than 2x faster after this
change alone;

o Record time it takes to do compression and report throughput achieved.

o Add simple first-level 256 entry hash table for de-dup code, so it's not
becoming a bottleneck at big files.

In the de-duplication mode, when found matching md5 checksum also read
back block and compare actual content. Just output original block
instead of back reference in the unlikely event of collision.

When -S is specified dump summary to stdout, not stderr, so it's
easier to capture and process it with external tools via pipe.

Add -S option to print out summary after compression has been
completed.

MFC after: 2 weeks

DIRDEPS_BUILD: Update dependencies.

Sponsored by: EMC / Isilon Storage Division

Improve mkuzip(8) and geom_uzip(4), merge in LZMA support from mkulzma(8)
and geom_uncompress(4):

1. mkuzip(8):

- Proper support for eliminating all-zero blocks when compressing an
image. This feature is already supported by the geom_uzip(4) module
and CLOOP format in general, so it's just a matter of making mkuzip(8)
match. It should be noted, however that this feature while it sounds
great, results in very slight improvement in the overall compression
ratio, since compressing default 16k all-zero block produces only 39
bytes compressed output block, which is 99.8% compression ratio. With
typical average compression ratio of amd64 binaries and data being
around 60-70% the difference between 99.8% and 100.0% is not that
great further diluted by the ratio of number of zero blocks in the
uncompressed image to the overall number of blocks being less than
0.5 (typically). However, this may be important from performance
standpoint, so that kernel are not spinning its wheels decompressing
those empty blocks every time this zero region is read. It could also
be important when you create huge image mostly filled with zero
blocks for testing purposes.

- New feature allowing to de-duplicate output image. It turns out that
if you twist CLOOP format a bit you can do that as well. And unlike
zero-blocks elimination, this gives a noticeable improvement in the
overall compression ratio, reducing output image by something like
3-4% on my test UFS2 3GB image consisting of full FreeBSD base system
plus some of the packages (openjdk, apache etc), about 2.3GB worth of
file data (800+MB compressed). The only caveat is that images created
with this feature "on" would not work on older versions of FeeBSDxi
kernel, hence it's turned off by default.

- provide options to control both features and document them in manual
page.

- merge in all relevant LZMA compression support from the mkulzma(8),
add new option to select between both.

- switch license from ad-hoc beerware into standard 2-clause BSD.

2. geom_uzip(4):

- implement support for de-duplicated images;

- optimize some code paths to handle "all-zero" blocks without reading
any compressed data;

- beef up manual page to explain that geom_uzip(4) is not limited only
to md(4) images. The compressed data can be written to the block
device and accessed directly via magic of GEOM(4) and devfs(4),
including to mount root fs from a compressed drive.

- convert debug log code from being compiled in conditionally into
being present all the time and provide two sysctls to turn it on or
off. Due to intended use of the module, it can be used in
environments where there may not be a luxury to put new kernel with
debug code enabled. Having those options handy allows debug issues
without as much problem by just having access to serial console or
network shell access to a box/appliance. The resulting additional
CPU cycles are just few int comparisons and branches, and those are
minuscule when compared to data decompression which is the main
feature of the module.

- hopefully improve robustness and resiliency of the geom_uzip(4) by
performing some of the data validation / range checking on the TOC
entries and rejecting to attach to an image if those checks fail.

- merge in all relevant LZMA decompression support from the
geom_uncompress(4), enable automatically when appropriate format is
indicated in the header.

- move compilation work into its own worker thread so that it does not
clog g_up. This allows multiple instances work in parallel utilizing
smp cores.

- document new knobs in the manual page.

Reviewed by: adrian
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D5333

Add META_MODE support.

Off by default, build behaves normally.
WITH_META_MODE we get auto objdir creation, the ability to
start build from anywhere in the tree.

Still need to add real targets under targets/ to build packages.

Differential Revision: D2796
Reviewed by: brooks imp

Convert to usr.bin/ to LIBADD
Reduce overlinking

use .Mt to mark up email addresses consistently (part3)

PR: 191174
Submitted by: Franco Fichtner <franco at lastsummer.de>

Fixed an embedded shell script.

Reviewed by: sobomax

mdoc: consistently spell our email addresses <foo@FreeBSD.org>

Reviewed by: ru

Build usr.bin/ with WARNS=6 by default.

Also add some missing $FreeBSD$ to keep svn happy.

Support character device as input file.

PR: 103500

Markup fixes.

A few minor corrections to the mkuzip.8 man page.

PR: 92576
Submitted by: Stefan Bethke

Tell the user exactly where the problem was.

- check for geom_uzip module presence using kldstat -m.
kldstat -m finds geom_uzip module even if it is compiled in statically.
- create output file with x bit set.
- build mkuzip on all architectures (verified with "make universe").
- fix typo in info message.

Make WARNS=6 clean, which should make it compiling on amd64.

Submitted by: Matteo Riondato <rionda@gufi.org>

Sort sections.

Added the EXIT STATUS section where appropriate.

Fix build: s/mkunzip.8/mkuzip.8/

Normalize the manpage.

Reviewed by: sobomax

Normalize the makefile.

Reviewed by: sobomax

o Print more info in the verbose mode;

o use zlib(3) function which computes maximum length of the output
buffer instead of rolling own version;

o allow size of input file to be not multiple of cluster size by applying
zero padding.

Clarify/extend in several places and make sure that everything matches reality.

Add mkuzip(8), non-GPL utility to compress filesystem images for use with
geom_uzip module. This is based on utility I wrote some 3 years ago for a
hack for md(4), which functionally was close to what geom_uzip does today.

Since I don't have a time to test that it compiles/works on other arches,
stick it to i386 only. Will do it later.

Unlike original cloop util, this one embedds FreeBSD-compatible shell code
into the generated image, not Linux one. Unfortunately severe space
restriction imposed by the CLOOP format doesn't allow to put conditional
code which will work both on Linux and FreeBSD. In fact it was quite a
challenge to fit necessary FreeBSD code into 127 bytes. ;-)