MFC r322951:

Respect MK_TCSH with build-tools and native-xtools

This helps reduce the WORLDTMP footprint slightly.

Based on a patch I submitted 5 years ago to GNATS.

PR: 174051
Relnotes: yes (anyone who cross-builds with MK_TCSH=yes will run into
build failures if the host doesn't have tcsh(1))
Reminded by: Fabian Keil <fk@fabiankeil.de>

MFC: r306375

Add a WITHOUT_DIALOG src.conf(5) knob.

It also turns off dependencies (bsdinstall, bsdconfig, dpv, tzsetup).

MFC r317658:

Redo r288270: Hookup mkcsmapper_static and mkesdb_static for 'make clean'

MFC r299529,r299540,r299576,r299896:

r299529,r299540:
Update libarchive to 3.2.0

New features:
- new bsdcat command-line utility
- LZ4 compression (in src only via external utility from ports)
- Warc format support
- 'Raw' format writer
- Zip: Support archives >4GB, entries >4GB
- Zip: Support encrypting and decrypting entries
- Zip: Support experimental streaming extension
- Identify encrypted entries in several formats
- New --clear-nochange-flags option to bsdtar tries to remove noschg and
similar flags before deleting files
- New --ignore-zeros option to bsdtar to handle concatenated tar archives
- Use multi-threaded LZMA decompression if liblzma supports it
- Expose version info for libraries used by libarchive

r299576,r299896:
Fix broken cpio behavior.

Relnotes: yes

MFC: r268745,r268746,r268747,r268748,r268749,r268861,r268750,r268751,r268763
r273769,r273771,r276771,r278810

New BSDL timeout(1) utility compatible with GNU timeout

Relnotes: yes

MFC r278193:

Add MK_FILE to control whether or not to build file(1), libmagic(3), etc

Sponsored by: EMC / Isilon Storage Division

MFC r278192:

Add the following options to enable/disable several features in the base system

WITHOUT_BOOTPARAMD - bootparamd
WITHOUT_BOOTPD - bootpd
WITHOUT_FINGER - finger, fingerd
WITHOUT_FTP - ftp, ftpd
WITHOUT_INETD - inetd
WITHOUT_RBOOTD - rbootd
WITHOUT_TCP_WRAPPERS - tcpd, et al
WITHOUT_TFTP - tftp, tftp-server
WITHOUT_TIMED - timed

Sponsored by: EMC / Isilon Storage Division

MFC r277676:

r277676:

Add MK_TALK knob for building the talk and talkd

Sponsored by: EMC / Isilon Storage Division

MFC r277675,r277726,r278070:

r277675:

Add MK_ISCSI knob for building the iscsi initiator, iscsi daemon, kernel
modules, etc

Sponsored by: EMC / Isilon Storage Division

r277726:

Build sbin/iscontrol again if MK_ISCSI != no

Pointyhat to: me

r278070:

Remove duplicate MK_ISCSI block and sort the conditional blocks so this error
won't crop up again in the future

Reported by: gjb

MFC r277663:

r277663:

Add MK_EE knob to control installing edit, ee, etc

Sponsored by: EMC / Isilon Storage Division

MFC r274116:

Add new libraries/utilities for data throughput visualization.
dpv(3): dialog progress view library
dpv(1): stream data from stdin or multiple paths with dialog progress view
figpar(3): configuration file parsing library

MFC r274120, r274121, r274123, r274124, r274144, r274146, r274159, r274192,
r274203, r274209, r274226, r274270, and r274851: Fixes following r274116

Reviews: D714
Relnotes: New libdpv/libfigpar and dpv(1) utility
Reviewed by: jelischer, shurd
Discussed at: MeetBSD California 2014 Vendor/Dev Summit
Discussed on: -current
Thanks to: ngie, ian, jelischer, shurd, bapt

MFC r267176, r267181, r268445 (ATF-related commits):

Phabric: https://reviews.freebsd.org/D706
Approved by: rpaulo (mentor)
Approved by: re (gjb)
Reviewed by: jmmv
Sponsored by: EMC / Isilon Storage Division

r267176:

Add the *_TESTS_SH_SED_* functionality to atf.test.mk.

This exists already in plain.test.mk and tap.test.mk and should have been
added to atf.test.mk too when the feature was first introduced.

(It is probably time to address the related TODOs but I will do that
separately.)

r267181:

Move atf-sh from /usr/bin/ to /usr/libexec/

In r266650, we made libatf-c and libatf-c++ private libraries so that no
components outside of the source tree could unintendedly depend on them.

This change does the same for the "atf-sh library" by moving the atf-sh
interpreter from its public location in /usr/bin/ to the private location
in /usr/libexec/. Our build system will ensure that our own test programs
use the right binary, but users won't be able to depend on atf-sh by
"mistake".

Committing this now to ride the UPDATING notice added with r267172 today.

r268445:

Fix atf-sh's integration_test

With the move of atf-sh into /usr/libexec in r267181, some of the
tests in the integration_test program broke because they could not
execute atf-sh from the path any longer.

This slipped through because I do have a local atf installation in
my home directory that appears in my path, hence the tests could
still execute my own version.

Fix this by forcing /usr/libexec to appear at the beginning of the
path when attempting to execute atf-sh.

To make upgrading easy (and to avoid an unnecessary entry in UPDATING),
make integration_test depend on the Makefile so that a rebuild of the
shell script is triggered. This requires a hack in the *.test.mk files
to ensure the Makefile is not treated as a source to the generated
program. Ugly, I know, but I don't have a better way of doing this at
the moment. Will think of one once I address the TODO in the *.test.mk
files that suggests generalizing the file generation functionality.

PR: 191052
Reviewed by: Garrett Cooper

MFC vtfontcvt improvements:

r267011: Make height and width optional arguments

Now defaults to a 16x8 font size. The height and width can be specified
using -h and -w respectively.

r267012: Make the bold font optional

r267035: Use a hash to speed up glyph deduplication

Walking a linked list of all glyphs to look for a duplicate is very slow
for large fonts (e.g., for CJK character sets). In my test the runtime
for a sample 40000 character font went from just over 80 seconds on
average to just over 2 seconds.

r267119: -w sets the width, not height

r267123: Support "GNU Unifont" format font data

The GNU Unifont .hex format is a text file. Each line represents one
glyph and consists of a four-digit hex code point, a colon, and pairs of
hex digits representing the bitmap. By default an 8x16 font is assumed,
with 16x16 double-width glyphs, resulting in either 32 or 64 hex digits
for the bitmap.

Our version of the file format supports comments at the top of the file
to set the height and width:

Each row of bitmap data is rounded up to byte width - for example, a
10-pixel wide font uses 4 characters per row.

See http://czyborra.com/unifont/ for more background on the original
format.

r267126: Accept space after BITMAP in .bdf parser

The Unifont BDF generator incorrectly adds a space after BITMAP, and
and that error has been widely propagated.

r267173: use -h height and -w width args

r267298: Hide stats by default and improve error handling

The font stats are interesting, but rather verbose.

r267301: Speed up bold glyph map deduplication

Perform an O(n) deduplication pass over the bold maps at the end, rather
than walking the normal map list to look for a duplicate glyph each time
a bold mapping entry is added.

r267324: handle failure writing output font

r267337: move to usr.bin/vtfontcvt

vtfontcvt is useful for end users to convert arbitrary bitmap fonts
for use by vt(4). It can also be used as a build tool, allowing us
to keep the source font data in the src tree rather than uuencoded
binaries.

Reviewed by: ray, wblock (D183)

r267366: Avoid leaking file pointer on error

CID: 1222506, 1222505

r268022: Rename the WITHOUT_VT_SUPPORT knob to WITHOUT_VT

The _SUPPORT knobs have a consistent meaning which differs from the
behaviour controlled by this knob. As the knob is opt-out and has not
appeared in a release the impact should be low.

r268172: correct width calculation (.hex files and commandline)

r268948: Use the standard way of printing the usage string

r268949: Remove redundant return statement after errx

Also update vtfontcvt(8), based on inclusion in FreeBSD 10.1

Sponsored by: The FreeBSD Foundation

MFC mkimg(1) -- revisions 268159, 268134, 266556, 266514, 266513,
266512, 266511, 266510, 266509, 266176, 265468, 265467,
265462, 265170, 263926, 263924, 263923, 263919 and 263918.

Revision 267182 changed mkimg.1 alongside other unrelated manpages.
The change to mkimg.1 has been applied without registering a merge
of the revision. This allows a future merge of r267182 to happen.

Relnotes: yes

Merge r267482,r267483,r267486,r267577,r267671,r267672 from head:

Remove send-pr and fix up all references to it. Replace it with a
stub send-pr directing people towards the web site.

MFC r261215, r261257

Merge from vendor branch importing dtc git
rev 6a15eb2350426d285130e4c9d84c0bdb6575547a

Don't build BSDL dtc if the GPL dtc is enabled.

MFC r263778:

Add a SUBDIR_PARALLEL option to bsd.subdir.mk, to allow make to process
all the SUBDIR entries in parallel, instead of serially. Apply this
option to a selected number of Makefiles, which can greatly speed up the
build on multi-core machines, when using make -j.

This can be extended to more Makefiles later on, whenever they are
verified to work correctly with parallel building.

I tested this on a 24-core machine, with make -j48 buildworld (N = 6):

before stddev after stddev
======= ====== ======= ======
real time 1741.1 16.5 959.8 2.7
user time 12468.7 16.4 14393.0 16.8
sys time 1825.0 54.8 2110.6 22.8

(user+sys)/real 8.2 17.1

E.g. the build was approximately 45% faster in real time. On machines
with less cores, or with lower -j settings, the speedup will not be as
impressive. But at least you can now almost max out a machine with
buildworld!

Submitted by: jilles

MFC r263833:

Enable parallel building for gnu/usr.bin and usr.bin/clang too.

Plug the ATF tests into the build.

This is a MFC into stable/10 of:
- r257849 Add libatf-c++ to the prebuild libs.
- r257853 Build and install the atf tests.
- r258233 Move all atf directories to the tests mtree.
- r258285 Fix the build of some ATF tests.

This change is "make tinderbox" clean on ref10-amd64 with the default
settings of WITHOUT_TESTS. It is likely for the WITH_TESTS build to
still be broken because not all relevant changes have been merged yet.

Set up the /usr/tests hierarchy.

This is a MFC of the following into stable/10:
- r257097 Set up the /usr/tests hierarchy.
- r257098 Add missing WITHOUTTESTS file.
- r257100 Add a tests(7) manual page.
- r257105 Disable WITHTESTS= for now.
- r257848 Fix buildworld when WITHTESTS is enabled.
- r257850 Subsume the functionality of MKATF into MKTESTS.
- r257851 Handle the removal of the test suite when WITHOUTTESTS=yes.
- r257852 Install category Kyuafiles from their category directories.
- r258232 Install BSD.tests.mtree when MKTESTS is yes.

Note that building with WITH_TESTS is still broken at this point (and
hence why WITHOUT_TESTS is the set as the default). Subsequent pullups
will fix the remaining issues.

MFC: r268745,r268746,r268747,r268748,r268749,r268861,r268750,r268751,r268763
r273769,r273771,r276771,r278810

New BSDL timeout(1) utility compatible with GNU timeout

Relnotes: yes

MFC r278193:

Add MK_FILE to control whether or not to build file(1), libmagic(3), etc

Sponsored by: EMC / Isilon Storage Division

MFC r278192:

Add the following options to enable/disable several features in the base system

WITHOUT_BOOTPARAMD - bootparamd
WITHOUT_BOOTPD - bootpd
WITHOUT_FINGER - finger, fingerd
WITHOUT_FTP - ftp, ftpd
WITHOUT_INETD - inetd
WITHOUT_RBOOTD - rbootd
WITHOUT_TCP_WRAPPERS - tcpd, et al
WITHOUT_TFTP - tftp, tftp-server
WITHOUT_TIMED - timed

Sponsored by: EMC / Isilon Storage Division

MFC r277676:

r277676:

Add MK_TALK knob for building the talk and talkd

Sponsored by: EMC / Isilon Storage Division

MFC r277675,r277726,r278070:

r277675:

Add MK_ISCSI knob for building the iscsi initiator, iscsi daemon, kernel
modules, etc

Sponsored by: EMC / Isilon Storage Division

r277726:

Build sbin/iscontrol again if MK_ISCSI != no

Pointyhat to: me

r278070:

Remove duplicate MK_ISCSI block and sort the conditional blocks so this error
won't crop up again in the future

Reported by: gjb

MFC r277663:

r277663:

Add MK_EE knob to control installing edit, ee, etc

Sponsored by: EMC / Isilon Storage Division

MFC r274116:

Add new libraries/utilities for data throughput visualization.
dpv(3): dialog progress view library
dpv(1): stream data from stdin or multiple paths with dialog progress view
figpar(3): configuration file parsing library

MFC r274120, r274121, r274123, r274124, r274144, r274146, r274159, r274192,
r274203, r274209, r274226, r274270, and r274851: Fixes following r274116

Reviews: D714
Relnotes: New libdpv/libfigpar and dpv(1) utility
Reviewed by: jelischer, shurd
Discussed at: MeetBSD California 2014 Vendor/Dev Summit
Discussed on: -current
Thanks to: ngie, ian, jelischer, shurd, bapt

MFC r267176, r267181, r268445 (ATF-related commits):

Phabric: https://reviews.freebsd.org/D706
Approved by: rpaulo (mentor)
Approved by: re (gjb)
Reviewed by: jmmv
Sponsored by: EMC / Isilon Storage Division

r267176:

Add the *_TESTS_SH_SED_* functionality to atf.test.mk.

This exists already in plain.test.mk and tap.test.mk and should have been
added to atf.test.mk too when the feature was first introduced.

(It is probably time to address the related TODOs but I will do that
separately.)

r267181:

Move atf-sh from /usr/bin/ to /usr/libexec/

In r266650, we made libatf-c and libatf-c++ private libraries so that no
components outside of the source tree could unintendedly depend on them.

This change does the same for the "atf-sh library" by moving the atf-sh
interpreter from its public location in /usr/bin/ to the private location
in /usr/libexec/. Our build system will ensure that our own test programs
use the right binary, but users won't be able to depend on atf-sh by
"mistake".

Committing this now to ride the UPDATING notice added with r267172 today.

r268445:

Fix atf-sh's integration_test

With the move of atf-sh into /usr/libexec in r267181, some of the
tests in the integration_test program broke because they could not
execute atf-sh from the path any longer.

This slipped through because I do have a local atf installation in
my home directory that appears in my path, hence the tests could
still execute my own version.

Fix this by forcing /usr/libexec to appear at the beginning of the
path when attempting to execute atf-sh.

To make upgrading easy (and to avoid an unnecessary entry in UPDATING),
make integration_test depend on the Makefile so that a rebuild of the
shell script is triggered. This requires a hack in the *.test.mk files
to ensure the Makefile is not treated as a source to the generated
program. Ugly, I know, but I don't have a better way of doing this at
the moment. Will think of one once I address the TODO in the *.test.mk
files that suggests generalizing the file generation functionality.

PR: 191052
Reviewed by: Garrett Cooper

MFC vtfontcvt improvements:

r267011: Make height and width optional arguments

Now defaults to a 16x8 font size. The height and width can be specified
using -h and -w respectively.

r267012: Make the bold font optional

r267035: Use a hash to speed up glyph deduplication

Walking a linked list of all glyphs to look for a duplicate is very slow
for large fonts (e.g., for CJK character sets). In my test the runtime
for a sample 40000 character font went from just over 80 seconds on
average to just over 2 seconds.

r267119: -w sets the width, not height

r267123: Support "GNU Unifont" format font data

The GNU Unifont .hex format is a text file. Each line represents one
glyph and consists of a four-digit hex code point, a colon, and pairs of
hex digits representing the bitmap. By default an 8x16 font is assumed,
with 16x16 double-width glyphs, resulting in either 32 or 64 hex digits
for the bitmap.

Our version of the file format supports comments at the top of the file
to set the height and width:

Each row of bitmap data is rounded up to byte width - for example, a
10-pixel wide font uses 4 characters per row.

See http://czyborra.com/unifont/ for more background on the original
format.

r267126: Accept space after BITMAP in .bdf parser

The Unifont BDF generator incorrectly adds a space after BITMAP, and
and that error has been widely propagated.

r267173: use -h height and -w width args

r267298: Hide stats by default and improve error handling

The font stats are interesting, but rather verbose.

r267301: Speed up bold glyph map deduplication

Perform an O(n) deduplication pass over the bold maps at the end, rather
than walking the normal map list to look for a duplicate glyph each time
a bold mapping entry is added.

r267324: handle failure writing output font

r267337: move to usr.bin/vtfontcvt

vtfontcvt is useful for end users to convert arbitrary bitmap fonts
for use by vt(4). It can also be used as a build tool, allowing us
to keep the source font data in the src tree rather than uuencoded
binaries.

Reviewed by: ray, wblock (D183)

r267366: Avoid leaking file pointer on error

CID: 1222506, 1222505

r268022: Rename the WITHOUT_VT_SUPPORT knob to WITHOUT_VT

The _SUPPORT knobs have a consistent meaning which differs from the
behaviour controlled by this knob. As the knob is opt-out and has not
appeared in a release the impact should be low.

r268172: correct width calculation (.hex files and commandline)

r268948: Use the standard way of printing the usage string

r268949: Remove redundant return statement after errx

Also update vtfontcvt(8), based on inclusion in FreeBSD 10.1

Sponsored by: The FreeBSD Foundation

MFC mkimg(1) -- revisions 268159, 268134, 266556, 266514, 266513,
266512, 266511, 266510, 266509, 266176, 265468, 265467,
265462, 265170, 263926, 263924, 263923, 263919 and 263918.

Revision 267182 changed mkimg.1 alongside other unrelated manpages.
The change to mkimg.1 has been applied without registering a merge
of the revision. This allows a future merge of r267182 to happen.

Relnotes: yes

Merge r267482,r267483,r267486,r267577,r267671,r267672 from head:

Remove send-pr and fix up all references to it. Replace it with a
stub send-pr directing people towards the web site.

MFC r261215, r261257

Merge from vendor branch importing dtc git
rev 6a15eb2350426d285130e4c9d84c0bdb6575547a

Don't build BSDL dtc if the GPL dtc is enabled.

MFC r263778:

Add a SUBDIR_PARALLEL option to bsd.subdir.mk, to allow make to process
all the SUBDIR entries in parallel, instead of serially. Apply this
option to a selected number of Makefiles, which can greatly speed up the
build on multi-core machines, when using make -j.

This can be extended to more Makefiles later on, whenever they are
verified to work correctly with parallel building.

I tested this on a 24-core machine, with make -j48 buildworld (N = 6):

before stddev after stddev
======= ====== ======= ======
real time 1741.1 16.5 959.8 2.7
user time 12468.7 16.4 14393.0 16.8
sys time 1825.0 54.8 2110.6 22.8

(user+sys)/real 8.2 17.1

E.g. the build was approximately 45% faster in real time. On machines
with less cores, or with lower -j settings, the speedup will not be as
impressive. But at least you can now almost max out a machine with
buildworld!

Submitted by: jilles

MFC r263833:

Enable parallel building for gnu/usr.bin and usr.bin/clang too.

Plug the ATF tests into the build.

This is a MFC into stable/10 of:
- r257849 Add libatf-c++ to the prebuild libs.
- r257853 Build and install the atf tests.
- r258233 Move all atf directories to the tests mtree.
- r258285 Fix the build of some ATF tests.

This change is "make tinderbox" clean on ref10-amd64 with the default
settings of WITHOUT_TESTS. It is likely for the WITH_TESTS build to
still be broken because not all relevant changes have been merged yet.

Set up the /usr/tests hierarchy.

This is a MFC of the following into stable/10:
- r257097 Set up the /usr/tests hierarchy.
- r257098 Add missing WITHOUTTESTS file.
- r257100 Add a tests(7) manual page.
- r257105 Disable WITHTESTS= for now.
- r257848 Fix buildworld when WITHTESTS is enabled.
- r257850 Subsume the functionality of MKATF into MKTESTS.
- r257851 Handle the removal of the test suite when WITHOUTTESTS=yes.
- r257852 Install category Kyuafiles from their category directories.
- r258232 Install BSD.tests.mtree when MKTESTS is yes.

Note that building with WITH_TESTS is still broken at this point (and
hence why WITHOUT_TESTS is the set as the default). Subsequent pullups
will fix the remaining issues.