Fix bspatch heap overflow vulnerability. [SA-16:29]

Fix multiple portsnap vulnerabilities. [SA-16:30]

Fix multiple libarchive vulnerabilities. [SA-16:31]

Approved by: so

Copy stable/10@r272459 to releng/10.1 as part of
the 10.1-RELEASE process.

Approved by: re (implicit)
Sponsored by: The FreeBSD Foundation

Copy head (r256279) to stable/10 as part of the 10.0-RELEASE cycle.

Approved by: re (implicit)
Sponsored by: The FreeBSD Foundation

MFV r248590,248594:
Update libarchive to 3.1.2

Some of new features:
- support for lrzip and grzip compression
- support for writing tar v7 format
- b64encode and uuencode filters
- support for __MACOSX directory in Zip archives
- support for lzop compresion (external utility)

Backport NFSv4 ACL fix from libarchive master branch.

Source:
https://github.com/libarchive/libarchive/commit/f67370d5

Obtained from: libarchive (master branch)

Update libarchive to 3.0.4

Update libarchive to 3.0.3

Some of new features:
- New readers: RAR, LHA/LZH, CAB reader, 7-Zip
- New writers: ISO9660, XAR
- Improvements to many formats, especially including ISO9660 and Zip
- Stackable write filters to write, e.g., tar.gz.uu in a single pass
- Exploit seekable input; new "seekable" Zip reader can exploit the Zip
Central Directory when it's available; the old "streamable" Zip reader
is still fully supported for cases where seeking is not possible.

Full release notes available at:
https://github.com/libarchive/libarchive/wiki/ReleaseNotes

Use contrib sources for building libarchive, tar and cpio.
Make "make test" fully operational.

MFC after: 2 weeks

Update libarchive, tar and cpio to version 2.8.5

The following additional vendor revisions are applied:

Revision 3740:
Use archive_clear_error() to clear the error markers.

Obtained from: http://code.google.com/p/libarchive
MFC after: 2 weeks

Import bugfix for reading and extracting of FreeBSD ISO images with tar.

Upstream revision 3645 (merge of 3642):
Change the mechanism handling a rr_moved directory,
which is Rockridge extension that can exceed the limitation of
a maximum directory depth of ISO 9660.
- Stop reading all entries at a time.
- Connect "CL" entry to "RE" entry dynamically, which "CL" and "RE"
have information to rebuild a full directory tree.
- Tweak some related tests since we use Headsort for re-ordering
entries and it cannot make a steady order when the keies of
the entries are the same.

http://code.google.com/p/libarchive/issues/detail?id=168

Reviewed by: kientzle
Approved by: re (kib)
Obtained from: libarchive (release/2.8, svn rev 3645)
MFC after: 3 days

Add compatibility for ISO images created with unfixed makefs that
violated ECMA-119 (ISO9660): allow reserved4 to be 0x20 in PVD.
This allows tar to read FreeBSD distribution ISO images created
with makefs prior to NetBSD bin/45217 bugfix (up to 9.0-BETA1).

In addition, merge following important bugfixes from
libarchive's release/2.8 branch:

Revision 2812:
Merge 2811 from trunk: Don't try to verify that compression-level=0
produces larger results than the default compression, since this isn't
true for all versions of liblzma.

Revision 2817:
Merge 2814 from trunk: Fix Issue 121 (mtree parser error)
http://code.google.com/p/libarchive/issues/detail?id=121

Revision 2820:
Fix issue 119.
Change the file location check that a file location does not exceed
volume block. New one is that a file content does not exceed volume
block(end of an ISO image). It is better than previous check even
if the issue did not happen.

While reading an ISO image generated by an older version of mkisofs
utility, a file location indicates the end the ISO image if its file
size is zero and it is the last file of all files of the ISO image,
so it is possible that the location value is the same as the number
of the total block of the ISO image.

http://code.google.com/p/libarchive/issues/detail?id=119

Revision 2955:
Issue 134: Fix libarchive 2.8 crashing in archive_write_finish() when
the open has failed and we're trying to write Zip format.

http://code.google.com/p/libarchive/issues/detail?id=134

Revision 2958:
Followup on Issue 134:
1) Port test_open_failure to libarchive 2.8 branch to test
the problem reported in Issue 134.
This test also shows that archive_read_open() sometimes
fails to report open errors correctly.
2) Fix the bug in archive_read.c
3) Comment out the tests that close functions are invoked
promptly when open fails; that's fully fixed in libarchive 3.0,
but I don't think it's worth fixing here.

Revision 3484:
Use uintmax_t with %ju

Revision 3487:
Fix issue 163.
Correctly allocate enough memory for a input buffer saved.

http://code.google.com/p/libarchive/issues/detail?id=163

Revision 3542:
Merge 2516, 2536 from trunk: Allow path table offset values of
0 and 18, which are used by some ISO writers.

Reviewed by: kientzle
Approved by: re (kib)
MFC after: 3 days

- Update libarchive to 2.8.4
- Add support for extracting xar and rpm archives
- Add libarchive_fe subdir (common code for tar and cpio)

Approved by: kientzle
MFC after: 2 weeks

Don't write data into an empty "file."

In particular, this check avoids a warning when
extracting directory entries from certain GNU tar
archives that store directory contents.

MFC after: 3 days

If the Zip reader doesn't see a PK signature block
because there's inter-entry garbage, just scan forward
to find the next one. This allows us to handle a lot
of Zip archives that have been modified in-place.

Thanks to: Gleb Kurtsou for sending me a sample archive

Remove redundant WARNS?=6 overrides and inherit the WARNS setting from
the toplevel directory.

This does not change any WARNS level and survives a make universe.

Approved by: ed (co-mentor)

Remove some test files that are no longer used.

UU decoder. Now that libarchive can recursively taste input streams,
you can do things like this: tar xvf archive.tar.gz.uu

A raft of test changes and improvements from the Googlecode repository.
In particular, this includes tests for the new features I've merged
over the last few days.

Fiz /usr/bin/unzip: A bug deep in libarchive's read-ahead logic
(incorrect handling of zero-length reads before the copy buffer is
allocated) is masked by the iso9660 taster. Tar and cpio both enable
that taster so were protected from the bug; unzip is susceptible.

This both fixes the bug and updates the test harness to exercise
this case.

Submitted by: Ed Schouten diagnosed the bug and drafted a patch
MFC after: 7 days

Update tests to match r195873, which corrected how hardlinked files
on iso9660 images were returned. While I'm poking around, update
some comments around this area to try to clarify what's going on and
what still remains to be improved.

Merge r990,r1044 from libarchive.googlecode.com:
read_support_format_raw() allows people to exploit libarchive's
automatic decompression support by simply stubbing out the
archive format handler.
The raw handler is not enabled by support_format_all(), of course.
It bids 1 on any non-empty input and always returns a single
entry named "data" with no properties set.

Merge r1061,r1062,r1063 from libarchive.googlecode.com:
Fix reading big-endian binary cpio archives, and add a test.
While I'm here, add a note about Solaris ACL extension for cpio,
which should be relatively straightforward to support.

Thanks to: Edward Napierala, who sent me a big-endian cpio archive
from a Solaris system he's been playing with.
Pointy hat: me

Merge r1034 from libarchive.googlecode.com:
Put a much larger file on the reference ISO for this test.

Merge r1058 from libarchive.googlecode.com: Require that each
test source file has exactly one DEFINE_TEST().

Merge r1032 from libarchive.googlecode.com:
Make test_fuzz a bit more sensitive by actually reading the body
of each entry instead of skipping it.
While I'm here, move the "UnsupportedCompress" macro into the
only file that still uses it.

Merge r1054,r1060 from libarchive.googlecode.com:
* assertEqualMem() now takes void * arguments
* Be a little smarter about what we hexdump when assertEqualMem() fails

Merge r1053,r1055,r1056,r1057,r1065 from libarchive.googlecode.com:
* Fix parsing of POSIX.1e ACLs from Solaris tar archives
* Test the above
* Preserve the order of POSIX.1e ACL entries
* Update tests whose results depended on the order of ACL entries
* Identify NFSv4 ACLs in Solaris tar archives and warn that
they're not yet supported. (In particular, don't try to parse
them as POSIX.1e ACLs.)

Thanks to: Edward Napierala sent me some Solaris 10 tar archives to test

Reading an mtree file is supposed to provide
access to the file data (if the file exists on
disk). This was broken for the first regular
file; fix it and add a test so it won't break again.

In particular, this fixes the following idiom for creating
a tar archive in which every file is owned by root:

tar cf - --format=mtree . \
| sed -e 's/uname=[a-z]*/uname=root/' -e 's/uid=[0-9]*/uid=0/' \
| tar cf - @-

Merge lots of test suite updates from libarchive.googlecode.com.

Merge from libarchive.googlecode.com r791, r879, r884, r948: Various
fixes to read_support_compression_program. In particular, failure of
the external program is detected a lot earlier, which gives much more
reasonable error handling.

Merge from libarchive.googlecode.com: Mostly a bunch of
corrections to the Windows support to reconcile differences
between Visual Studio and Cygwin. Includes parts of
revisions 757, 774, 787, 815, 817, 819, 820, 844, and 886.

Of particular note, r886 overhauled the UTF-8/Unicode conversions to
work correctly regardless of whether the local system uses 16-bit
or 32-bit wchar_t. (I assume that systems with 16-bit wchar_t
use UTF-16 and those with 32-bit wchar_t use UCS-4.) This revision
also added a preference for wcrtomb() (which is thread-safe) on
platforms that support it.

Merge r718 from libarchive.googlecode.com: Some additional
tests of restoring files to disk with unusual characters, specifically
to exercise Windows issues.

Fix spelling.

Merge r348 from libarchive.googlecode.com: Suppress testing invalid
conversions if there aren't any. In particular, Cygwin's "C" locale
has no invalid inputs for wctomb().

Merge r596,r690 from libarchive.googlecode.com: Minor style and compile
warning fixes for test_read_pax_truncated.c.

Merge r335,653,676 from libarchive.googlecode.com: Instead of
conditioning tests on HAVE_ZLIB, etc, just ask libarchive for the
service and handle the failure coming back from libarchive. This
gives us better test coverage of common client usage where clients
simply try to use libarchive services and handle the errors coming
back instead of trying to second-guess which libarchive services are
compiled in.

Merge r280,281,496,595,675,712 from libarchive.googlecode.com: Various
test improvements, including some work on Windows compatibility and an
extra check to verify that no test leaves open file descriptors
around.

Merge r558,567,569,571,581,582,583,598 from libarchive.googlecode.com:
Support Joliet extensions. This currently ignores Rockridge extensions
if both exist on the same disk unless the '!joliet' option is provided.
e.g.: tar -xvf example.iso --options '!joliet'
Thanks to: Andreas Henriksson

Merge r658 from libarchive.googlecode.com: Only flush and close the
file if it was actually opened. Test for this case.

Merge r714,r715 from libarchive.googlecode.com: Fix Debian bug #516577.
Don't crash if client does not provide a skip function. Extend one
of the test cases to use archive_read_open2() with only a read callback.

Merge the rest of r453 from libarchive.googlecode.com: Test using
libarchive APIs to read extended attributes from disk on FreeBSD.

Merge r659 from libarchive.googlecode.com: Correctly report "none"
as the compression name when no other read filter bid. Add some
assertions to various tests to verify that read filters are properly
setting the textual name as well as the compression code.

Merge r448 from libarchive.googlecode.com: Suppress testing write_disk
failures on Windows for now. Someday this will be revisited.

Merge r420,r494 from libarchive.googlecode.com: Prettify the test
harness a bit: remove a dead comment, tweak the wording of the
summary report.

Merge r435,r443 from libarchive.googlecode.com: Read config files from
include path; this makes it easier to support multiple build frameworks.

Merge r416 from libarchive.googlecode.com:

Restoring POSIX.1e Extended Attributes on FreeBSD, part 1

This implements the basic ability to restore extended attributes
on FreeBSD, including a test suite.

Merge r399,401,402,405,415,430,440,452,453,458,506,533,536,538,544,590
from libarchive.googlecode.com: Add a new "archive_read_disk" API
that provides the important service of reading metadata from the
disk. In particular, this will make it possible to remove all
knowledge of extended attributes, ACLs, etc, from clients such
as bsdtar and bsdcpio.

Closely related, this API also provides pluggable uid->uname
and gid->gname lookup and caching services similar to
the uname->uid and gname->gid services provided by archive_write_disk.
Remember this is also required for correct ACL management.

Documentation is still pending...

Merge r394,r396 from libarchive.googlecode.com: Plug some memory
leaks in the ACL test, correctly mark that FreeBSD has acl_get_perm_np().

Merge r386,r395,r451 from libarchive.googlecode.com: On Windows, break
into the debugger on test setup failures (otherwise, the console window
just goes away and you can't see what went wrong). On all platforms,
clean up a stray buffer before exiting.

Merge r551,r561 from libarchive.googlecode.com: Update gzip read filter
to fully take advantage of the new peek/consume I/O support.
In particular, this now properly handles concatenated gzip streams.

Merge r364, r378, r379, r393, and r539 from libarchive.googlecode.com:
This is the last phase of the "big decompression refactor" that
puts a lazy reblocking layer between each pair of read filters.
I've also changed the terminology for this area---the two kinds
of objects are now called "read filters" and "read filter bidders"---and
moved ownership of these objects to the archive_read core.

This greatly simplifies implementing new read filters, which
can now use peek/consume I/O semantics both for bidding (arbitrary
look-ahead!) and for reading streams (look-ahead simplifies handling
concatenated streams, for instance).

The first merge here is the overhaul proper; the remainder are small
fixes to correct errors in the initial implementation.

Merge r357 from libarchive.googlecode.com: bzip2 compression
support can always be enabled even if bzlib doesn't exist on
this platform; don't give up until we fail to open the file.

Merge r362 from libarchive.googlecode.com: Minor fix to
custom argument parser.

Merge r341,r345,r346,347 from libarchive.googlecode.com: Style
fixes to test harness and a few extra guards to detect tests
that can't succeed on certain platforms.

Merge r294:337,r348:350 from libarchive.googlecode.com: A lot
of work to make libarchive work on Windows.

Merge r294 from libarchive.googlecode.com: Skip testing for
locale-based failures on systems where the "C" locale is so permissive
that it cannot possibly fail. In particular, this fixes a test
problem on Cygwin.

Fix ARCHIVE_EXTRACT_SPARSE handling in libarchive.
Add a test to exercise this feature.
This should fix --sparse/-S support in tar.

Thanks to: Daichi GOTO
MFC after: 1 week

Merge-from-Googlecode r419

In archive_write_disk: If archive_write_header() fails to create
the file, that's a failure and should return ARCHIVE_FAILED.
Metadata restore failures still return ARCHIVE_WARN, because
that's non-critical. Fix test_write_disk_secure test to
verify the correct return code in one case; add test_write_disk_failures
to do another very simple test of restore failure.

This should fix cpio coredumping when it tries to restore to
a write-protected directory.

Thanks to: Giorgos Keramidas
MFC after: 30 days

Teach get_refdir() about FreeBSD's /usr/obj convention.
In development, I run libarchive_test frequently by hand
and it gets tedious having to specify a suitable -r path
all of the time.

Fill in a missing const

Exit earlier on failure in this test.

Once the test has failed, exit before the segfault.

Update the ar write test to give more detailed information about failures.

Obey the TMPDIR, TMP, TEMP, or TEMPDIR environment variables
when choosing a scratch directory for the tests. Fallback
to "/tmp", of course.

New tests:
* support for bzip2 file with multiple concatenated bzip2 streams
* support for bzip2 file with junk after bzip2 stream
* support for gzip file with junk after gzip stream
* "fuzz" tester randomly modifies a bunch of input files in order to try
to crash libarchive (this found an amusing hang in the ISO9660 code
when trying to read images that advertised a zero blocksize).

This test is implemented, but commented out for now:
* support for gzip file with multiple concatenated gzip streams

Recover after test failure.

Style: skipping() function correctly counts this as a skipped test.

Style improvements.

Better reporting of test failures.

Recover from a test failure here.

When running on a filesystem that lacks ACL support,
just SKIP the test, don't report a test failure.

If we don't know the file size, leave the file size unset.
If it's not a regular file, don't return any data, even if the size is unknown.

Update the Zip test with a hand-tweaked Zip archive that has a
directory (with length-at-end set), a regular file without
length-at-end set, and a regular file with length-at-end set and a bad
CRC. Update the test code to verify that the file size is unset
for the regular file with length-at-end.

MFC after: 7 days

Add test to verify ACL restores on FreeBSD, verify kern/128203 (mtime
restore fails if ACL provided), apply patch from kern/128203, and verify fix.

PR: kern/128203
Submitted by: Udo Schweigert
MFC after: 3 days

Style: Mark private data 'static', correct an old comment.

MFC after: 3 days

Verify high-res birthtime support on FreeBSD.

MFC after: 30 days

MfP4: restore birth time data to disk + more thorough tests for
time restore to disk.

MFC after: 30 days

MfP4: Store/read birthtime data in pax format.

Submitted by: Pedro Giffuni
MFC after: 30 days

MfP4: Support for storing birthtime in archive_entry objects.

Submitted by: Pedro Giffuni
MFC after: 30 days

Trying to distinguish different "versions" of mtree files was a silly idea.

Submitted by: Joerg Sonnenberger

Fix a type-aliasing problem in the libarchive test suite,
restore WARNS=6.

Correct a leaking archive_entry in the test suite.

Correct the comments here.

PR: bin/126849
MFC after: 3 days

When restoring a directory, allow symlinks to be followed. The full
logic here gets a little complex, but the net effect is that the
SECURE_SYMLINKS flag will prevent us from ever following a symlink.
Without it, we'll only follow symlinks to dirs. bsdtar specifies
SECURE_SYMLINKS by default, suppresses it for -P.

I've also beefed up the write_disk_secure test to verify this
behavior.

PR: bin/126849

MfP4: Handle entries with unset size properly: Regular files with
unspecified size are "unlimited" (required by Zip reader, which
sometimes does not know the uncompressed size of an entry until it
gets to the end). Also, hardlinks with unspecified (or zero) size do
not overwrite the data on disk nor do they set metadata. This is
compatible with GNU tar and NetBSD pax behavior.

Style: Use ARCHIVE_VERSION_NUMBER to conditionalize tests on
the particular libarchive version being tested instead of the
deprecated ARCHIVE_API_VERSION and ARCHIVE_VERSION_STAMP macros.

MfP4: set/unset tracking for atime, ctime, mtime, and size fields.
This generalizes the existing set/unset tracking for hardlink/symlink
fields and extends it to cover non-string fields. Eventually, this
will be further extended to cover most fields.

In particular, this is needed to correctly detect when time fields
are missing (for example, reading ustar archives doesn't set atime or
ctime) for proper time restore and is helpful when trying to determine
whether to overwrite data when restoring hardlinks.

This commit updates the tests but not the docs.

Use the "start_time" (time at which the restore process began)
to fill in a missing atime instead of substituting mtime.

PR: bin/124915

Make libarchive_test a little smarter about trying to find
its reference files when you don't specify -r. It now checks
a couple of likely nearby directories to see if any of them
have a particular known file.

When atime wasn't specified, it should get set == mtime.
This is much stronger than just asserting that it's not zero.

If no atime was specified (for example, when extracting from ustar
archives), set atime == mtime. Before this, atime would get restored
to 0.

Alphabetize the list of tests.

Fix up test suite so it works again with the libarchive 1.x API.
In particular, FreeBSD 6 still uses the libarchive 1.x API and
this correction will permit MFCing new libarchive features back
to FreeBSD 6.

Correct test_tar_large so it works correctly with the libarchive 1.x API.

Explain why two apparently-identical archive_read_data() calls
really should have different return values.

Exclude .cvsignore files from $FreeBSD$ checking

Add $FreeBSD$ to the uuencoded test files that didn't have them already.

Add missing $FreeBSD$

Some tests won't build at WARNS level 6 due to aliasing violations.
Add missing -I. so the tests will build when ${.OBJDIR} != ${.CURDIR}.
${.OBJDIR} does not need to be spelled out.

Add regression test for CRC32 check. The test file has been modified to
include an invalid checksum for file2.

Approved by: kientzle

Allow the tests to build without libdmalloc.

Split out the reference zip file for ease of maintenance.

Refactor one of the ISO extraction tests: Move the reference
file into a separate file (instead of embedding it in the C code)
and use later timestamps (timestamps too close to the Epoch fail
predictably on systems that lack timegm(), whose mktime() doesn't
support dates before the Epoch and which are running in timezones
with negative offsets from GMT). The goal here is to test the ISO
extraction, not the local platform's time support.

MfP4: test harness improvements.

archive.h is no longer constructed from archive.h.in,
so we can rename it and drop some no-longer-necessary
build magic from the Makefile.

Fix the new generic link resolver in libarchive to never match
dirs as hardlinks. In particular, this fixes some recent ports
build failures.

Thanks to: Kris Kennaway

Until the old archive.h.in gets renamed to archive.h in the repository,
we still need some Makefile trickery to ensure archive.h is
correctly built for the test harness.

MFp4: libarchive 2.5.4b. (Still 'b' until I get a bit more
feedback, but the 2.5 branch is shaping up nicely.)

In addition to many small bug fixes and code improvements:
* Another iteration of versioning; I think I've got it right now.
* Portability: A lot of progress on Windows support (though I'm
not committing all of the Windows support files to FreeBSD CVS)
* Explicit tracking of MBS, WCS, and UTF-8 versions of strings
in archive_entry; the archive_entry routines now correctly return
NULL only when something is unset, setting NULL properly clears
string values. Most charset conversions have been pushed down to
archive_string.
* Better handling of charset conversion failure when writing or
reading UTF-8 headers in pax archives
* archive_entry_linkify() provides multiple strategies for
hardlink matching to suit different format expectations
* More accurate bzip2 format detection
* Joerg Sonnenberger's extensive improvements to mtree support
* Rough support for self-extracting ZIP archives. Not an ideal
approach, but it works for the archives I've tried.
* New "sparsify" option in archive_write_disk converts blocks of nulls
into seeks.
* Better default behavior for the test harness; it now reports
all failures by default instead of coredumping at the first one.

Free up the entry objects allocated during this test.

Release the buffers used for exercising the compress code.

Ignore a few more common files.

GC a reference to the defunct TESTFILES variable.

A subtle point: "pax interchange format" mandates that all strings
(including pathname, gname, uname) be stored in UTF-8. This usually
doesn't cause problems on FreeBSD because the "C" locale on FreeBSD
can convert any byte to Unicode/wchar_t and from there to UTF-8. In
other locales (including the "C" locale on Linux which is really
ASCII), you can get into trouble with pathnames that cannot be
converted to UTF-8.

Libarchive's pax writer truncated pathnames and other strings at the
first nonconvertible character. (ouch!) Other archivers have worked
around this by storing unconvertible pathnames as raw binary, a
practice which has been sanctioned by the Austin group. However,
libarchive's pax reader would segfault reading headers that weren't
proper UTF-8. (ouch!) Since bsdtar defaults to pax format, this
affects bsdtar rather heavily.

To correctly support the new "hdrcharset" header that is going into
SUS and to handle conversion failures in general, libarchive's pax reader
and writer have been overhauled fairly extensively. They used to do
most of the pax header processing using wchar_t (Unicode); they now do
most of it using char so that common logic applies to either UTF-8 or
"binary" strings.

As a bonus, a number of extraneous conversions to/from wchar_t have
been eliminated, which should speed things up just a tad.

Thanks to: Bjoern Jacke for originally reporting this to me
Thanks to: Joerg Sonnenberger for noting a bad typo in my first draft of this
Thanks to: Gunnar Ritter for getting the standard fixed
MFC after: 5 days

Ignore some built files.

Don't lie. If a string can't be converted to a wide (Unicode) string,
return a NULL instead of an incomplete string. Expand the test coverage
to verify the correct behavior here.

New public functions archive_entry_copy_link() and archive_entry_copy_link_w()
override the currently set link value, whether that's a hardlink
or a symlink. Plus documentation update and tests.

Rework the versioning implementation and test to match the
new interface. Mark the functions that are going away in
libarchive 3.0.

In particular, archive_version_string() now computes the
string rather than assuming that it will be created by the
build infrastructure. Eventually, this will allow some
simplification of the build infrastructure.

Support for writing 'compress' format, thanks to Joerg Sonnenberger.

Current 'ar' read support in libarchive can only handle a GNU/SVR4
filename table whose size is less than 65536 bytes.

The original intention was to not consume the filename table, so the
client will have a chance to look at it. To achieve that, the library
call decompressor->read_ahead to read(look ahead) but do not call
decompressor->consume to consume the data, thus a limit was raised
since read_ahead call can only look ahead at most BUFFER_SIZE(65536)
bytes at the moment, and you can not "look any further" before you
consume what you already "saw".

This commit will turn GNU/SVR4 filename table into "archive format
data", i.e., filename table will be consumed by libarchive, so the
65536-bytes limit will be gone, but client can no longer have access
to the content of filename table.

'ar' support test suite is changed accordingly. BSD ar(1) is not
affected by this change since it doesn't look at the filename table.

Reported by: erwin
Discussed with: jkoshy, kientzle
Reviewed by: jkoshy, kientzle
Approved by: jkoshy(mentor), kientzle

Portability: Eliminate the need for uudecode by incorporating
uudecode into the main test driver and invoking it just-in-time
within the various tests.

Also, incorporate a number of improvements to the main test support
code that have proven useful on other projects where I've used this
framework.

Tighten up the semantics of acl_next() and xattr_next() when you
hit the end of the ACL or xattr list.

Thanks to: Jeff Johnson for pointing out the obvious typo

The test_assert() function that backs my custom assert() macro
now returns a value, which supports such convenient
constructs as:
if (assert(NULL != foo())) {
}

Also be careful to setlocale("C") for each new test to
avoid locale pollution.

Also a couple of minor portability enhancements.

Add a test to verify compatibility with archives with
odd hardlinks. I need to extend this to test pax extended
archives with bodies attached to hardlinks and other less-common cases.

Explain a subtle API change that was made recently.
Even though I believe this is a good change, it does
have the potential to break certain clients, so it's
good to document the reasoning behind the change.

Properly pad symlinks when writing cpio "newc" format.
Thanks to: Jesse Barker for reporting this.
MFC after: 7 days

The previous commit caused the archive_write_disk interface to
start obeying filesize limits; this test wasn't properly setting
file sizes before trying to write file data.

Issues with hardlinks in newc-format files prompted me to
write a new test to exercise the hardlink strategies used
by different archive formats (tar, old cpio, new cpio).
This uncovered two problems, both fixed by this commit:

1) Enforce file size when writing files to disk.

2) When restoring hardlink entries, if they have data associated, go
ahead and open the file so we can write the data.

In particular, this fixes bsdtar/bsdcpio extraction of new cpio
formats where the "original" is empty and the subsequent "hardlink"
entry actually carries the data. It also provides correct behavior
for old cpio archives where hardlinked entries have their bodies
stored multiple times in the archive; the last body should always be
the one that ends up in the final file. The new pax format also
permits (but does not require) hardlinks to carry file data; again,
the last contents should always win.

Note that with any of these, a size of zero on a hardlink simply means
that the hardlink carries no data; it does not mean that the file has
zero size. A non-zero size on a hardlink does provide the file size.

Thanks to: John Baldwin, for reminding me about this long-standing bug
and sending me a simple example archive that prompted this test case

Since the tar bidder can never get called more than once, it
doesn't need to compensate for this situation.

While here, fix a minor longstanding bug that empty tar archives
(which begin with at least 512 zero bytes) never properly reported
their format. In particular, this fixes the output of:
bsdtar tvvf /dev/zero

And, of course, a new test to verify that libarchive correctly
recognizes the format of such files.

Extensive improvements to the libarchive_test test program that
exercises and verifies the libarchive APIs:

* Improved error reporting; hexdumps are now provided for
many file/memory content differences.
* Overall status more clearly counts "tests" and "assertions"
* Reference files can now be stored on disk instead of having
to be compiled into the test program itself. A couple of
tests have been converted to this more natural structure.
* Several memory leaks corrected so that leaks within libarchive
itself can be more easily detected and diagnosed.
* New test: GNU tar compatibility
* New test: Zip compatibility
* New test: Zero-byte writes to a compressed archive entry
* New test: archive_entry_strmode() format verification
* New test: mtree reader
* New test: write/read of large (2G - 1TB) entries to tar archives
(thanks to recent performance work, this test only requires a few seconds)
* New test: detailed format verification of cpio odc and newc writers
* Many minor additions/improvements to existing tests as well.

Correct the cpio writers to not accept data for non-regular files.
In particular, the previous code led to archives that had
non-empty bodies following directory entries. Not a fatal
problem, as bsdtar and GNU cpio are both happy to just skip
this bogus data, but it still shouldn't be there.

MFC after: 3 days

Fix the archive_write_data() function so it always returns
number of bytes written, even when used to write files to
disk. Extend the test suite to verify the correct return
values for archive_write_data() and archive_write_data_block().

Thanks to: Bruce Mah, for stepping in promptly to back out the
earlier broken version of this fix
Thanks to: Colin Percival, for pointing out the correct fix
MFC after: 5 days
Approved by: re (ksmith)
Pointy hat: \me

Revert the last commit to libarchive. It introduced some regresssions,
most noticably the incorrect extraction of files by bsdtar.

This commit reverts:

src/lib/libarchive/archive_write_disk.c 1.15
src/lib/libarchive/test/test_write_disk.c 1.4

Approved by: re (implicitly)

Correct the return value from archive_write_data()
(when used to restore files to disk) to match:
* The documentation
* The return values of this function when used
to write files into an archive.

Approved by: re (bmah)
Pointy hat: \me
MFC after: 5 days

This commit updates libarchive to be compatible with
GNU tar 1.17's implementation of --posix --sparse,
at the cost of losing compatibility with GNU tar 1.16.
Fortunately, the 1.17 implementation actually makes sense,
so the libarchive code is now a bit more straightforward
than before.

Background: GNU tar 1.16 defined a new way to store
sparse files in --posix archives. Unfortunately,
the implementation incorrectly inserted several
blocks of null padding after each such entry.
As a result, non-GNU tar implementations saw the
archive as truncated after any sparse entry.
This was fixed in GNU tar 1.17 at the cost of
losing compatibility with GNU tar 1.16 for this
new format (which is not the default, so hopefully
rarely used). Libarchive recently gained support
for reading the GNU tar 1.16 formats; this commit
updates it to read the GNU tar 1.17 variant instead.

Approved by: re (ksmith for libarchive portion)
Approved by: re (blanket for libarchive_test portion)
MFC after: 5 days

Make suid/sgid restore be "opportunistic" if
owner restore is not requested. If you ask
for permissions to be restored but not owner,
you will now get no error if suid/sgid bits
cannot be set. (It's a security hole to restore
suid/sgid bits if the owner/group aren't restored.)

This fixes an obscure problem where a simple
"tar -xf" with no other options will sometimes
fail gratuitously because of suid/sgid bits.
This is causing occasional problems for people
using bsdtar as a drop-in replacement for
"that other tar program." ;-)

Note: If you do ask for owner restore, then suid/sgid
restore failures still issue an error. This
only suppresses the error in the case where an
suid/sgid bit restore fails because of an owner
mismatch and owner restore was not requested.

Approved by: re (bmah)
MFC after: 7 days

Update the tests for reading the various GNU tar sparse formats.
In particular:
* Include a second entry in all of the test archives (to catch errors
with intermediate padding)
* Test the GNU tar 1.17 version of "posix sparse format 1.0"
instead of the GNU tar 1.16 version (the latter is no longer
supported by GNU tar).

Right now, libarchive fails this test because I originally
implemented the GNU tar 1.16 version of "posix sparse format 1.0".
I'll fix libarchive shortly.

Approved by: re (blanket, libarchive testing)

Two minor nits:
* Allow libarchive_test to compile on Interix again.
* Track the test name (not just line number) when counting skipped tests.

Thanks to: Joerg Sonnenberger
Approved by: re (blanket; libarchive testing)

Clarify one test.

Approved by: re (blanket, libarchive testing)

Use an uncompressed test archive for gtar sparse format 1.0 format,
as that better exercises some internal read-combining logic than the
compressed archive.

Approved by: re (blanket, libarchive testing)

Report each failed test once, but keep a count and report that count
if there was more than one. In particular, this simplifies
test_tar_filenames.c, which has a tendency to be very noisy otherwise.

Approved by: re (blanket, libarchive testing)

Extend the basic tar reading test to exercise most types of
entries. This doesn't cover everything yet, but it's a big improvement.

Approved by: re (blanket, libarchive testing)

Make the test for reading gtar sparse entries more robust;
it now verifies that the returned blocks have the correct data
at the correct file offsets, ignoring any null padding that
may exist.

Approved by: re (blanket, libarchive test suite)

New test suite test_read_pax_truncated probes libarchive
behavior with truncated or damaged pax archives. This
tests most of the cases covered by the recent security advisory.

Approved by: re (blanket, libarchive test suite)

New file "read_open_memory.c" is a custom variant of
archive_read_open_memory.c that tries to test border
cases. In particular, it copies over each returned block
so that formats or decompressors that read past the end
of a returned block will break.

Approved by: re (blanket, libarchive test suite)

Fix running individual tests via "libarchive_test <number> <number> ..."

Approved by: re (blanket)

Make test suite work with libarchive 1.3.1: Take advantage of
ARCHIVE_VERSION_STAMP to selectively disable tests that don't
apply to that version; new "skipping()" function reports skipped
tests; modify final summary to report component test failures and
skips.

Note: I don't currently intend to MFC the test suite itself;
anyone interested should just checkout and use this version
of the test suite, which should work for any library version.

Approved by: re (Ken Smith, blanket)

Ouch. I partially screwed up the last commit by
enabling a test that's not ready yet. <sigh>

Pointy hat: /me
Approved by: re@

Support for writing the 'newc' cpio format, plus a minimal test harness
for the cpio formats.

Thanks to: Rudolf Marek
Approved by: re@

Make 'ar' write test a tad more portable.

Read support for the new GNU tar sparse formats added in gtar 1.15 and
gtar 1.16.

Add some options to libarchive_test:
-k: like make -k, try to keep going after errors.
-q: quiet

Fix a broken function declaration.

Options spring cleanup:
- Add and document the KVM and KVM_SUPPORT options that
are needed for the ifmcstats(3) makefile
- Garbage collect unused variables
- Add missing inclusion of bsd.own.mk where needed

Approved by: kan (mentor)
Reviewed by: ru

Don't lose leading '/' for pathnames exactly 101 bytes long.
Also, update the test harness to exercise this case.

libarchive 2.2.3
* "compression_program" support uses an external program
* Portability: no longer uses "struct stat" as a primary
data interchange structure internally
* Part of the above: refactor archive_entry to separate
out copy_stat() and stat() functions
* More complete tests for archive_entry
* Finish archive_entry_clone()
* Isolate major()/minor()/makedev() in archive_entry; remove
these from everywhere else.
* Bug fix: properly handle decompression look-ahead at end-of-data
* Bug fixes to 'ar' support
* Fix memory leak in ZIP reader
* Portability: better timegm() emulation in iso9660 reader
* New write_disk flags to suppress auto dir creation and not
overwrite newer files (for future cpio front-end)
* Simplify trailing-'/' fixup when writing tar and pax
* Test enhancements: fix various compiler warnings, improve
portability, add lots of new tests.
* Documentation: document new functions, first draft of
libarchive_internals.3

MFC after: 14 days
Thanks to: Joerg Sonnenberger (compression_program)
Thanks to: Kai Wang (ar)
Thanks to: Colin Percival (many small fixes)
Thanks to: Many others who sent me various patches and problem reports.

Clean up a lot of memory leaks in the libarchive test harness.

Overhaul of 'ar' support:
* use "AR_GNU" as the format name instead of AR_SVR4 (it's what everyone is going to call it anyway)
* Simplify numeric parsing to unsigned (none of the numeric values should ever be negative); don't run off end of numeric fields.
* Finish parsing the common header fields before the next I/O request (which might dump the contents)
* Be smarter about format guessing and trimming filenames.
* Most of the magic values are only used in one place, so just inline them.
* Many more comments.
* Be smarter about handling damaged entries; return something reasonable.
* Call it a "filename table" instead of a "string table"
* Update tests.

Enable selection of 'ar', 'arbsd', and 'argnu' formats by name
(this allows bsdtar to create ar format archives).

The 'ar' writer still needs some work; it should reject
entries that aren't regular files and should probably also
strip leading paths from filenames.

Conventionally, tar archives have always included a trailing '/'
for directories. bsdtar used to add this, but that recently got
lost somehow. So now I'm adding it back in libarchive.
The only odd part of doing this in libarchive: Adding a directory to
a tar archive and then reading it back again can yield a different name.

Add a test case to exercise some boundary conditions with
tar filenames and ensure that trailing slashes are added to
dir names only as necessary.

Thanks to: Oliver Lehmann for bringing this regression to my attention.

Portability: Don't use mkdtemp() when mkdir() will suffice.
If we can't create the dir, just give up.

Enable 'ar' support; hook it up to the build and
enable it with _read_support_format_all().

'ar' format support for libarchive, contributed by Kai Wang.

Notice when mkdir() fails.
Don't change permissions on an existing dir unless _EXTRACT_PERM
is requested.

In particular, bsdtar -x should not edit mode of existing dirs
now; bsdtar -xp will.

Libarchive 2.0.23:
* The ACL formatter was mis-formatting entries which had a
user/group ID but no name. Make the parser tolerant of
these, so that old archives can be correctly restored;
fix the formatter to generate correct entries.
* Fix overwrite detection by introducing a new "FAILED" return
code that indicates the current entry cannot be continued
but the archive as a whole is still sound.
* Header cleanup: Remove some unused headers, add some that
are required with new Linux systems.

New tests for system-independent ACL support.
These tests verify that archive_entry objects can store and return
ACL data and that pax format archives can read and write ACL
information. These do not (yet) test that ACL data is read or
written to disk correctly. (And hence would not have caught the
recent snafu about ACL read-from-disk being turned off.)

libarchive 2.0
* libarchive_test program exercises many of the core features
* Refactored old "read_extract" into new "archive_write_disk", which
uses archive_write methods to put entries onto disk. In particular,
you can now use archive_write_disk to create objects on disk
without having an archive available.
* Pushed some security checks from bsdtar down into libarchive, where
they can be better optimized.
* Rearchitected the logic for creating objects on disk to reduce
the number of system calls. Several common cases now use a
minimum number of system calls.
* Virtualized some internal interfaces to provide a clearer separation
of read and write handling and make it simpler to override key
methods.
* New "empty" format reader.
* Corrected return types (this ABI breakage required the "2.0" version bump)
* Many bug fixes.