MFC r368207,368607:

MFC r368207:
Update libarchive to 3.5.0

Relevant vendor changes:
Issue #1258: add archive_read_support_filter_by_code()
PR #1347: mtree digest reader support
Issue #1381: skip hardlinks pointing to itself on extraction
PR #1387: fix writing of cpio archives with hardlinks without file type
PR #1388: fix rdev field in cpio format for device nodes
PR #1389: completed support for UTF-8 encoding conversion
PR #1405: more formats in archive_read_support_format_by_code()
PR #1408: fix uninitialized size in rar5_read_data
PR #1409: system extended attribute support
PR #1435: support for decompression of symbolic links in zipx archives
Issue #1456: memory leak after unsuccessful archive_write_open_filename

MFC r368607:
Sync libarchive with vendor.

Vendor changes:
Issue #1461: Unbreak build without lzma
Issue #1462: warc reader: Fix build with gcc11
Issue #1463: Fix code compatibility in test_archive_read_support.c
Issue #1464: Use built-in strnlen on platforms where not available
Issue #1465: warc reader: fix undefined behaviour in deconst() function

MFC r361294:
Update libarchive to 3.4.3

Relevant vendor changes:
PR #1352: support negative zstd compression levels
PR #1359: improve zstd version checking
PR #1348: support RHT.security.selinux from GNU tar
PR #1357: support for archives compressed with pzstd
PR #1367: fix issues in acl tests
PR #1372: child handling cleanup
PR #1378: fix memory leak from passphrase callback

MFC r356212,r356366,r356416,r357785
Update libarchive to version 3.4.2

Relevant vendor changes (r356212):
Issue #351: Refactor and implement private state logic for write filters
PR #1252: RAR5 reader - verify window size for solid files (OSS-Fuzz 15482)
PR #1255: zip writer - don't append unused NUL for directories
PR #1260: Fix sparse file offset overflow on 32-bit systems
PR #1263: UNICODE filename support for reading lha/lzh format
Issue #1276: Bugfix and optimize archive_wstring_append_from_mbs()
PR #1288: Add the "xattrhdr" option to pax write options
PR #1295: 7z reader - fix reading archives with digests in PackInfo
PR #1296: RAR5 reader - verify window size for multivolume archives
PR #1297: ZIP reader - support LZMA_STREAM_END marker in 'lzma alone' files
Issue #1298: Fix a heap-buffer-overflow in archive_string_append_from_wcs()
OSS-Fuzz 19360, 19362: LHA reader - plug two memory leaks on error
Fix possible off-by-one when dealing with readlink(2)

Relevant vendor changes (r356366):
Issue #1302: Plug memory leak on failure of archive_write_client_open()

Relevant vendor changes (r356416):
Issue #1302: Re-do fix for archive_write_client_open()

Relevant vendor changes (r357785):
PR #1289: atomic extraction support (bsdtar -x --safe-writes)
PR #1308: big endian fix for UTF16 support in LHA reader
PR #1326: reject RAR5 files that declare invalid header flags
Issue #987: fix support 7z archive entries with Delta filter
Issue #1317: fix compression output buffer handling in XAR writer
Issue #1319: fix uname or gname longer than 32 characters in pax writer
Issue #1325: fix use after free when archiving hardlinks in ISO9660 or XAR
Use localtime_r() and gmtime_r() instead of localtime() and gmtime()

MFC r348993,349135:
Sync libarchive with vendor including security fixes

r348993:
- version bumped to 3.4.0
- check_symlinks_fsobj() without chdir() and fchdir()
- bsdtar.1 manpage fixes
- patches from OpenBSD to libarchive_fe/passphrase.c

r349135:
PR #1212: RAR5 reader - window_mask was not updated correctly
(OSS-Fuzz 15278)
OSS-Fuzz 15120: RAR reader - extend use after free bugfix

MFC r338600:
Update libarchive to 3.3.3

As all important changes have already been merged from libarchive git
this is just a version number bump, documentation update and some
polishing for cpio tests. Other source code changes are not relevant to
FreeBSD.

Relnotes: yes

MFH r324148:
Sync libarchive with vendor.

Relevant vendor changes:
PR #905: Support for Zstandard read and write filters
PR #922: Avoid overflow when reading corrupt cpio archive
Issue #935: heap-based buffer overflow in xml_data (CVE-2017-14166)
OSS-Fuzz 2936: Place a limit on the mtree line length
OSS-Fuzz 2394: Ensure that the ZIP AES extension header is large enough
OSS-Fuzz 573: Read off-by-one error in RAR archives (CVE-2017-14502)

Security: CVE-2017-14166, CVE-2017-14502

MFC r320927,320931,320932:
Bump libarchive to 3.3.2

Vendor changes:
PR #901: don't depend on stdin in a testcase

Relnotes: yes

MFC r315636,315876,316095:
Sync libarchive with vendor

Vendor changes/bugfixes (FreeBSD-related):
r315636:
PR 867 (bsdcpio): show numeric uid/gid when names are not found
PR 870 (seekable zip): accept files with valid ZIP64 EOCD headers
PR 880 (pax): Fix handling of "size" pax header keyword
PR 887 (crypto): Discard 3072 bytes instead of 1024 of first keystream
OSS-Fuzz issue 806 (mtree): rework mtree_atol10 integer parser
Break ACL read/write code into platform-specific source files

r315876:
Store extended attributes with extattr_set_link() if no fd is provided
Add extended attribute tests to libarchive and bsdtar
Fix tar's test_option_acls
Support the UF_HIDDEN file flag

r316095:
Constify variables in several places
Unify platform ACL code in a single source file
Fix unused variable if compiling on FreeBSD without NFSv4 ACL support

MFC r314571:
Update libarchive to version 3.3.1 (and sync with latest vendor dist)

Notable vendor changes:
PR #501: improvements in ACL path handling
PR #724: fix hang when reading malformed cpio files
PR #864: fix out of bounds read with malformed GNU tar archives
Documentation, style, test suite improvements and typo fixes.

New options to bsdtar that enable or disable reading and/or writing of:
Access Control Lists (--acls, --no-acls)
Extended file flags (--fflags, --no-fflags)
Extended attributes (--xattrs, --no-xattrs)
Mac OS X metadata (Mac OS X only) (--mac-metadata, --no-mac-metadata)

MFC r307861:
Update libarchive to 3.2.2

Most of the post-3.2.1 fixes have already been merged. This update contains
just the version bump and some fixes to the test framework.

MFC r302075:

Update libarchive to 3.2.1 (bugfix and security fix release)

List of vendor fixes:
- fix exploitable heap overflow vulnerability in Rar decompression
(vendor issue 719, CVE-2016-4302, TALOS-2016-0154)
- fix exploitable stack based buffer overflow vulnebarility in mtree
parse_device functionality (vendor PR 715, CVE-2016-4301, TALOS-2016-0153)
- fix exploitable heap overflow vulnerability in 7-zip read_SubStreamsInfo
(vendor issue 718, CVE-2016-4300, TALOS-2016-152)
- fix integer overflow when computing location of volume descriptor
(vendor issue 717)
- fix buffer overflow when reading a crafred rar archive (vendor issue 521)
- fix possible buffer overflow when reading ISO9660 archives on machines
where sizeof(int) < sizeof(size_t) (vendor issue 711)
- tar and cpio should fail if an input file named on the command line is
missing (vendor issue 708)
- fix incorrect writing of gnutar filenames that are exactly 512 bytes
long (vendor issue 682)
- allow tests to be run from paths that are equal or longer than 128
characters (vendor issue 657)
- add memory allocation errors in archive_entry_xattr.c (vendor PR 603)
- remove dead code in archive_entry_xattr_add_entry() (vendor PR 716)
- fix broken decryption of ZIP files (vendor issue 553)
- manpage style, typo and description fixes

Post-3.2.1 vendor fixes:
- fix typo in cpio version reporting (Vendor PR 725, 726)
- fix argument range of ctype functions in libarchive_fe/passphrase.c
- fix ctype use and avoid empty loop bodies in WARC reader

Security: CVE-2016-4300, CVE-2016-4301, CVE-2016-4302

MFC r299529,r299540,r299576,r299896:

r299529,r299540:
Update libarchive to 3.2.0

New features:
- new bsdcat command-line utility
- LZ4 compression (in src only via external utility from ports)
- Warc format support
- 'Raw' format writer
- Zip: Support archives >4GB, entries >4GB
- Zip: Support encrypting and decrypting entries
- Zip: Support experimental streaming extension
- Identify encrypted entries in several formats
- New --clear-nochange-flags option to bsdtar tries to remove noschg and
similar flags before deleting files
- New --ignore-zeros option to bsdtar to handle concatenated tar archives
- Use multi-threaded LZMA decompression if liblzma supports it
- Expose version info for libraries used by libarchive

r299576,r299896:
Fix broken cpio behavior.

Relnotes: yes

MFC: xz 5.2.2.

Relnotes: yes

MFC r291329:

Remove redundant DPSRCS which were already in SRCS.

MFC r289195:

Integrate the tests from lib/libarchive, usr.bin/cpio, and usr.bin/tar in to
the FreeBSD test suite

functional_test.sh was ported from bin/sh/tests/functional_test.sh, as a
small wrapper around libarchive_test, bsdcpio_test, and bsdtar_test provided
by upstream.

A handful of testcases in lib/libarchive/tests have been disabled as they
were failing when run with kyua test (see BROKEN_TESTS in
lib/libarchive/tests/Makefile)

As a sidenote: this removes the check/test targets from the Makefiles as they
don't match the pattern used in the rest of the FreeBSD test suite.

Sponsored by: EMC / Isilon Storage Division

Conflicts:
lib/libarchive/test
usr.bin/cpio/test

MFC r264400,r265836:

r264400:

NO_MAN= has been deprecated in favor of MAN= for some time, go ahead
and finish the job. ncurses is now the only Makefile in the tree that
uses it since it wasn't a simple mechanical change, and will be
addressed in a future commit.

r265836:

Remove last two NO_MAN= in the tree. In both of these cases, MAN= is
what is needed.

Copy head (r256279) to stable/10 as part of the 10.0-RELEASE cycle.

Approved by: re (implicit)
Sponsored by: The FreeBSD Foundation

MFV r248590,248594:
Update libarchive to 3.1.2

Some of new features:
- support for lrzip and grzip compression
- support for writing tar v7 format
- b64encode and uuencode filters
- support for __MACOSX directory in Zip archives
- support for lzop compresion (external utility)

Update libarchive to 3.0.4

Update libarchive to 3.0.3

Some of new features:
- New readers: RAR, LHA/LZH, CAB reader, 7-Zip
- New writers: ISO9660, XAR
- Improvements to many formats, especially including ISO9660 and Zip
- Stackable write filters to write, e.g., tar.gz.uu in a single pass
- Exploit seekable input; new "seekable" Zip reader can exploit the Zip
Central Directory when it's available; the old "streamable" Zip reader
is still fully supported for cases where seeking is not possible.

Full release notes available at:
https://github.com/libarchive/libarchive/wiki/ReleaseNotes

Use contrib sources for building libarchive, tar and cpio.
Make "make test" fully operational.

MFC after: 2 weeks

Sync bsdcpio with vendor branch release/2.8:

Revision 3770:
Merge r3768 from trunk: Fix typo in dev/ino verification for cpio formats.

Obtained from: http://code.google.com/p/libarchive
MFC after: 2 weeks

Update libarchive, tar and cpio to version 2.8.5

The following additional vendor revisions are applied:

Revision 3740:
Use archive_clear_error() to clear the error markers.

Obtained from: http://code.google.com/p/libarchive
MFC after: 2 weeks

Correctly link bsdcpio and bsdtar against libmd and libcrpyto
by applying the change from r221472 (libarchive).

Reviewed by: kientzle
Approved by: re (kib)
MFC after: 3 days

Append to CPIO_SRCS instead of using TAR_SRCS,
the variable name is misleading

MFC after: 2 weeks

Update bsdcpio to 2.8.4
Use common code from lib/libarchive/libarchive_fe

Approved by: kientzle
MFC after: 2 weeks

Fix typo in bsdcpio manual:
s/libarchive_formats/libarchive-formats

PR: 150300
Submitted by: Kurt Jaeger <fbsd-pr at opsec dot eu>
Patch by: Kurt Jaeger
Approved by: keramida (mentor)
MFC after: 1 week

Fix typos, spelling, formatting and mdoc mistakes found by Nobuyuki while
translating these manual pages. Minor corrections by me.

Submitted by: Nobuyuki Koganemaru <n-kogane@syd.odn.ne.jp>

Fix -R when used with -p. Previously, the
uname and gname weren't overwritten, so the
disk restore would use those to lookup the
original uid/gid again. Clearing the uname
and gname prevents this.

Reported by: swell.k
MFC after: 7 days

Enable liblzma support in libarchive
Adjust dependencies for programs using libarchive
Add xz and linkage against liblzma to rescue system

Approved by: kientzle, delphij (mentor)
MFC after: 2 weeks

bsdcpio 2.8.3

Remove GNU cpio after fix of CVE-2010-0624.

Note that this is actually a no-op for most users, as this GNU
cpio was broken on -HEAD and 8-STABLE since last March until
the recent fix.

FreeBSD 8.0+ uses BSD cpio by default and the code is being
actively maintained.

Blessed by: kientzle
With hat: secteam
MFC after: 3 days

Fixed dependencies (make checkdpadd).

Fix common misspelling of hierarchy

Pointed out by: bf1783 at gmail
Approved by: np (cxgb), kientzle (tar, etc.), philip (mentor)

Support -V option to print a dot for each file processed.
Also, change the existing -vi behavior to send the filenames to
stderr rather than stdout, as GNU cpio does.

PR: bin/141175
Submitted by: Philip Kizer
MFC after: 14 days

Build usr.bin/ with WARNS=6 by default.

Also add some missing $FreeBSD$ to keep svn happy.

This addresses some issues with my earlier -R fix that
were pointed out by Brooks Davis and Alexey Dokuchaev:
* It now tries to lookup arguments as names first, then tries
to parse them as numbers. In particular, this makes the
behavior consistent with POSIX conventions when usernames
consist entirely of digits.
* It now uses strtoul() for the numeric parsing.

Finally, I've included an update to the test harness
to exercise the new numeric cases for -R.

Approved by: re (kib)

This fixes bsdcpio's -R option to accept numeric
user or group Ids as well as user or group names.
In particular, this fixes freesbie2, which uses
-R 0:0 to copy a bunch of files so that the result
will be owned by root.

Also fixes a related bug that mixed-up the uid
and gid specified by -R when in passthrough mode.

Thanks to Dominique Goncalves for reporting this
regression.

Approved by: re (kib)

When compiled for the release crunches, be a bit
more selective about what libarchive features we pull in:
* No compression support
* Only cpio and ustar writing
* Only cpio and tar/pax readers
This reduces a statically linked, stripped binary from 900k to 680k
and completely eliminates the dependency on libcrypto.

Make -lcrypto usage dependent on whether or not we're building with OpenSSL.

Merge from libarchive.googlecode.com:
* Lots of new tests.
* New -n / --numeric-uid-gid option
* More sanity-checking of arguments
* Various Windows portability improvements
* Sync up version number to 2.7.0

Custom command line parser for cpio; this is a little more
code but should be a lot fewer cross-platform compatibility
headaches.

Make the GCC-specific __dead2 markers conditional on whether
we're compiling under GCC.

Format the output of -itv for real. In particular:
* Lookup uname/gname if not provided by the archive (I copied the
uname/gname lookup cache from bsdtar)
* Format device number instead of size for device nodes
* Format date.

There's still a few improvements that I could copy from
bsdtar, especially the locale-aware safe_fprintf() code
and the locale-aware setup for day_first date formatting.
(And, of course, I need to think through a clean way to
push this stuff down into libarchive.)

Thanks to Peter Wemm for reminding me of this overlooked TODO item.

Don't destroy the archive until after you finish pulling useful
information out of it. As reported by Giorgos Keramidas.

MfP4: Verify correct interaction with umask: Add another
file with different permissions and set a non-zero umask
during the actual copy tests. The extra entry increases
the size of the test archives of course, so adjust the
expected sizes.

Update the total archive byte counters when writing entries to disk using
archive_write_disk.
Update cpio to use this to emit block counts in -p mode.
Update cpio tests to verify these block counts.

straighten out the "clean" target

svn:ignore built files

cpio -v emits a line for every item copied.

Update the passthrough_dotdot test to reproduce a
problem reported by Kris Kennaway.

PR: bin/124924

Test for proper handling of "cpio -p .."

PR: bin/124924

Test for a bug reported by Bernd Walter: In passthrough mode,
copying "dir/file" and then copying "dir" results in
"File on disk is not older; skipping" for the "dir" because
it was implicitly created by "dir/file." Among other sins,
this means that "dir" ends up with the wrong permissions
and ownership.

This is actually a libarchive bug; fix is forthcoming.

Comment a couple of places where bsdcpio and gcpio 2.9 disagree.
The number of blocks read from ustar archives is just an implementation
difference. The failure of bsdcpio to emit a block count to stderr
in -p mode is a real bug in bsdcpio.

The newc-format verification is now a little smarter about
following the archive structure. In particular, it no longer
crashes if you run it against GNU cpio 2.9 (although it does
still complain a lot more than it should).

cpio should restore file flags and ACLs when they're available

Mark functions as __dead2 in order to help the LLVM static checker
understand which code paths aren't possible.

This commit eliminates 117 false positive bug reports of the form
"allocate memory; error out if pointer is NULL; use pointer".

MfP4: Preserve permissions by default.

In particular, this fixes the oddity that -dumpl would apply
umask to copied dirs (which are created in the target tree)
but not to "copied" files (which are only linked). After
this change:

$ ls -ld a a/b a/b/c
d--x-w-r-- 3 tim tim 512 Jul 29 20:08 a
drwxr----x 3 tim tim 512 Jul 29 20:09 a/b
dr----x-w- 2 tim tim 512 Jul 29 20:09 a/b/c
$ (echo a; echo a/b; echo a/b/c) | cpio -dumpl o
$ cd o
$ ls -ld a a/b a/b/c
d--x-w-r-- 3 tim tim 512 Jul 29 20:08 a
drwxr----x 3 tim tim 512 Jul 29 20:09 a/b
dr----x-w- 2 tim tim 512 Jul 29 20:09 a/b/c

Add --no-preserve-owner, which seems to be required by some ports.

Thanks to: Erwin Lansing

Preserve ownership if cpio is run as root.

bsdcpio is now at version 1.0.0.

In -pl mode, only hardlink regular files. I need to test
other implementations, but it's clear that dirs and symlinks,
at least, shouldn't be hardlinked.

Remove an unused define.

Don't try to hardlink directories. While I'm here, expand some
comments to make this section of code a little clearer.

As reported by Alexey Shuvaev, -dumpl overwrote files after
linking them, with predictably bad results.

In -p mode, don't gaurd against '..' in paths. We continue to
check in -i mode unless --insecure is specified.

PR: bin/124924

If we're using -l and can't hardlink the file because of a cross-device
link, just ignore the -l option and copy the file instead.
In particular, this should fix the COPYTREE_* macros used in the
ports infrastructure which use -l to preserve space but often get
used for cross-device copies.

Rework line-processing framework to add support for --null and
to eliminate a callback.

Various long options for GNU cpio compat.

MfP4: test improvements, mostly for portability.

Include bsd.own.mk to pick up the definition of MK_GNU_CPIO

1. Make the BSD version of cpio the default [1]
a. The BSD version will be built and installed unless
WITHOUT_BSD_CPIO is defined.
b. The GNU version will not be built or installed unless
WITH_GNU_CPIO is defined. If this is defined, the symlink
in /usr/bin will be to the GNU version whether the BSD
version is present or not.

When these changes are MFCed the defaults should be flipped.

2. Add a knob to disable the building of GNU grep. This will
make it easier for those that want to test the BSD version in
the ports.

Approved by: kientzle [1]

bsdcpio is always installed as 'bsdcpio', symlink it to 'cpio'
only if WITH_BSDCPIO is defined.

Initial commit of bsdcpio 0.9.11b.

A new implementation of cpio that uses libarchive as it's back-end
archiving/dearchiving infrastructure. Includes test harness;
"make check" in the bsdcpio directory to build and run the test
harness.