Remove $FreeBSD$: one-line sh pattern

Remove /^\s*#[#!]?\s*\$FreeBSD\$.*$\n/

sh: Restore $((x)) error checking after fix for $((-9223372036854775808))

SVN r342880 was designed to fix $((-9223372036854775808)) and things like
$((0x8000000000000000)) but also broke error detection for values of
variables without dollar sign ($((x))).

For compatibility, overflow in plain literals continues to be ignored and
the value is clamped to the boundary (except 9223372036854775808 which is
changed to -9223372036854775808).

Reviewed by: se (although he would like error checking to be removed)
MFC after: 2 weeks
X-MFC-with: r342880
Differential Revision: https://reviews.freebsd.org/D18926

Fix an edge case when parsing large numbers which resulted in inconsistent
results between an expression that refers to a variable by name and the
same expression that includes the same variable by value.

Submitted by: se@
MFC after: 1 week

sh: Fix ${param?} default error message

If word in ${param?word} is missing, the shell shall write a default error
message. So expanding ${param?} when param is not set should write an error
message like

sh: param: parameter not set

This was broken by r316417.

PR: 233585

sh: Allow unquoted newlines in word in ${param+word} etc.

POSIX requires accepting unquoted newlines in word in parameter expansions
like ${param+word}, ${param#word}, although the Bourne shell did not support
it, it is not commonly used and might make it harder to find a missing
closing brace.

It was also strange that something like

foo="${bar#
}"

was rejected.

Reported by: Martijn Dekker via Robert Elz

sh: Add some tests for command substitution final newline stripping.

sh: Add tests for NUL byte in command substitution output.

sh: Fix executing wrong command with ${unsetvar#$(cmdsubst)}$(cmdsubst).

The parsed internal representation of words consists of a byte string with a
list of nodes (commands in command substitution). Each unescaped CTLBACKQ or
CTLBACKQ | CTLQUOTE byte corresponds to an entry in the list.

If param in ${param#%##%%word} is not set, the word is not expanded (in a
deviation of POSIX shared with other ash variants and ksh93). Erroneously,
the pointer in the list of commands (argbackq) was not advanced. This caused
the wrong command to be executed later if the outer word contained another
command substitution.

Example:
echo "${unsetvar#$(echo a)}$(echo b)"
wrote "a" but should write "b".

MFC after: 1 week

sh: Fix crash if a -T trap is taken during command substitution.

Code like t=$(stat -f %m "$file") segfaulted if -T was active and a trap
was taken while the shell was waiting for the child process to finish.

What happened was that the dotrap() call in waitforjob() was hit. This
re-entered command execution (including expand.c) at a point not expected by
expbackq(), and global state (unallocated stack string and argbackq) was
corrupted.

To fix this, change expbackq() to prepare for command execution to be
re-entered.

Reported by: bdrewery
MFC after: 1 week

sh: Add some already working tests that exercise new code paths.

Path generation was not according to collate

Approved by: jilles

Merge ^/user/ngie/release-pkg-fix-tests to unbreak how test files are installed
after r298107

Summary of changes:

- Replace all instances of FILES/TESTS with ${PACKAGE}FILES. This ensures that
namespacing is kept with FILES appropriately, and that this shouldn't need
to be repeated if the namespace changes -- only the definition of PACKAGE
needs to be changed
- Allow PACKAGE to be overridden by callers instead of forcing it to always be
`tests`. In the event we get to the point where things can be split up
enough in the base system, it would make more sense to group the tests
with the blocks they're a part of, e.g. byacc with byacc-tests, etc
- Remove PACKAGE definitions where possible, i.e. where FILES wasn't used
previously.
- Remove unnecessary TESTSPACKAGE definitions; this has been elided into
bsd.tests.mk
- Remove unnecessary BINDIRs used previously with ${PACKAGE}FILES;
${PACKAGE}FILESDIR is now automatically defined in bsd.test.mk.
- Fix installation of files under data/ subdirectories in lib/libc/tests/hash
and lib/libc/tests/net/getaddrinfo
- Remove unnecessary .include <bsd.own.mk>s (some opportunistic cleanup)

Document the proposed changes in share/examples/tests/tests/... via examples
so it's clear that ${PACKAGES}FILES is the suggested way forward in terms of
replacing FILES. share/mk/bsd.README didn't seem like the appropriate method
of communicating that info.

MFC after: never probably
X-MFC with: r298107
PR: 209114
Relnotes: yes
Tested with: buildworld, installworld, checkworld; buildworld, packageworld
Sponsored by: EMC / Isilon Storage Division

Fix build.

Sponsored by: The FreeBSD Foundation

First pass to fix the 'tests' packages.

Sponsored by: The FreeBSD Foundation

sh: Add tests for #/##/%/%% on $* and $@.

Although POSIX leaves things like ${*#X} unspecified, it occasionally occurs
in practice. Add some tests that seem to work sensibly.

sh: Add more tests for exotic IFS splitting.

sh: Add simple tests for globbing/splitting command substitution.

sh: Fix corruption of CTL* bytes in positional parameters in redirection.

EXP_REDIR was not being checked for while expanding positional parameters in
redirection, so CTL* bytes were not being prefixed where they should be.

MFC after: 1 week

sh: Mask off shift distance (<< and >>) in arithmetic.

In C, shift distances equal to or larger than the number of bits in the
operand result in undefined behaviour. As part of eliminating undefined
behaviour in arithmetic, mask off the distance like Java and JavaScript
specify and C on x86 usually does.

Assumption: conversion from unsigned to signed retains the two's complement
bits.
Assumption: uintmax_t has no padding bits.

Convert bin/sh/tests to ATF

The new code uses a "test discovery mechanism" to determine
what tests are available for execution

The test shell can be specified via:

kyua test -v test_suites.FreeBSD.bin.sh.test_shell=/path/to/test/sh

Sponsored by: EMC / Isilon Storage Division
Approved by: jmmv (mentor)
Reviewed by: jilles (maintainer)

Migrate tools/regression/bin/ tests to the new layout.

This change is a proof of concept on how to easily integrate existing
tests from the tools/regression/ hierarchy into the /usr/tests/ test
suite and on how to adapt them to the new layout for src.

To achieve these goals, this change:

- Moves tests from tools/regression/bin/<tool>/ to bin/<tool>/tests/.
- Renames the previous regress.sh files to legacy_test.sh.
- Adds Makefiles to build and install the tests and all their supporting
data files into /usr/tests/bin/.
- Plugs the legacy_test test programs into the test suite using the new
TAP backend for Kyua (appearing in 0.8) so that the code of the test
programs does not have to change.
- Registers the new directories in the BSD.test.dist mtree file.

Reviewed by: freebsd-testing
Approved by: rpaulo (mentor)