MFC r315005: sh: Fix executing wrong command with ${x#$(y)}$(z).

The parsed internal representation of words consists of a byte string with a
list of nodes (commands in command substitution). Each unescaped CTLBACKQ or
CTLBACKQ | CTLQUOTE byte corresponds to an entry in the list.

If param in ${param#%##%%word} is not set, the word is not expanded (in a
deviation of POSIX shared with other ash variants and ksh93). Erroneously,
the pointer in the list of commands (argbackq) was not advanced. This caused
the wrong command to be executed later if the outer word contained another
command substitution.

Example:
echo "${unsetvar#$(echo a)}$(echo b)"
wrote "a" but should write "b".

MFC r314686: sh: Fix crash if a -T trap is taken during command substitution.

Code like t=$(stat -f %m "$file") segfaulted if -T was active and a trap
was taken while the shell was waiting for the child process to finish.

What happened was that the dotrap() call in waitforjob() was hit. This
re-entered command execution (including expand.c) at a point not expected by
expbackq(), and global state (unallocated stack string and argbackq) was
corrupted.

To fix this, change expbackq() to prepare for command execution to be
re-entered.

In stable/10, there is more global state that needs to be restored than in
stable/11 and head.

Reported by: bdrewery

MFC r278818: sh: Add stsavestr(), like savestr() but allocates using
stalloc().

MFC r288430: wordexp: Rewrite to make WRDE_NOCMD reliable.

Shell syntax is too complicated to detect command substitution and unquoted
operators reliably without implementing much of sh's parser. Therefore, have
sh do this detection.

While changing sh's support anyway, also read input from a pipe instead of
arguments to avoid {ARG_MAX} limits and improve privacy, and output count
and length using 16 instead of 8 digits.

The basic concept is:
execl("/bin/sh", "sh", "-c", "freebsd_wordexp ${1:+\"$1\"} -f "$2",
"", flags & WRDE_NOCMD ? "-p" : "", <pipe with words>);

The WRDE_BADCHAR error is still implemented in libc. POSIX requires us to
fail strings containing unquoted braces with code WRDE_BADCHAR. Since this
is normally not a syntax error in sh, there is still a need for checking
code in libc, we_check().

The new we_check() is an optimistic check that all the characters
<newline> | & ; < > ( ) { }
are quoted. To avoid duplicating too much sh logic, such characters are
permitted when quoting characters are seen, even if the quoting characters
may themselves be quoted. This code reports all WRDE_BADCHAR errors; bad
characters that get past it and are a syntax error in sh return WRDE_SYNTAX.

Although many implementations of WRDE_NOCMD erroneously allow some command
substitutions (and ours even documented this), there appears to be code that
relies on its security (codesearch.debian.net shows quite a few uses).
Passing untrusted data to wordexp() still exposes a denial of service
possibility and a fairly large attack surface.

This is also a MFC of r286830 to reduce conflicts. I changed the code
somewhat to avoid changes from r286941; in particular, WRDE_BADVAL can still
only be returned if WRDE_UNDEF was passed.

Relnotes: yes
Security: fixes command execution with wordexp(untrusted, WRDE_NOCMD)

MFC r287148: sh: Fix out of bounds read when there is no ] after a [:class:].

The initial check for a matching ] was incorrect if a ] may be consumed by a
[:class:]. The subsequent loop assumed that there must be a ].

Remove the initial check and make the loop cope with a missing ].

Found with afl-fuzz.

MFC r278826,279330: sh: Various cleanups to expand.c:

* Remove some gotos.
* Remove unused parameter.
* Remove duplicate code.

This MFC is to avoid conflicts in the MFC of r287148.

MFC r272575: sh: Eliminate some gotos.

This MFC is to avoid conflicts in the MFC of r287148.

MFC r276037: sh: Remove EXP_REDIR.

EXP_REDIR was supposed to generate pathnames in redirection if exactly one
file matches, as permitted but not required by POSIX in interactive mode. It
is unlikely this will be implemented.

No functional change is intended.

MFC r274854: sh: Use DQSYNTAX only while expanding, not SQSYNTAX.

Quoting during expansion only cares about CCTL, which is the same for
DQSYNTAX and SQSYNTAX.

MFC r273920: sh: Fix corruption of CTL* bytes in positional parameters in
redirection.

EXP_REDIR was not being checked for while expanding positional parameters in
redirection, so CTL* bytes were not being prefixed where they should be.

MFC r268576: sh: Correctly handle positional parameters beyond INT_MAX on
64-bit systems.

Currently, there can be no more than INT_MAX positional parameters. Make
sure to treat all higher ones as unset to avoid incorrect results and
crashes.

On 64-bit systems, our atoi() takes the low 32 bits of the strtol() and
sign-extends them.

On 32-bit systems, the call to atoi() returned INT_MAX for too high values
and there is not enough address space for so many positional parameters, so
there was no issue.

PR: 195918

MFC r263777: sh: Fix possible memory leaks and double frees with unexpected
SIGINT.

MFC r263195: sh: Add some consts.

MFC r262755: sh: Make argstr() return where it stopped and simplify expari()
using this.

Sync sh(1) in stable/10 to head.

This is a MFC of all the commits listed below.

My original goal of this change was to only merge the move of the tests
from tools/regression/bin/ into the new layout (which include tests for
sh(1) and other tools as well). However, doing so is tricky due to the
ongoing work in sh(1) and, especially, the many changes to its tests
since stable/10 was first branched.

Merging everything is the simplest way to achieve this goal and, as a
bonus point, we get various fixes and miscellaneous improvements into
the branch.

Per jilles' suggestion, I'm avoiding the merge of a couple of changes
(r256850 and r257506) that required depending kernel changes. I'm also
avoiding very recent changes that have not had a long enough time to be
validated in current.

This is "make tinderbox" clean.

r256735 sh: Remove one syscall when waiting for a foreground job.
r257399 sh: Allow trapping SIGINT/SIGQUIT after ignore because of '&'.
r257504 sh: Reorder union node to reduce its size on 64-bit platforms.
r257920 sh: Add a test case for would-be assignments that are not due to quoting.
r257929 sh: Properly quote alias output from command -v.
r258489 sh: Add tests for the </dev/null implicit in a background command.
r258533 sh: Add more tests for the </dev/null implicit in a background command.
r258535 sh: Make <&0 disable the </dev/null implicit in a background command.
r258776 sh: Prefer memcpy() to strcpy() in most cases. Remove the scopy macro.
r259047 sh: Split set -x output into a separate function.
r259210 Migrate tools/regression/bin/ tests to the new layout.
r259844 sh: Remove an unused variable.
r259846 sh: Initialize OPTIND=1 even if it came from the environment.
r259874 sh: Simplify code related to PPID variable.
r259946 sh: Don't check input for non-whitespace if history is disabled.
r260246 sh(1): Discourage use of -e.
r260506 Run the sh(1) and test(1) tests as unprivileged.
r260586 Mark the bin/pax tests as requiring perl.
r260634 Use TAP_TESTS_PERL to register the legacy_test in bin/pax.
r260635 Replace hand-crafted Kyuafiles with automatic generation.
r260654 sh: Remove SIGWINCH handler and just check for resize before every read.
r261121 sh: Add test for nested alias.
r261125 sh: Solve the alias recursion problem in a less hackish way.
r261141 sh: Do not depend on parse/execute split in new alias test.
r261160 sh: Add tests for alias names after another alias.
r261192 sh: Allow aliases to force alias substitution on the following word.
r262533 sh: Make expari() static.
r262565 sh: Do not corrupt internal representation if LINENO inner expansion fails.
r262697 sh: Simplify expari().

Reviewed by: jilles

MFC r287148: sh: Fix out of bounds read when there is no ] after a [:class:].

The initial check for a matching ] was incorrect if a ] may be consumed by a
[:class:]. The subsequent loop assumed that there must be a ].

Remove the initial check and make the loop cope with a missing ].

Found with afl-fuzz.

MFC r278826,279330: sh: Various cleanups to expand.c:

* Remove some gotos.
* Remove unused parameter.
* Remove duplicate code.

This MFC is to avoid conflicts in the MFC of r287148.

MFC r272575: sh: Eliminate some gotos.

This MFC is to avoid conflicts in the MFC of r287148.

MFC r276037: sh: Remove EXP_REDIR.

EXP_REDIR was supposed to generate pathnames in redirection if exactly one
file matches, as permitted but not required by POSIX in interactive mode. It
is unlikely this will be implemented.

No functional change is intended.

MFC r274854: sh: Use DQSYNTAX only while expanding, not SQSYNTAX.

Quoting during expansion only cares about CCTL, which is the same for
DQSYNTAX and SQSYNTAX.

MFC r273920: sh: Fix corruption of CTL* bytes in positional parameters in
redirection.

EXP_REDIR was not being checked for while expanding positional parameters in
redirection, so CTL* bytes were not being prefixed where they should be.

MFC r268576: sh: Correctly handle positional parameters beyond INT_MAX on
64-bit systems.

Currently, there can be no more than INT_MAX positional parameters. Make
sure to treat all higher ones as unset to avoid incorrect results and
crashes.

On 64-bit systems, our atoi() takes the low 32 bits of the strtol() and
sign-extends them.

On 32-bit systems, the call to atoi() returned INT_MAX for too high values
and there is not enough address space for so many positional parameters, so
there was no issue.

PR: 195918

MFC r263777: sh: Fix possible memory leaks and double frees with unexpected
SIGINT.

MFC r263195: sh: Add some consts.

MFC r262755: sh: Make argstr() return where it stopped and simplify expari()
using this.

Sync sh(1) in stable/10 to head.

This is a MFC of all the commits listed below.

My original goal of this change was to only merge the move of the tests
from tools/regression/bin/ into the new layout (which include tests for
sh(1) and other tools as well). However, doing so is tricky due to the
ongoing work in sh(1) and, especially, the many changes to its tests
since stable/10 was first branched.

Merging everything is the simplest way to achieve this goal and, as a
bonus point, we get various fixes and miscellaneous improvements into
the branch.

Per jilles' suggestion, I'm avoiding the merge of a couple of changes
(r256850 and r257506) that required depending kernel changes. I'm also
avoiding very recent changes that have not had a long enough time to be
validated in current.

This is "make tinderbox" clean.

r256735 sh: Remove one syscall when waiting for a foreground job.
r257399 sh: Allow trapping SIGINT/SIGQUIT after ignore because of '&'.
r257504 sh: Reorder union node to reduce its size on 64-bit platforms.
r257920 sh: Add a test case for would-be assignments that are not due to quoting.
r257929 sh: Properly quote alias output from command -v.
r258489 sh: Add tests for the </dev/null implicit in a background command.
r258533 sh: Add more tests for the </dev/null implicit in a background command.
r258535 sh: Make <&0 disable the </dev/null implicit in a background command.
r258776 sh: Prefer memcpy() to strcpy() in most cases. Remove the scopy macro.
r259047 sh: Split set -x output into a separate function.
r259210 Migrate tools/regression/bin/ tests to the new layout.
r259844 sh: Remove an unused variable.
r259846 sh: Initialize OPTIND=1 even if it came from the environment.
r259874 sh: Simplify code related to PPID variable.
r259946 sh: Don't check input for non-whitespace if history is disabled.
r260246 sh(1): Discourage use of -e.
r260506 Run the sh(1) and test(1) tests as unprivileged.
r260586 Mark the bin/pax tests as requiring perl.
r260634 Use TAP_TESTS_PERL to register the legacy_test in bin/pax.
r260635 Replace hand-crafted Kyuafiles with automatic generation.
r260654 sh: Remove SIGWINCH handler and just check for resize before every read.
r261121 sh: Add test for nested alias.
r261125 sh: Solve the alias recursion problem in a less hackish way.
r261141 sh: Do not depend on parse/execute split in new alias test.
r261160 sh: Add tests for alias names after another alias.
r261192 sh: Allow aliases to force alias substitution on the following word.
r262533 sh: Make expari() static.
r262565 sh: Do not corrupt internal representation if LINENO inner expansion fails.
r262697 sh: Simplify expari().

Reviewed by: jilles