MFC r315005: sh: Fix executing wrong command with ${x#$(y)}$(z).

The parsed internal representation of words consists of a byte string with a
list of nodes (commands in command substitution). Each unescaped CTLBACKQ or
CTLBACKQ | CTLQUOTE byte corresponds to an entry in the list.

If param in ${param#%##%%word} is not set, the word is not expanded (in a
deviation of POSIX shared with other ash variants and ksh93). Erroneously,
the pointer in the list of commands (argbackq) was not advanced. This caused
the wrong command to be executed later if the outer word contained another
command substitution.

Example:
echo "${unsetvar#$(echo a)}$(echo b)"
wrote "a" but should write "b".

MFC r314686: sh: Fix crash if a -T trap is taken during command substitution.

Code like t=$(stat -f %m "$file") segfaulted if -T was active and a trap
was taken while the shell was waiting for the child process to finish.

What happened was that the dotrap() call in waitforjob() was hit. This
re-entered command execution (including expand.c) at a point not expected by
expbackq(), and global state (unallocated stack string and argbackq) was
corrupted.

To fix this, change expbackq() to prepare for command execution to be
re-entered.

In stable/10, there is more global state that needs to be restored than in
stable/11 and head.

Reported by: bdrewery

MFC r314637: sh: Add some already working tests that exercise new code paths

MFC r269902,r270101:

r269902:

Convert bin/sh/tests to ATF

The new code uses a "test discovery mechanism" to determine
what tests are available for execution

The test shell can be specified via:

kyua test -v test_suites.FreeBSD.bin.sh.test_shell=/path/to/test/sh

Sponsored by: EMC / Isilon Storage Division
Approved by: jmmv (mentor)
Reviewed by: jilles (maintainer)

r270101 (by jilles):

sh: Don't hardcode relative paths in the tests stderr files.

These paths have had to be adjusted to changes in the testsuite runner
several times, so modify the tests to remove the need for such adjustment.

A cp in functional_test.sh is now unneeded, but this matters little in
performance.

MFC r268436,273243,278806,280920,286971: sh: Add various tests.

These new tests from head pass with unmodified stable/10 sh.

MFC r273920: sh: Fix corruption of CTL* bytes in positional parameters in
redirection.

EXP_REDIR was not being checked for while expanding positional parameters in
redirection, so CTL* bytes were not being prefixed where they should be.

Sync sh(1) in stable/10 to head.

This is a MFC of all the commits listed below.

My original goal of this change was to only merge the move of the tests
from tools/regression/bin/ into the new layout (which include tests for
sh(1) and other tools as well). However, doing so is tricky due to the
ongoing work in sh(1) and, especially, the many changes to its tests
since stable/10 was first branched.

Merging everything is the simplest way to achieve this goal and, as a
bonus point, we get various fixes and miscellaneous improvements into
the branch.

Per jilles' suggestion, I'm avoiding the merge of a couple of changes
(r256850 and r257506) that required depending kernel changes. I'm also
avoiding very recent changes that have not had a long enough time to be
validated in current.

This is "make tinderbox" clean.

r256735 sh: Remove one syscall when waiting for a foreground job.
r257399 sh: Allow trapping SIGINT/SIGQUIT after ignore because of '&'.
r257504 sh: Reorder union node to reduce its size on 64-bit platforms.
r257920 sh: Add a test case for would-be assignments that are not due to quoting.
r257929 sh: Properly quote alias output from command -v.
r258489 sh: Add tests for the </dev/null implicit in a background command.
r258533 sh: Add more tests for the </dev/null implicit in a background command.
r258535 sh: Make <&0 disable the </dev/null implicit in a background command.
r258776 sh: Prefer memcpy() to strcpy() in most cases. Remove the scopy macro.
r259047 sh: Split set -x output into a separate function.
r259210 Migrate tools/regression/bin/ tests to the new layout.
r259844 sh: Remove an unused variable.
r259846 sh: Initialize OPTIND=1 even if it came from the environment.
r259874 sh: Simplify code related to PPID variable.
r259946 sh: Don't check input for non-whitespace if history is disabled.
r260246 sh(1): Discourage use of -e.
r260506 Run the sh(1) and test(1) tests as unprivileged.
r260586 Mark the bin/pax tests as requiring perl.
r260634 Use TAP_TESTS_PERL to register the legacy_test in bin/pax.
r260635 Replace hand-crafted Kyuafiles with automatic generation.
r260654 sh: Remove SIGWINCH handler and just check for resize before every read.
r261121 sh: Add test for nested alias.
r261125 sh: Solve the alias recursion problem in a less hackish way.
r261141 sh: Do not depend on parse/execute split in new alias test.
r261160 sh: Add tests for alias names after another alias.
r261192 sh: Allow aliases to force alias substitution on the following word.
r262533 sh: Make expari() static.
r262565 sh: Do not corrupt internal representation if LINENO inner expansion fails.
r262697 sh: Simplify expari().

Reviewed by: jilles

Migrate tools/regression/bin/ tests to the new layout.

This change is a proof of concept on how to easily integrate existing
tests from the tools/regression/ hierarchy into the /usr/tests/ test
suite and on how to adapt them to the new layout for src.

To achieve these goals, this change:

- Moves tests from tools/regression/bin/<tool>/ to bin/<tool>/tests/.
- Renames the previous regress.sh files to legacy_test.sh.
- Adds Makefiles to build and install the tests and all their supporting
data files into /usr/tests/bin/.
- Plugs the legacy_test test programs into the test suite using the new
TAP backend for Kyua (appearing in 0.8) so that the code of the test
programs does not have to change.
- Registers the new directories in the BSD.test.dist mtree file.

Reviewed by: freebsd-testing
Approved by: rpaulo (mentor)

sh: Reject ++ and -- in arithmetic.

POSIX does not require ++ and -- in arithmetic. It is probably more useful
to reject them than to treat ++x and --x as x silently.

Note that the behaviour of increment and decrement can be obtained via
(x+=1), ((x+=1)-1), (x-=1) and ((x-=1)+1).

PR: bin/176444

sh: Pass $? to command substitution containing compound/multiple commands.

Example:
false; echo $(echo $?; :)

sh: Add testcase that $? is preserved into a simple command substitution.

The test builtins/trap6.0 already uses this but having it separate eases
diagnosis if this would break.

sh: Don't lose $? when backquoted command ends with semicolon or newline.

An empty simple command was added and overwrote the exit status with 0.

This affects `...` but not $(...).

Example:
v=`false;`; echo $?

sh: Expand assignment-like words specially for export/readonly/local.

Examples:
export x=~
now expands the tilde
local y=$1
is now safe, even if $1 contains IFS characters or metacharacters.

For a word to "look like an assignment", it must start with a name followed
by an equals sign, none of which may be quoted.

The special treatment applies when the first word (potentially after
"command") is "export", "readonly" or "local". There may be quoting
characters but no expansions. If "local" is overridden with a function there
is no special treatment ("export" and "readonly" cannot be overridden with a
function).

If things like
local arr=(1 2 3)
are ever allowed in the future, they cannot call a "local" function. This
would either be a run-time error or it would call the builtin.

This matches Austin Group bug #351, planned for the next issue of POSIX.1.

PR: bin/166771

sh: Add tests where "export" does not parse differently.

It is planned to expand variable assignments as assignments (no word
splitting, different tilde expansion) when they follow a "declaration
utility" (export, readonly or local). However, a quoted character cannot be
part of a "name" so things like \v=~ are not assignments, and the existing
behaviour applies.

sh: Add a test for variables with underscores in arithmetic.

Things like $((_x+1)) are broken in stable/8 sh but work in stable/9 and
head.

sh: Add testcases that should not be broken by future optimizations.

sh: Reduce unnecessary forks with eval.

The eval special builtin now runs the code with EV_EXIT if it was run
with EV_EXIT itself.

In particular, this eliminates one fork when a command substitution contains
an eval command that ends with an external program or a subshell.

This is similar to what r220978 did for functions.

sh: Fix $? in heredocs on simple commands.

PR: bin/41410

sh: Add already working testcases for $? in here-document.

If the here-document is attached to a compound command or subshell, $?
already works properly. This is both a workaround for bin/41410 and a
requirement for a true fix for bin/41410.

PR: bin/41410
MFC after: 1 week

sh: Fix unquoted $@/$* if IFS=''.

If IFS is null, unquoted $@/$* should still expand to separate words.
This differs from quoted $@ (which does not depend on IFS) in that pathname
generation is performed and empty words are removed.

sh: Add UTF-8 support to pattern matching.

?, [...] patterns match codepoints instead of bytes. They do not match
invalid sequences. [...] patterns must not contain invalid sequences
otherwise they will not match anything. This is so that ${var#?} removes the
first codepoint, not the first byte, without putting UTF-8 knowledge into
the ${var#pattern} code. However, * continues to match any string and an
invalid sequence matches an identical invalid sequence. (This differs from
fnmatch(3).)

sh: Add UTF-8 support to ${#var}.

If the current locale uses UTF-8, ${#var} counts codepoints (more precisely,
bytes b with (b & 0xc0) != 0x80).

sh: Apply set -u to variables in arithmetic.

Note that this only applies to variables that are actually used.
Things like (0 && unsetvar) do not cause an error.

Exp-run done by: pav (with some other sh(1) changes)

sh: Do not word split "${#parameter}".

This is only a problem if IFS contains digits, which is unusual but valid.

Because of an incorrect fix for PR bin/12137, "${#parameter}" was treated
as ${#parameter}. The underlying problem was that "${#parameter}"
erroneously added CTLESC bytes before determining the length. This
was properly fixed for PR bin/56147 but the incorrect fix was not backed
out.

Reported by: Seeker on forums.freebsd.org
MFC after: 2 weeks

sh: Add test for bin/12137.

sh: Add test for obscure and ambiguous ${#?}.

sh: Fix some parameter expansion variants ${#...}.

These already worked: $# ${#} ${##} ${#-} ${#?}
These now work as well: ${#+word} ${#-word} ${##word} ${#%word}

There is an ambiguity in the standard with ${#?}: it could be the length of
$? or it could be $# giving an error in the (impossible) case that it is not
set. We continue to use the former interpretation as it seems more useful.

sh: Add some tests for ${#parameter}.

sh: Unset some more locale vars in two tests that may cause them to break.

sh: Detect dividing the smallest integer by -1.

This overflows and on some architectures such as amd64 it generates SIGFPE.
Generate an error on all architectures.

sh: Add tests for new features in arithmetic.

sh: Avoid side effects from builtins in optimized command substitution.

Change the criterion for builtins to be safe to execute in the same process
in optimized command substitution from a blacklist of only cd, . and eval to
a whitelist.

This avoids clobbering the main shell environment such as by $(exit 4) and
$(set -x).

The builtins jobid, jobs, times and trap can still show information not
available in a child process; this is deliberately permitted. (Changing
traps is not.)

For some builtins, whether they are safe depends on the arguments passed to
them. Some of these are always considered unsafe to keep things simple; this
only harms efficiency a little in the rare case they are used alone in a
command substitution.

sh: Add two tests for special cases in command substitution that already
work in stable/8.

sh: Don't do optimized command substitution if expansions have side effects.

Before considering to execute a command substitution in the same process,
check if any of the expansions may have a side effect; if so, execute it in
a new process just like happens if it is not a single simple command.

Although the check happens at run time, it is a static check that does not
depend on current state. It is triggered by:
- expanding $! (which may cause the job to be remembered)
- ${var=value} default value assignment
- assignment operators in arithmetic
- parameter substitutions in arithmetic except ${#param}, $$, $# and $?
- command substitutions in arithmetic

This means that $((v+1)) does not prevent optimized command substitution,
whereas $(($v+1)) does, because $v might expand to something containing
assignment operators.

Scripts should not depend on these exact details for correctness. It is also
imaginable to have the shell fork if and when a side effect is encountered
or to create a new temporary namespace for variables.

Due to the $! change, the construct $(jobs $!) no longer works. The value of
$! should be stored in a variable outside command substitution first.

sh: Add test for optimized command substitution.

This test verifies that certain expansions without side effects do not
cause the command substitution to be executed in a child process.

This is not a correctness requirement, but it involves a nontrivial amount
of code and it would be unfortunate if it stopped working.

sh: Make expansion errors in optimized command substitution non-fatal.
Command substitutions consisting of a single simple command are executed in
the main shell process but this should be invisible apart from performance
and very few exceptions such as $(trap).

sh: Add a testcase for cmdsubst errors that already works properly.
If a command substitution consists of one special builtin and there is a
redirection error, this should not abort the outer shell.
It was fixed in r201366 by ignoring special builtin properties for command
substitutions consisting of one builtin.

Restore two commented-out tests from plus-minus1.0 to a new file.

These two cases pass on -CURRENT but fail on stable/8.

Reviewed by: jilles

Remove commented-out test that's covered in plus-minus2.0 anyway.

Discussed with: jilles

sh: arith: Disallow decimal constants starting with 0 (containing 8 or 9).

Constants in arithmetic starting with 0 should be octal only.

This avoids the following highly puzzling result:
$ echo $((018-017))
3
by making it an error instead.

sh: Fix corruption of command substitutions with special chars after newline

The CTLESC byte to protect a special character was output before instead of
after a newline directly preceding the special character.

The special handling of newlines is because command substitutions discard
all trailing newlines.

sh: Add a test for r216387 (long arithmetic expression in here document).

sh: Add a test that manipulates various long strings.

It is quite effective at detecting mistakes in memalloc.c and code using it.

It is somewhat slow, but some of the patches in my queue improve it.

sh: Fix some issues with CTL* bytes and ${var#pat}.

subevalvar() incorrectly assumed that CTLESC bytes were present iff the
expansion was quoted. However, they are present iff various processing such
as word splitting is to be done later on.

Example:
v=@$e@$e@$e@
y="${v##*"$e"}"
echo "$y"
failed if $e contained the magic CTLESC byte.

Exp-run done by: pav (with some other sh(1) changes)

sh: Do IFS splitting on word in ${v+word} and ${v-word}.

The code is inspired by NetBSD sh somewhat, but different because we
preserve the old Almquist/Bourne/Korn ability to have an unquoted part in a
quoted ${v+word}. For example, "${v-"*"}" expands to $v as a single field if
v is set, but generates filenames otherwise.

Note that this is the only place where we split text literally from the
script (the similar ${v=word} assigns to v and then expands $v). The parser
must now add additional markers to allow the expansion code to know whether
arbitrary characters in substitutions are quoted.

Example:
for i in ${$+a b c}; do echo $i; done

Exp-run done by: pav (with some other sh(1) changes)

sh: Only accept a '}' inside ${v+-=?...} if double-quote state matches.
If double-quote state does not match, treat the '}' literally.

This ensures double-quote state remains the same before and after a
${v+-=?...} which helps with expand.c.

It makes things like
${foo+"\${bar}"}
which I have seen in the wild work as expected.

Exp-run done by: pav (with some other sh(1) changes)

sh: Make double-quotes quote a '}' inside ${v#...} and ${v%...}.

Exp-run done by: pav (with some other sh(1) changes)
PR: bin/57554

Do not assume in growstackstr() that a "precious" character will be
immediately written into the stack after the call. Instead let the caller
manage the "space left".

Previously, growstackstr()'s assumption causes problems with STACKSTRNUL()
where we want to be able to turn a stack into a C string, and later
pretend the NUL is not there.

This fixes a bug in STACKSTRNUL() (that grew the stack) where:
1. STADJUST() called after a STACKSTRNUL() results in an improper adjust.
This can be seen in ${var%pattern} and ${var%%pattern} evaluation.
2. Memory leak in STPUTC() called after a STACKSTRNUL().

Reviewed by: jilles

Allow one to regression test 'sh' changes without having to install
a potentially bad /bin/sh first.

sh: Remove remnants of '!!' to negate pattern.

This Almquist extension was disabled long ago.

In pathname generation, components starting with '!!' were treated as
containing wildcards, causing unnecessary readdir (which could fail, causing
pathname generation to fail while it should not).

sh: Test that all bytes from 128 to 255 can be used in IFS.

To avoid multibyte issues, this test forces ISO8859-1 charset.

This also passes on stable/8.

sh: Test that all bytes from 1 to 127 can be used in IFS.

This also passes on stable/8.

sh: Fix heap-based buffer overflow in pathname generation.

The buffer for generated pathnames could be too small in some cases. It
happened to be always at least PATH_MAX long, so there was never an overflow
if the resulting pathnames would be usable.

This bug may be abused if a script subjects input from an untrusted source
to pathname generation, which a bad idea anyhow. Most shell scripts do not
work on untrusted data. secteam@ says no advisory is necessary.

PR: bin/148733
Reported by: Changming Sun snnn119 at gmail com
MFC after: 10 days

sh: Add more testcases for ${var:-word}.

Whether POSIX requires these is unclear.

They pass with 8-stable sh as well.

sh: Remove comment that the comma operator is missing in arithmetic
expansion.

The comma operator is not listed in POSIX.1-2008 XCU 1.1.2.1 Arithmetic
Precision and Operations (referenced by XCU 2.6.4 Arithmetic Expansion) and
is therefore not required.

sh: Fix pathname expansion with quoted slashes like *\/.

These are git commits 36f0fa8fcbc8c7b2b194addd29100fb40e73e4e9 and
d6d06ff5c2ea0fa44becc5ef4340e5f2f15073e4 in dash.

Because this is the first code I'm importing from dash to expand.c, add the
Herbert Xu copyright notice which is in dash's expand.c.

When pathname expanding *\/, the CTLESC representing the quoted state was
erroneously taken as part of the * pathname component. This CTLESC was then
seen by the pattern matching code as escaping the '\0' terminating the
string.

The code is slightly different because dash converts the CTLESC characters
to backslashes and removes all the other CTL* characters to allow
substituting glob(3).

The effect of the bug was also slightly different from dash (where nothing
matched at all). Because a CTLESC can escape a '\0' in some way, whether
files were included despite the bug depended on memory that should not be
read. In particular, on many machines /*\/ expanded to a strict subset of
what /*/ expanded to.

Example:
echo /*"/null"

This should print /dev/null, not /*/null.

PR: bin/146378
Obtained from: dash

sh: Add some simple testcases for pathname expansion.

sh: Add some more tests for ${v#...} and ${v%...}.

These pass on stable/8 as well.

sh: Add testcases for double-quotes within quoted ${var+-...} (non-POSIX).

POSIX leaves things like "${var+"word"}" undefined.
We follow traditional ash behaviour here.
Hence, these testcases also work on stable/8.

sh: Add test for nested arithmetic substitution.

Pre-r206145 sh changes nested arithmetic substitution to parentheses, which
is not always correct, as exercised by this test.

sh: Add test for some associativity in arithmetic.

sh: Do tilde expansion in substitutions.

This applies to word in ${v-word}, ${v+word}, ${v=word}, ${v?word} (which
inherits quoting from the outside) and in ${v%word}, ${v%%word}, ${v#word},
${v##word} (which does not inherit any quoting).

In all cases tilde expansion is only attempted at the start of word, even if
word contains spaces. This agrees with POSIX and other shells.

This is the last part of the patch tested in the exp-run.

Exp-run done by: erwin (with some other sh(1) changes)

sh: Fix duplicate variable name in test.

sh: Add some testcases for the working parts of tilde expansion.

sh: Allow quoting pattern match characters in ${v%pat} and ${v#pat}.

Note that this depends on r206145 for allowing pattern match characters to
have their special meaning inside a double-quoted expansion like "${v%pat}".

PR: bin/117748
Exp-run done by: erwin (with some other sh(1) changes)

sh: Fix various things about expansions:
* remove the backslash from \} inside double quotes inside +-=?
substitutions, e.g. "${$+\}a}"
* maintain separate double-quote state for ${v#...} and ${v%...};
single and double quotes are special inside, even in a double-quoted
string or here document
* keep track of correct order of substitutions and arithmetic

This is different from dash's approach, which does not track individual
double quotes in the parser, trying to fix this up during expansion.
This treats single quotes inside "${v#...}" incorrectly, however.

This is similar to NetBSD's approach (as submitted in PR bin/57554), but
recognizes the difference between +-=? and #% substitutions hinted at in
POSIX and is more refined for arithmetic expansion and here documents.

PR: bin/57554
Exp-run done by: erwin (with some other sh(1) changes)

sh: Add some testcases for the working parts of ${v%...} and ${v#...}.

sh: Add tests for " and $ inside `.

sh: Add some testcases for ${v=w}, ${v-w}, ${v+w}.

These expansions, which were already in the Bourne shell, work correctly for
the most part. The testcases are only about the parts that already work
correctly.

sh: arith: Add a test for a bug in the dash arith code,
which I plan to import at some point.
Our current code handles it fine and it should stay that way.

sh: arith: Test an octal constant as well.

sh: Add a regression test that tries out all arithmetic ops.

MFC after: 1 week

sh: Fix some bugs with backquoted builtins:
- correctly handle error output in $(builtin 2>&1), clarify out1/out2 vs
output/errout in the code
- treat all builtins as regular builtins so errors do not abort the shell
and variable assignments do not persist
- respect the caller's INTOFF

Some bugs still exist:
- expansion errors may still abort the shell
- some side effects of expansions and builtins persist

sh: arith: Return only 0 and 1 from && and ||.

This agrees with C, POSIX and other shells.

sh: Exempt $@ and $* from set -u

This seems more useful and will likely be in the next POSIX standard.

Also document more precisely in the man page what set -u does (note that
$@, $* and $! are the only special parameters that can ever be unset, all
the others are always set, although they may be empty).

Add some tests for ${var?} and set -u.

Add some tests for r194975 and r194977.

Approved by: ed (mentor) (implicit)