bin: Automated cleanup of cdefs and other formatting

Apply the following automated changes to try to eliminate
no-longer-needed sys/cdefs.h includes as well as now-empty
blank lines in a row.

Remove /^#if.*\n#endif.*\n#include\s+<sys/cdefs.h>.*\n/
Remove /\n+#include\s+<sys/cdefs.h>.*\n+#if.*\n#endif.*\n+/
Remove /\n+#if.*\n#endif.*\n+/
Remove /^#if.*\n#endif.*\n/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/types.h>/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/param.h>/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/capsicum.h>/

Sponsored by: Netflix

bin: Remove ancient SCCS tags.

Remove ancient SCCS tags from the tree, automated scripting, with two
minor fixup to keep things compiling. All the common forms in the tree
were removed with a perl script.

Sponsored by: Netflix

sh: implement PS1 \D to print current time

\D{format} yields the result of calling strftime(3) with the provided
format and the current time.

When PS4 can use this, it will enable us to easily generate timestamps
when tracing script execution.

Differential Revision: https://reviews.freebsd.org/D35840

sh: reindent a for loop in parser.c

Reduce indentation level before a commit that will add new code here.

Reviewed by: jilles
Differential Revision: https://reviews.freebsd.org/D37926

Remove $FreeBSD$: one-line .c pattern

Remove /^[\s*]*__FBSDID\("\$FreeBSD\$"\);?\s*\n/

sh(1): Allow non-printing characters in prompt strings

Introduce new prompt format characters:

- '\[' starts the sequence of non-printing chatacters
- '\]' ends the sequence of non-printing characters

Within these sequences, the following characters are now supported:

- '\a' emits ASCII BEL (0x07, 007) character
- '\e' emits ASCII ESC (0x1b, 033) character
- '\r' emits ASCII CR (0x0d, 015) character
- '\n' emits ASCII CRLF sequence

These can be used to embed ANSI sequences into prompt strings.

Example in .shrc:

PS1="\[\e[7m\]\u@\h\[\e[0m\]:\w \\$ "

This tries to maintain some degree of compatibility with GNU bash,
that uses GNU readline library (which behaves slightly different from
BSD editline): It has two "non-printing boundary" characters:

- RL_PROMPT_START_IGNORE (\001)
- RL_PROMPT_END_IGNORE (\002)

while BSD editline only has one (when using EL_PROMPT_ESC setting), so
for this purpose, ASCII \001 was chosen and both \[ and \] emits
this character.

And while here, enlarge PROMPTLEN from 128 to 192 characters.

Reviewed by: jilles
Approved by: jilles
Differential Revision: https://reviews.freebsd.org/D37701

sh: Fix heredoc at certain places in case and for

After an unescaped newline, there may be a here-document. Some places in
case and for did not check for one.

Reviewed by: bdrewery
Differential Revision: https://reviews.freebsd.org/D32628

Make sh(1) support \u in PS1. This removes one fork/exec on interactive
shell startup.

Reviewed by: 0mp (man page), jilles
MFC after: 2 weeks
Sponsored by: DARPA, AFRL
Differential Revision: https://reviews.freebsd.org/D18790

Make sh(1) collapse $HOME into "~" in PS1.

Reviewed by: jilles
MFC after: 2 weeks
Sponsored by: DARPA, AFRL
Differential Revision: https://reviews.freebsd.org/D18663

sh: Split CNL syntax category to avoid a check on state[level].syntax

No functional change is intended.

sh: Allow unquoted newlines in word in ${param+word} etc.

POSIX requires accepting unquoted newlines in word in parameter expansions
like ${param+word}, ${param#word}, although the Bourne shell did not support
it, it is not commonly used and might make it harder to find a missing
closing brace.

It was also strange that something like

foo="${bar#
}"

was rejected.

Reported by: Martijn Dekker via Robert Elz

General further adoption of SPDX licensing ID tags.

Mainly focus on files that use BSD 3-Clause license.

The Software Package Data Exchange (SPDX) group provides a specification
to make it easier for automated tools to detect and summarize well known
opensource licenses. We are gradually adopting the specification, noting
that the tags are considered only advisory and do not, in any way,
superceed or replace the license texts.

Special thanks to Wind River for providing access to "The Duke of
Highlander" tool: an older (2014) run over FreeBSD tree was useful as a
starting point.

Renumber copyright clause 4

Renumber cluase 4 to 3, per what everybody else did when BSD granted
them permission to remove clause 3. My insistance on keeping the same
numbering for legal reasons is too pedantic, so give up on that point.

Submitted by: Jan Schaumann <jschauma@stevens.edu>
Pull Request: https://github.com/freebsd/freebsd/pull/96

The (i < PROMPTLEN - 1) test added by r300442 in the code for the default
case of \c in the prompt format string is a no-op. We already passed
this test at the top of the loop, and i has not yet been incremented in
this path. Change this test to (i < PROMPTLEN - 2).

Reported by: Coverity
CID: 1008328
Reviewed by: cem
MFC after: 1 week

Hopefully fix Coverity CID 1008328 (Out-of-bounds write) in /bin/sh.

Replace the magic constant 127 in the loop interation count with
"PROMPTLEN - 1".

gethostname() is not guaranteed to NUL terminate the destination
string if it is too short. Decrease the length passed to gethostname()
by one, and add a NUL at the end of the buffer to make sure the
following loop to find the end of the name properly terminates.

The default: case is the likely cause of Coverity CID 1008328. If
i is 126 at the top of the loop interation where the default case
is triggered, i will be incremented to 127 by the default case,
then incremented to 128 at the top of the loop before being compared
to 127 (PROMPTLENT - 1) and terminating the loop. Then the NUL
termination code after the loop will write to ps[128]. Fix by
checking for overflow before incrementing the index and storing the
second character in the buffer.

These fixes are not guaranteed to satisfy Coverity. The code that
increments i in the 'h'/'H' and 'w'/'W' cases may be beyond its
capability to analyze, but the code appears to be safe.

Reported by: Coverity
CID: 1008328
Reviewed by: jilles, cem
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D6482

sh: Handle empty hostname and $PWD when building prompt

If the hostname is empty and \h is used in $PS1,
the remainder of the prompt following \h will be empty.
Likewise for $PWD and \w. Fix it.

Reviewed by: jilles
MFC after: 1 week
Sponsored by: Dell Inc.
Differential Revision: https://reviews.freebsd.org/D6188

sh: Write LINENO value to stack string directly.

sh: Simplify code by removing variable bracketed_name.

sh(1): replace 0 with NULL for pointers.

Found with devel/coccinelle.

Reviewed by: jilles

sh: Remove a redundant STPUTC check.

sh: Optimize setprompt(0).

Avoid doing work to print an empty prompt (such as when reading scripts).

wordexp: Rewrite to make WRDE_NOCMD reliable.

Shell syntax is too complicated to detect command substitution and unquoted
operators reliably without implementing much of sh's parser. Therefore, have
sh do this detection.

While changing sh's support anyway, also read input from a pipe instead of
arguments to avoid {ARG_MAX} limits and improve privacy, and output count
and length using 16 instead of 8 digits.

The basic concept is:
execl("/bin/sh", "sh", "-c", "freebsd_wordexp ${1:+\"$1\"} -f "$2",
"", flags & WRDE_NOCMD ? "-p" : "", <pipe with words>);

The WRDE_BADCHAR error is still implemented in libc. POSIX requires us to
fail strings containing unquoted braces with code WRDE_BADCHAR. Since this
is normally not a syntax error in sh, there is still a need for checking
code in libc, we_check().

The new we_check() is an optimistic check that all the characters
<newline> | & ; < > ( ) { }
are quoted. To avoid duplicating too much sh logic, such characters are
permitted when quoting characters are seen, even if the quoting characters
may themselves be quoted. This code reports all WRDE_BADCHAR errors; bad
characters that get past it and are a syntax error in sh return WRDE_SYNTAX.

Although many implementations of WRDE_NOCMD erroneously allow some command
substitutions (and ours even documented this), there appears to be code that
relies on its security (codesearch.debian.net shows quite a few uses).
Passing untrusted data to wordexp() still exposes a denial of service
possibility and a fairly large attack surface.

Reviewed by: wblock (man page only)
MFC after: 2 weeks
Relnotes: yes
Security: fixes command execution with wordexp(untrusted, WRDE_NOCMD)

sh: Allow empty << EOF markers.

sh: Don't create bad parse result when postponing a bad substitution error.

An invalid substitution like ${var@} does not cause a parse error but is
stored in the intermediate representation, to be written as part of the
error message. If there is a CTL* byte in the stored part, this confuses
some code such as the code to skip an unused alternative such as in
${var-alternative}.

To keep things simple, do not store CTL* bytes.

Found with afl-fuzz.

MFC after: 1 week

sh: Avoid negative character values from $'\Uffffffff' etc.

The negative value was not expected and generated the low 8 bits as a byte,
which may be an invalid character encoding.

The final shift in creating the negative value was undefined as well.

Make the temporary variable unsigned to fix this.

sh: Prefer "" to nullstr where possible.

sh: Prepend "$0: " to error messages if there is no command name.

sh: Allow backslash-newline continuation in more places:

* directly after a $
* directly after ${
* between the characters of a multi-character operator token
* within a parameter name

sh: Make parseredir() a proper function instead of an emulated nested
function.

sh: Remove more gotos.

sh: Fix LINENO and prompt after $'\0 and newline.

sh: Remove arbitrary length limit on << EOF markers.

This also simplifies the code.

sh: Make checkend() a real function instead of an emulated nested function.

No functional change is intended, but the generated code is slightly
different.

sh: Add some const keywords.

sh: Allow aliases to force alias substitution on the following word.

If an alias's value ends with a space or tab, the next word is also
checked for aliases.

This is a POSIX feature. It is useful with utilities like command and
nohup (alias them to themselves followed by a space).

sh: Simplify list() in the parser.

The erflag argument was only used by old-style (``) command substitutions.
We can remove it and handle the special case in the command substitution
code.

sh: Separate out nbinary allocation into a function.

sh: Use makename() where possible.

sh: Add a function for the case where one token is required in the parse.

sh: Cast -1 to pointer rather than pointer to variable of wrong type.

NEOF needs to be a non-null pointer distinct from valid union node pointers.
It is not dereferenced.

The new NEOF is much like SIG_ERR except that it is an object pointer
instead of a function pointer.

The variable tokpushback can now be static.

sh: Disallow empty simple commands.

As per POSIX, a simple command must have at least one redirection,
assignment word or command word.

These occured in rare cases such as eval "f()" .

The extension of allowing no commands inside { }, if, while, for, etc.
remains.

sh: Remove unnecessary reset functions.

These are already handled by exception handlers.

sh: Allow a lone redirection before '|', ';;' or ';&'.

Example: </dev/null | :

PR: 181240
MFC after: 1 week

sh: Remove an incorrect comment.

sh: Remove #define MKINIT.

MKINIT only served for the removed mkinit. Many variables can be static now.

sh: Remove mkinit.

Replace the RESET blocks with regular functions and a reset() function that
calls them all.

This code generation tool is unusual and does not appear to provide much
benefit. I do not think isolating the knowledge about which modules need to
be reset is worth an almost 500-line build tool and wider scope for
variables used by the reset functions.

Also, relying on reset functions is often wrong: the cleanup should be done
in exception handlers so that no stale state remains after 'command eval'
and the like.

sh: Fix various compiler warnings.

It now passes WARNS=7 with clang on i386.

GCC 4.2.1 does not understand setjmp() properly so will always trigger
-Wuninitialized. I will not add the volatile keywords to suppress this.

sh: Fix crash when parsing '{ } &'.

MFC after: 1 week

sh: Don't lose $? when backquoted command ends with semicolon or newline.

An empty simple command was added and overwrote the exit status with 0.

This affects `...` but not $(...).

Example:
v=`false;`; echo $?

sh: Remove special support for background simple commands.

It expands the arguments in the parent shell process, which is incorrect.

sh: Add case statement fallthrough (with ';&' instead of ';;').

Replacing ;; with the new control operator ;& will cause the next list to be
executed as well without checking its pattern, continuing until a list ends
with ;; or until the end of the case statement. This is like omitting
"break" in a C "switch" statement.

The sequence ;& was formerly invalid.

This feature is proposed for the next POSIX issue in Austin Group issue
#449.

sh: Do parameter expansion before printing PS4 (set -x).

The function name expandstr() and the general idea of doing this kind of
expansion by treating the text as a here document without end marker is from
dash.

All variants of parameter expansion and arithmetic expansion also work (the
latter is not required by POSIX but it does not take extra code and many
other shells also allow it).

Command substitution is prevented because I think it causes too much code to
be re-entered (for example creating an unbounded recursion of trace lines).

Unfortunately, our LINENO is somewhat crude, otherwise PS4='$LINENO+ ' would
be quite useful.

sh: Expand aliases after assignments and redirections.

sh: Allow terminating a heredoc with a terminator at EOF without a newline.

This is sometimes used with eval or old-style command substitution, and most
shells other than ash derivatives allow it.

It can also be used with scripts that violate POSIX's requirement on the
application that they end in a newline (scripts must be text files except
that line length is unlimited).

Example:
v=`cat <<EOF
foo
EOF`
echo $v

This commit does not add support for the similar construct with new-style
command substitution, like
v=$(cat <<EOF
foo
EOF)
This continues to require a newline after the terminator.

sh: Add \u/\U support (in $'...') for UTF-8.

Because we have no iconv in base, support for other charsets is not
possible.

Note that \u/\U are processed using the locale that was active when the
shell started. This is necessary to avoid behaviour that depends on the
parse/execute split (for example when placing braces around an entire
script). Therefore, UTF-8 encoding is implemented manually.

sh: Add $'quoting' (C-style escape sequences).

A string between $' and ' may contain backslash escape sequences similar to
the ones in a C string constant (except that a single-quote must be escaped
and a double-quote need not be). Details are in the sh(1) man page.

This construct is useful to include unprintable characters, tabs and
newlines in strings; while this can be done with a command substitution
containing a printf command, that needs ugly workarounds if the result is to
end with a newline as command substitution removes all trailing newlines.

The construct may also be useful in future to describe unprintable
characters without needing to write those characters themselves in 'set -x',
'export -p' and the like.

The implementation attempts to comply to the proposal for the next issue of
the POSIX specification. Because this construct is not in POSIX.1-2008,
using it in scripts intended to be portable is unwise.

Matching the minimal locale support in the rest of sh, the \u and \U
sequences are currently not useful.

Exp-run done by: pav (with some other sh(1) changes)

sh: Detect an error for ${#var<GARBAGE>}.

In particular, this makes things like ${#foo[0]} and ${#foo[@]} errors
rather than silent equivalents of ${#foo}.

PR: bin/151720
Submitted by: Mark Johnston
Exp-run done by: pav (with some other sh(1) changes)

sh: Do not word split "${#parameter}".

This is only a problem if IFS contains digits, which is unusual but valid.

Because of an incorrect fix for PR bin/12137, "${#parameter}" was treated
as ${#parameter}. The underlying problem was that "${#parameter}"
erroneously added CTLESC bytes before determining the length. This
was properly fixed for PR bin/56147 but the incorrect fix was not backed
out.

Reported by: Seeker on forums.freebsd.org
MFC after: 2 weeks

sh: Fix some parameter expansion variants ${#...}.

These already worked: $# ${#} ${##} ${#-} ${#?}
These now work as well: ${#+word} ${#-word} ${##word} ${#%word}

There is an ambiguity in the standard with ${#?}: it could be the length of
$? or it could be $# giving an error in the (impossible) case that it is not
set. We continue to use the former interpretation as it seems more useful.

sh: Fix two things about {(...)} <redir:

* In {(...) <redir1;} <redir2, do not drop redir1.
* Maintain the difference between (...) <redir and {(...)} <redir:
In (...) <redir, the redirection is performed in the child, while in
{(...)} <redir it should be performed in the parent (like {(...); :;}
<redir)

sh: Allow arbitrary large numbers in CHECKSTRSPACE.
Reduce "stack string" API somewhat and simplify code.
Add a check for integer overflow of the "stack string" length (probably
incomplete).

Remove duplicate check, turning dead code into live code.

Coverity CID: 5114
Reviewed by: jilles

sh: Code size optimizations to "stack string" memory allocation:
* Prefer one CHECKSTRSPACE with multiple USTPUTC to multiple STPUTC.
* Add STPUTS macro (based on function) and use it instead of loops that add
nul-terminated strings to the stack string.

No functional change is intended, but code size is about 1K less on i386.

sh: Fix some issues with aliases and case, by importing dash checkkwd code.

This moves the function of the noaliases variable into the checkkwd
variable. This way it is properly reset on errors and aliases can be used
normally in the commands for each case (the case labels recognize the
keyword esac but no aliases).

The new code is clearer as well.

Obtained from: dash

sh: Use iteration instead of recursion to evaluate semicolon lists.
This reduces CPU and memory usage when executing long lists (such
as long functions).

sh: Tweak some string constants to reduce code size.

* Reduce some needless differences.
* Shorten some error messages that should not happen.

sh: Reject function names ending in one of !%*+-=?@}~

These do something else in ksh: name=(...) is an array or compound variable
assignment and the others are extended patterns.

This is the last patch of the ones tested in the exp run.

Exp-run done by: pav (with some other sh(1) changes)

sh: Detect various additional errors in the parser.

Apart from detecting breakage earlier or at all, this also fixes a segfault
in the testsuite. The "handling" of the breakage left an invalid internal
representation in some cases.

Examples:
echo a; do echo b
echo `) echo a`
echo `date; do do do`

Exp-run done by: pav (with some other sh(1) changes)

sh: Error out on various specials/keywords in the wrong place in backticks.

Example:
echo `date)`

Exp-run done by: pav (with some other sh(1) changes)
Obtained from: NetBSD (Christos Zoulas, NetBSD PR 11317)

sh: Do IFS splitting on word in ${v+word} and ${v-word}.

The code is inspired by NetBSD sh somewhat, but different because we
preserve the old Almquist/Bourne/Korn ability to have an unquoted part in a
quoted ${v+word}. For example, "${v-"*"}" expands to $v as a single field if
v is set, but generates filenames otherwise.

Note that this is the only place where we split text literally from the
script (the similar ${v=word} assigns to v and then expands $v). The parser
must now add additional markers to allow the expansion code to know whether
arbitrary characters in substitutions are quoted.

Example:
for i in ${$+a b c}; do echo $i; done

Exp-run done by: pav (with some other sh(1) changes)

sh: Only accept a '}' inside ${v+-=?...} if double-quote state matches.
If double-quote state does not match, treat the '}' literally.

This ensures double-quote state remains the same before and after a
${v+-=?...} which helps with expand.c.

It makes things like
${foo+"\${bar}"}
which I have seen in the wild work as expected.

Exp-run done by: pav (with some other sh(1) changes)

sh: Make double-quotes quote a '}' inside ${v#...} and ${v%...}.

Exp-run done by: pav (with some other sh(1) changes)
PR: bin/57554

sh: Ignore double-quotes in arithmetic rather than treating them as quotes.

This provides similar behaviour, but allows a simpler parser.

This changes r206473.

Exp-run done by: pav (with some other sh(1) changes)

sh: Do not allow overriding a special builtin with a function.
This is a syntax error.

POSIX does not say explicitly whether defining a function with the same name
as a special builtin is allowed, but it does say that it is impossible to
call such a function.

A special builtin can still be overridden with an alias.

This commit is part of a set of changes that will ensure that when
something looks like a special builtin to the parser, it is one. (Not the
other way around, as it remains possible to call a special builtin named
by a variable or other substitution.)

Exp-run done by: pav (with some other sh(1) changes)

sh: Make sure defined functions can actually be called.

Add some conservative checks on function names:
- Disallow expansions or quoting characters; these can only be called via
strange control characters
- Disallow '/'; these functions cannot be called anyway, as exec.c assumes
they are pathnames
- Make the CTL* bytes work properly in function names.

These are syntax errors.

POSIX does not require us to support more than names (letters, digits and
underscores, not starting with a digit), but I do not want to restrict it
that much at this time.

Exp-run done by: pav (with some other sh(1) changes)

sh: Change ! within a pipeline to start a new pipeline instead.

This is how ksh93 treats ! within a pipeline and makes the ! in
a | ! b | c
negate the exit status of the pipeline, as if it were
a | { ! b | c; }

Side effect: something like
f() ! a
is now a syntax error, because a function definition takes a command,
not a pipeline.

Exp-run done by: pav (with some other sh(1) changes)

A new jail(8) with a configuration file, to replace the work currently done
by /etc/rc.d/jail.

In the spirit of r90111, depend on c89 and remove the "STATIC" macro
and its usage.

Consistently use "STATIC" for all functions in order to be able to set
breakpoints with in a debugger. And use naked "static" for variables.

Noticed by: bde

sh: Add __dead2 to two functions that do not return.

Apart from helping static analyzers, this also appears to reduce the size of
the binary slightly.

sh: Fix crash due to uninitialized here-document.

If an ; or & token was followed by an EOF token, pending here-documents were
left uninitialized. Execution would crash, either in the main shell process
for literal here-documents or in a child process for expanded
here-documents. In the latter case the problem is hard to detect apart from
the core dumps and log messages.

Side effect: slightly different retries on inputs where EOF is not
persistent.

Note that tools/regression/bin/sh/parser/heredoc6.0 still causes a similar
crash in a child process. The text passed to eval is malformed and should be
rejected.

sh: Allow a background command consisting solely of redirections.

Example:
</dev/null &

MFC after: 2 weeks

sh: There cannot be a TNOT in simplecmd(), remove checks.

simplecmd() only handles simple commands and function definitions, neither
of which involves the ! keyword. The initial token on entry to simplecmd()
is one of the following: TSEMI, TAND, TOR, TNL, TEOF, TWORD, TRP.

sh: Use $PWD instead of getcwd() for the \w and \W prompt expansions.

This ensures that the logical working directory (which may include
symlinks) is shown and is similar to the default behaviour of the pwd
builtin.

sh: Fix compilation with -DNO_HISTORY.

The LINENO code uses snprintf() and relied on "myhistedit.h" to pull in the
necessary <stdio.h>.

Compiling with -DNO_HISTORY disables all editing and history support and
allows linking without -ledit -ltermcap. This may be useful for embedded
systems.

MFC after: 2 weeks

sh: Fix a crash if a heredoc was not properly ended and parsing continued.

Example (in interactive mode):
cat <<EOF && )
The next command typed caused sh to segfault, because the state for the here
document was not reset.

Like parser_temp, this uses the fact that the parser is not re-entered.

sh: Change interaction of command substitution and here documents.

If a command substitution contains a newline token, this no longer starts
here documents of outer commands. This way, we follow POSIX's idea of the
command substitution being a separate script more closely. It also matches
other shells better and is consistent with newline characters in quotes not
starting here documents.

The extension tested in parser/heredoc3.0 ($(cat <<EOF)\ntext\nEOF\n)
continues to be supported.

In particular, this change allows things like
cat <<EOF && echo `pwd`
(a `` command substitution after a here document)
which formerly silently used an empty file as the here document, because the
EOF of the inner command "pwd" also forced an empty here document.

MFC r199282: sh: Allow a newline before "in" in a for command,
as required by POSIX.

sh: Partially revert r206146, allowing double-quotes in arithmetic.

These do pretty much nothing (except that parentheses are ignored), but
people seem to use them and allowing them does not hurt much.

Single-quotes seem not to be used and cause silently different behaviour
with ksh93 character constants.

sh: Remove special handling for ' and " in arithmetic.
They will be treated like normal characters, resulting in a runtime
arithmetic expression error.

Exp-run done by: erwin (with some other sh(1) changes)

sh: Fix various things about expansions:
* remove the backslash from \} inside double quotes inside +-=?
substitutions, e.g. "${$+\}a}"
* maintain separate double-quote state for ${v#...} and ${v%...};
single and double quotes are special inside, even in a double-quoted
string or here document
* keep track of correct order of substitutions and arithmetic

This is different from dash's approach, which does not track individual
double quotes in the parser, trying to fix this up during expansion.
This treats single quotes inside "${v#...}" incorrectly, however.

This is similar to NetBSD's approach (as submitted in PR bin/57554), but
recognizes the difference between +-=? and #% substitutions hinted at in
POSIX and is more refined for arithmetic expansion and here documents.

PR: bin/57554
Exp-run done by: erwin (with some other sh(1) changes)

sh: Treat unexpected newlines in substitutions as a syntax error.

The old approach was wrong because PS2 was not used and seems unlikely to
parse extensions (ksh93's ${ COMMAND} may well fail to parse).

Exp-run done by: erwin (with some other sh(1) changes)

sh: Fix longjmp clobber warnings in parser.c.

Make parsebackq a function instead of an emulated nested function.
This puts the setjmp usage in a smaller function where it is easier to avoid
bad optimizations.

Create the altix project branch. The altix project will add support
for the SGI Altix 350 to FreeBSD/ia64. The hardware used for porting
is a two-module system, consisting of a base compute module and a
CPU expansion module. SGI's NUMAFlex architecture can be an excellent
platform to test CPU affinity and NUMA-aware features in FreeBSD.

MFC r204276:

Fix expansion of \W in prompt strings when the working directory is "/".
The prompt string was truncated after \W when the working directory was "/".

PR: bin/89410

Fix expansion of \W in prompt strings when the working directory is "/".
The prompt string was truncated after \W when the working directory was "/".

PR: bin/89410
Submitted by: Dr Balwinder Singh Dheeman
MFC after: 1 week

sh: Fix some bugs with backquoted builtins:
- correctly handle error output in $(builtin 2>&1), clarify out1/out2 vs
output/errout in the code
- treat all builtins as regular builtins so errors do not abort the shell
and variable assignments do not persist
- respect the caller's INTOFF

Some bugs still exist:
- expansion errors may still abort the shell
- some side effects of expansions and builtins persist

Fix memory leak when parsing backticks (``).

sh: Various warning fixes (from WARNS=6 NO_WERROR=1):
- const
- initializations to silence -Wuninitialized (it was safe anyway)
- remove nested extern declarations
- rename "index" locals to "idx"

sh: Constify various strings.

Most of this is adding const keywords, but setvar() in var.c had to be
changed somewhat more.

MFC r198173: sh: show more info about syntax errors in command substitution:
the line number where the command substitution started.
This applies to both the $() and `` forms but is most useful for ``
because the other line number is relative to the enclosed text there.
(For older versions, -v can be used as a workaround.)

Fix various things about SIGINT handling:
* exception handlers are now run with interrupts disabled, which avoids
many race conditions
* fix some cases where SIGINT only aborts one command and continues the
script, in particular if a SIGINT causes an EINTR error which trumped the
interrupt.

Example:
sh -c 'echo < /some/fifo; echo This should not be printed'
The fifo should not have writers. When pressing ctrl+c to abort the open,
the shell used to continue with the next command.

Example:
sh -c '/bin/echo < /some/fifo; echo This should not be printed'
Similar. Note, however, that this particular case did not and does not work
in interactive mode with job control enabled.

sh: Some changes to stderr flushing:
* increase buffer size from 100 to 256 bytes
* remove implied flush from out2str(), in particular this avoids unnecessary
flushing in the middle of a -x tracing line
* rename dprintf() to out2fmt_flush(), make it flush out2 and use this
function in various places where flushing is desired after an error
message

sh: Allow a newline before "in" in a for command, as required by POSIX.

sh: Show more information about syntax errors in command substitution:
the line number where the command substitution started.
This applies to both the $() and `` forms but is most useful for ``
because the other line number is relative to the enclosed text there.
(For older versions, -v can be used as a workaround.)

sh: Disallow mismatched quotes in backticks (`...`).

Due to the amount of code removed by this, it seems that allowing unmatched
quotes was a deliberate imitation of System V sh and real ksh. Most other
shells do not allow unmatched quotes (e.g. bash, zsh, pdksh, NetBSD /bin/sh,
dash).

PR: bin/137657

sh: Improve handling of setjmp/longjmp volatile:
- remove ineffective and unnecessary (void) &var; [1]
- remove some unnecessary volatile keywords
- add a necessary volatile keyword
- save the old handler before doing something that could use the saved
value

Submitted by: Christoph Mallon [1]
Approved by: ed (mentor)

correctly test for __GNUC__ macro (non-GCC compilers do not have it defined at all)

Parse 'cmd1 && ! cmd2 | cmd3' correctly, the bang should apply to the entire
pipeline cmd2 | cmd3 and not just cmd2.

PR: 130298
Submitted by: Jilles Tjoelker

- Import the HEAD csup code which is the basis for the cvsmode work.

Fix checking if a variable name is LINENO. As STPUTC changes the pointer if it
needs to enlarge the buffer, we must not keep a pointer to the beginning.

PR: ports/123879

Expand $LINENO to the current line number. This is required by SUSv3's "User
Portability Utilities" option.

Often configure scripts generated by the autotools test if $LINENO works and
refuse to use /bin/sh if not.

Package test run by: pav

When parsing an invalid parameter expansion (eg. ${} or ${foo@bar}) do not
issue a syntax error immediately but save the information that it is erroneous
for later when the parameter expansion is actually done. This means eg. "false
&& ${}" will not generate an error which seems to be required by POSIX.
Include the invalid parameter expansion in the error message (sometimes
abbreviated with ... because recovering it would require a lot of code).

PR: 105078
Submitted by: emaste

Do not forget to increment the input line counter
when reading a word spanning multiple lines.

PR: bin/101094
MFC after: 5 days

Remove a hack for an ancient gdb.

Put the comparison with PEOF into a new macro is_eof(). Don't use it if the
character comes from a string.

Use assignment rather than comparison.

Include missing headers.

Support \H, \h, \w, \W, \$ string expansion in the prompt.

Submitted by: mini

Remove clause 3 from the UCB licenses.

OK'ed by: imp, core

Replace home-grown dup2() implementation with actual dup2() calls. This
should slightly reduce the number of system calls in critical portions of
the shell, and select a more efficient path through the fdalloc code.

Reviewed by: bde

Changes following CScout analysis:

- Removed dead declarations
- Made objects that should have been declared as static, static.

The changes use STATIC instead of static, following the existing
convention in the rest of the code.

Approved by: schweikh (mentor)
MFC after: 2 weeks

Disallow empty condition parts of "if", "while" and "until" compound
commands. Commands like "if then ... fi" and "while do ... done" are no
longer accepted. Bodies of compound commands are still allowed to be
empty, because even though POSIX does not allow them, most shells do.

Remove bits and pieces of support for atty, which was made obsolete by
adding history and vi/emacs-style line editing to the shell itself.
Atty was a user-mode terminal emulator (like screen and window) that did
line editing and history.

Allow a left parenthesis before patterns in case blocks. POSIX requires
us to accept this, but I've never seen a script that uses it.

Allow empty case/esac statements; POSIX requires this, and recent versions
of autoconf are generating scripts that use this feature.

PR: 43275 35879
Submitted by: Dan Nelson <dnelson@allantgroup.com>

Replace various spellings with FALLTHROUGH which is lint()able

Allow redirections by themselves between "&&" and "||" operators.
For example, >/dev/null && echo foo

Pointed out by: FUJISHIMA Satsuki
MFC after: 1 week

Don't allow "||" or "&&" to be the first tokens of a command.

PR: 40386
MFC after: 2 weeks

Consistently use FBSDID

Minor const cleanup.

Don't discard qualifiers we don't need to discard.

Implement the -C (-o noclobber) option, which prevents existing regular
files from being overwritten by shell redirection.

o __P has been reoved
o Old-style K&R declarations have been converted to new C89 style
o register has been removed
o prototype for main() has been removed (gcc3 makes it an error)
o int main(int argc, char *argv[]) is the preferred main definition.
o Attempt to not break style(9) conformance for declarations more than
they already are.
o Change
int
foo() {
...
to
int
foo(void)
{
...

BASESYNTAX, DQSYNTAX, SQSYNTAX and ARISYNTAX handles negative indexes.
Allow those to be used to properly quote characters in the shell
control character range.

``|'' should be more binding than ``!'' so that this isn't broken:

if ! echo bla | wc -c ; then
echo broken
fi

Obtained from: NetBSD

A much better (more correct) fix for handling ``!'' characters

Obtained from: NetBSD

Handle ``!'' characters when they appear as second and subsequent
parts of an && or || expression.

This makes this expression work as expected:

if true && ! false; then echo yes; fi

Implement the <> redirection operator.

Disable part of my 8-bits fixes from December 1999.

Serious fix still needed, see discussion on -current
(Subject: /bin/sh dumps core with here-document of 8bit text)

Problem in this code originally spotted by
Jun Kuriyama <kuriyama@FreeBSD.org>

Fix parsing of string for eval command.

PR: 18447
Submitted by: Koji Mori <mori@tri.asanuma.co.jp>

Fix warnings, some of them serious because sh violated name
spaces reserved by the header files it includes.

mkinit.c still produces C code with redundant declarations, although
they are more harmless since they automatically derived from the right
places.

Fix ${#varname} (getting length of string) when in double-quotes.

Approved-by: jkh

PR: bin/12137
Submitted by: "Danny J. Zerkel" <dzerkel@columbus.rr.com>

Second part of 8-bit fixes.

First round of 8-bit fixes.

Fix "subscript has type `char'" warnings by casting to int, as
discussed on -arch.

$Id$ -> $FreeBSD$

Be more consistent with handling of quote mark control character.
Don't output double-quotes inside variable expansion/arithmetic
expansion region in here-documents. When leaving the arithmetic
expansion syntax mode, adjust the dblquote flag according to
previous syntax, in order to avoid splitting of quoted variables.

Better handling of word splitting. Don't record the same region
multiple times when performing nested variable expansion, and
preserve some quoting information in order to avoid removing
apparently empty expansion result.

Add rcsid. Spelling.

Use the __unused attribute where warranted.

Revert changes from rev 1.16 to 1.17 for now. Closes PR 2879.

Revert $FreeBSD$ to $Id$

Make the long-awaited change from $Id$ to $FreeBSD$

This will make a number of things easier in the future, as well as (finally!)
avoiding the Id-smashing problem which has plagued developers for so long.

Boy, I'm glad we're not using sup anymore. This update would have been
insane otherwise.

Merge in NetBSD mods and -Wall cleaning.

Obtained from: NetBSD, me

Ok, lets try this again, shall we? It was definatly my mistake, not
Steve's.. :-]

ack! back these out so I can see what I did wrong. It looks like a
patch-by-hand botch, but it sig-11's during make world.

Fix for PR#1248, sh doesn't expand past ${9}

Submitted by: Steve Price <sprice@hiwaay.net>

Fix for PR#1287. This makes sh behave sensibly in case statements in the
face of aliases. Note, bash doesn't do aliases while running scripts, but
"real" ksh does..

Also:
Reduce redundant .Nm macros in (unused) bltin/echo.1
nuke error2, it's hardly used.
More -Wall cleanups
dont do certain history operations if NO_HISTORY defined
handle quad_t's from resource limits

Submitted by: Steve Price <sprice@hiwaay.net> (minor tweaks by me)

Merge of 4.4-Lite2 sh source, plus some gcc -Wall cleaning. This is a
merge of parallel duplicate work by Steve Price and myself. :-]

There are some changes to the build that are my fault... mkinit.c was
trying (poorly) to duplicate some of the work that make(1) is designed to
do. The Makefile hackery is my fault too, the depend list was incomplete
because of some explicit OBJS+= entries, so mkdep wasn't picking up their
source file #includes.

This closes a pile of /bin/sh PR's, but not all of them..

Submitted by: Steve Price <steve@bonsai.hiwaay.net>, peter

Import the 4.4BSD-Lite2 /bin/sh sources

Requested by: joerg

(Note, this is mostly going to be conflicts, which is expected. Our entire
sh source has a mainline, so this should not change anything except for
a few new files appearing. I dont think they are a problem)

Fix the borokeness that crept in with rev 1.10 of parser.c, the sh
didn't correctly start background jobs anymore. Strange that nobody
was complaining...

Add a dummy target for `builtins' in the Makefile, to prevent it
from attempting to build this file by compiling builtins.c. :-/

The shell incorrectly gave & precedence over ;. This breaks the
traditional behaviour, and it violates Posix.2.

Fixes PR # bin/880: /bin/sh incorrectly parse...

Fixes also an earlier problem report about the shell not evaluating
loops correctly. (Not files via GNATS.)

Submitted by: nnd@itfs.nsk.su (Nickolay N. Dudorov)

Sigh. This will become a never ending story. :-(

When comparing my recent parser change against the ash in 1.1.5.1, i
found that a couple of other problems in the same area has been fixed
there, but not in 2.2. Semicolons and EOF do also delimit words...

Make the shell handle a null command in a &&/|| sequence correctly.
The && and || tokens do also terminate a command, not only the
newline.

While i was at it, disabled trace code by default, it served no good
purpose since it required the use of a debugger anyway to be turned
on. Instead, placed a hint in the Makefile on how to turn it on.

This makes the shell ~ 10 % faster and ~ 4 KB smaller. :)

Pointed out by: jan@physik.TU-Berlin.DE (Jan Riedinger)

sh(1) incorrectly ignored an EOF condition when looking for the
closing backquote in a `foo` substitution.

Discovered by: Martin Welk <mw@theatre.pandora.sax.de>

Remove trailing whitespace.

Reviewed by: phk

What I think is a more correct fix for the handling of backslashes
inside backquotes. Reversed my previous fix.

Fix a bug with handling backslash escapes inside some quotes.
Should solve our problems with edit-pr.

Added $Id$

With '!' being made into a keyword (yech!), case cases didn't work properly.
This should fix it (passed my test cases). Originally discovered with
perl's Configure (well, in FreeBSD, I don't know how the NetBSD folks
discovered it).

Reviewed by: sef
Submitted by: jtc@cygnus.com
Obtained from: NetBSD

BSD 4.4 Lite bin Sources