bin: Automated cleanup of cdefs and other formatting

Apply the following automated changes to try to eliminate
no-longer-needed sys/cdefs.h includes as well as now-empty
blank lines in a row.

Remove /^#if.*\n#endif.*\n#include\s+<sys/cdefs.h>.*\n/
Remove /\n+#include\s+<sys/cdefs.h>.*\n+#if.*\n#endif.*\n+/
Remove /\n+#if.*\n#endif.*\n+/
Remove /^#if.*\n#endif.*\n/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/types.h>/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/param.h>/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/capsicum.h>/

Sponsored by: Netflix

bin: Remove ancient SCCS tags.

Remove ancient SCCS tags from the tree, automated scripting, with two
minor fixup to keep things compiling. All the common forms in the tree
were removed with a perl script.

Sponsored by: Netflix

Remove $FreeBSD$: one-line .c pattern

Remove /^[\s*]*__FBSDID\("\$FreeBSD\$"\);?\s*\n/

sh: Fix ${param?} default error message

If word in ${param?word} is missing, the shell shall write a default error
message. So expanding ${param?} when param is not set should write an error
message like

sh: param: parameter not set

This was broken by r316417.

PR: 233585

sh: Fix formal overflow in pointer arithmetic

The intention is to lower the value of the pointer, which according to ubsan
cannot be done by adding an unsigned quantity.

Reported by: kevans
Approved by: re (kib)
MFC after: 1 week

sh: Don't have [ match any [[:class:]]

Submitted by: Robert Elz
MFC after: 3 days

sh: Fix '-' from quoted arithmetic in case/glob pattern range.

It does not make much sense to generate the '-' in a pattern bracket
expression using arithmetic expansion, but it does not make sense to forbid
it either.

Try to avoid reprocessing the string if it is unnecessary.

sh: Update TOUR and comments for some code changes, some of them old.

Also, improve some terminology in TOUR and comments.

sh: Simplify handling of newlines in command substitution.

Unless we need to split on newlines, just append them as normal and remove
them at the end.

sh: Remove global state from collate_range_cmp().

The global state is not used across invocations of collate_range_cmp().

sh: Don't scan word twice in ${param=word}.

sh: Don't scan word twice in ${param#%##%%word}.

If word is expanded, use the found end instead of iterating over the data
again.

sh: Don't scan word twice in ${param+-word}.

There is no longer a case where argbackq has already been advanced but the
string pointer needs to be advanced.

sh: Fix build with -DDEBUG=2.

With the new expansion code (word splitting during instead of after other
expansion processing), tracing the result of command substitution is no
longer possible, so stop trying.

sh: Remove unused return value of subevalvar_misc().

sh: Get rid of global variable argbackq.

sh: Fix executing wrong command with ${unsetvar#$(cmdsubst)}$(cmdsubst).

The parsed internal representation of words consists of a byte string with a
list of nodes (commands in command substitution). Each unescaped CTLBACKQ or
CTLBACKQ | CTLQUOTE byte corresponds to an entry in the list.

If param in ${param#%##%%word} is not set, the word is not expanded (in a
deviation of POSIX shared with other ash variants and ksh93). Erroneously,
the pointer in the list of commands (argbackq) was not advanced. This caused
the wrong command to be executed later if the outer word contained another
command substitution.

Example:
echo "${unsetvar#$(echo a)}$(echo b)"
wrote "a" but should write "b".

MFC after: 1 week

sh: Fix crash if a -T trap is taken during command substitution.

Code like t=$(stat -f %m "$file") segfaulted if -T was active and a trap
was taken while the shell was waiting for the child process to finish.

What happened was that the dotrap() call in waitforjob() was hit. This
re-entered command execution (including expand.c) at a point not expected by
expbackq(), and global state (unallocated stack string and argbackq) was
corrupted.

To fix this, change expbackq() to prepare for command execution to be
re-entered.

Reported by: bdrewery
MFC after: 1 week

Renumber copyright clause 4

Renumber cluase 4 to 3, per what everybody else did when BSD granted
them permission to remove clause 3. My insistance on keeping the same
numbering for legal reasons is too pedantic, so give up on that point.

Submitted by: Jan Schaumann <jschauma@stevens.edu>
Pull Request: https://github.com/freebsd/freebsd/pull/96

sh: Fix a clang warning.

Submitted by: bdrewery

Path generation was not according to collate

Approved by: jilles

Back out non-collating [a-z] ranges.
Instead of changing the whole course to another POSIX-permitted way
for consistency and uniformity I decide to completely ignore missing
regex fucntionality and focus on fixing bugs in what we have now,
too many small obstacles we have choicing other way, counting ports.
Corresponding libc changes are backed out in r302824.

After removing collation for [a-z] ranges in r302512, do it here too.

Approved by: jilles

sh: Fix some dead stores.

Found by: clang static analyzer

sh: Constify internal representation in expand.c.

Forbid (temporary or permanent) modifications of the strings in NARG nodes
during expansion.

Tilde expansion now needs to copy the username for the terminating '\0'.

sh: Reduce size of options table.

sh: Remove redundant CTLQUOTEMARK checks.

With the new expand.c code, the intermediate representation passed to the
pathname generation code only contains CTLESC, not CTLQUOTEMARK.

CTLQUOTEMARK now only occurs in the text of NARG nodes (output of the
parser).

sh: Reindent expandmeta().

sh: Perform pathname generation during the first expansion phases.

This avoids the need to add and remove CTLESC bytes if pathname generation
will not be performed (set -f).

Side effect: the order of operations is slightly different: pathname
generation in ${$+* $(CMD)} will not see filesystem changes from CMD.

sh: Perform IFS splitting during the first expansion phases.

This simplifies the code and should be faster in some cases.

Side effect: the order of operations is different so that the value of IFS
used when IFS is modified during expansion (${IFS:=...}, ${IFS=...} or
$((...IFS=...))) may be different. Note that this order is highly unportable
between shells.

sh: Split subevalvar() in #/##/%/%% and =/? parts.

sh: Fix ""$@, which should not use the special case for "$@".

"$@" should expand to no words if there are no positional parameters, but
""$@ should always expand to at least an empty word.

sh: Avoid copying argv for simple commands.

Add dummy entries before and after so arglist's array is directly usable as
argv.

sh: Make struct arglist an array instead of a linked list.

This simplifies the code (e.g. allowing use of qsort(3) instead of a
hand-rolled mergesort) and should have better cache properties.

The waste of unused args arrays after resizes is approximately the same as
the savings from getting rid of the next pointers.

At the same time, remove a piece of global state and move some duplicated
code into a function.

wordexp: Rewrite to make WRDE_NOCMD reliable.

Shell syntax is too complicated to detect command substitution and unquoted
operators reliably without implementing much of sh's parser. Therefore, have
sh do this detection.

While changing sh's support anyway, also read input from a pipe instead of
arguments to avoid {ARG_MAX} limits and improve privacy, and output count
and length using 16 instead of 8 digits.

The basic concept is:
execl("/bin/sh", "sh", "-c", "freebsd_wordexp ${1:+\"$1\"} -f "$2",
"", flags & WRDE_NOCMD ? "-p" : "", <pipe with words>);

The WRDE_BADCHAR error is still implemented in libc. POSIX requires us to
fail strings containing unquoted braces with code WRDE_BADCHAR. Since this
is normally not a syntax error in sh, there is still a need for checking
code in libc, we_check().

The new we_check() is an optimistic check that all the characters
<newline> | & ; < > ( ) { }
are quoted. To avoid duplicating too much sh logic, such characters are
permitted when quoting characters are seen, even if the quoting characters
may themselves be quoted. This code reports all WRDE_BADCHAR errors; bad
characters that get past it and are a syntax error in sh return WRDE_SYNTAX.

Although many implementations of WRDE_NOCMD erroneously allow some command
substitutions (and ours even documented this), there appears to be code that
relies on its security (codesearch.debian.net shows quite a few uses).
Passing untrusted data to wordexp() still exposes a denial of service
possibility and a fairly large attack surface.

Reviewed by: wblock (man page only)
MFC after: 2 weeks
Relnotes: yes
Security: fixes command execution with wordexp(untrusted, WRDE_NOCMD)

sh: Add set -o nolog.

POSIX requires this to prevent entering function definitions in history but
this implementation does nothing except retain the option's value. In ksh88,
function definitions were usually entered in the history file, even when
they came from ~/.profile and the $ENV file, to allow displaying their
definitions.

This is also the first option that does not have a letter.

sh: Fix out of bounds read when there is no ] after a [:class:].

The initial check for a matching ] was incorrect if a ] may be consumed by a
[:class:]. The subsequent loop assumed that there must be a ].

Remove the initial check and make the loop cope with a missing ].

Found with afl-fuzz.

MFC after: 1 week

Fix unitialized variable that broke sh on PowerPC starting with r278826.

sh: Various cleanups to expand.c:

* Remove some gotos.
* Remove unused parameter.
* Remove duplicate code.

sh: Prefer "" to nullstr where possible.

sh: Add stsavestr(), like savestr() but allocates using stalloc().

sh: Remove EXP_REDIR.

EXP_REDIR was supposed to generate pathnames in redirection if exactly one
file matches, as permitted but not required by POSIX in interactive mode. It
is unlikely this will be implemented.

No functional change is intended.

MFC after: 1 week

sh: Use DQSYNTAX only while expanding, not SQSYNTAX.

Quoting during expansion only cares about CCTL, which is the same for
DQSYNTAX and SQSYNTAX.

sh: Fix corruption of CTL* bytes in positional parameters in redirection.

EXP_REDIR was not being checked for while expanding positional parameters in
redirection, so CTL* bytes were not being prefixed where they should be.

MFC after: 1 week

Treat IFS separators in "$*" as quoted.

This makes a difference if IFS starts with *, ?, [ or a CTL* byte.

sh: Remove more gotos.

sh: Eliminate some gotos.

sh: Correctly handle positional parameters beyond INT_MAX on 64-bit systems.

Currently, there can be no more than INT_MAX positional parameters. Make
sure to treat all higher ones as unset to avoid incorrect results and
crashes.

On 64-bit systems, our atoi() takes the low 32 bits of the strtol() and
sign-extends them.

On 32-bit systems, the call to atoi() returned INT_MAX for too high values
and there is not enough address space for so many positional parameters, so
there was no issue.

sh: Consistently treat ${01} like $1.

Leading zeroes were ignored when checking whether a positional parameter is
set, but not when expanding its value. Ignore leading zeroes in any case.

sh: Fix possible memory leaks and double frees with unexpected SIGINT.

sh: Add some consts.

sh: Make argstr() return where it stopped and simplify expari() using this.

sh: Simplify expari().

Redo expari() like evalvar(). This makes the logic more understandable and
avoids possible problems if arithmetic expansion occurs if CTLESC characters
are not generated (looking backwards for CTLARI is not generally possible in
that case but the old code tried anyway).

This adds an extra argstr() recursion.

sh: Do not corrupt internal representation if LINENO inner expansion fails.

Example:
f() { : ${LINENO+$((1/0))}; }
and call this function twice.

sh: Make expari() static.

sh: Prefer memcpy() to strcpy() in most cases. Remove the scopy macro.

sh: Fix various compiler warnings.

It now passes WARNS=7 with clang on i386.

GCC 4.2.1 does not understand setjmp() properly so will always trigger
-Wuninitialized. I will not add the volatile keywords to suppress this.

sh: Expand here documents in the current process.

Expand here documents at the same point other redirections are expanded but
use a non-fork subshell environment (like simple command substitutions) for
compatibility. Substitition errors result in an empty here document like
before.

As a result, a fork is avoided for short (<4K) expanded here documents.

Unexpanded here documents (with quoted end marker after <<) are not affected
by this change. They already only forked when >4K.

Side effects:
* Order of expansion is slightly different.
* Slow expansions are not executed in parallel with the redirected command.
* A non-fork subshell environment is subtly different from a forked process.

sh: Make various functions static.

sh: Make patmatch() non-recursive.

sh: Use dirent.d_type in pathname generation.

This improves performance for globs where a slash or another component
follows a component with metacharacters by eliminating unnecessary attempts
to open directories that are not.

sh: Cache de->d_namlen in a local variable.

sh: Add support for named character classes in bracket expressions.

Example:
case x in [[:alpha:]]) echo yes ;; esac

sh: Fix duplicate prototypes for builtins.

Have mkbuiltins write the prototypes for the *cmd functions to builtins.h
instead of builtins.c and include builtins.h in more .c files instead of
duplicating prototypes for *cmd functions in other headers.

sh: Save/restore changed variables in optimized command substitution.

In optimized command substitution, save and restore any variables changed by
expansions (${var=value} and $((var=assigned))), instead of trying to
determine if an expansion may cause such changes.

If $! is referenced in optimized command substitution, do not cause jobs to
be remembered longer.

This fixes $(jobs $!) again, simplifies the man page and shortens the code.

sh: Fix locale-dependent ranges in bracket expressions.

When I added UTF-8 support in r221646, the LC_COLLATE-based ordering broke
because of sign extension of char.

Because of libc restrictions, this does not work for UTF-8. For UTF-8
locales, ranges always use character code order.

sh: Do parameter expansion before printing PS4 (set -x).

The function name expandstr() and the general idea of doing this kind of
expansion by treating the text as a here document without end marker is from
dash.

All variants of parameter expansion and arithmetic expansion also work (the
latter is not required by POSIX but it does not take extra code and many
other shells also allow it).

Command substitution is prevented because I think it causes too much code to
be re-entered (for example creating an unbounded recursion of trace lines).

Unfortunately, our LINENO is somewhat crude, otherwise PS4='$LINENO+ ' would
be quite useful.

sh: Fix unquoted $@/$* if IFS=''.

If IFS is null, unquoted $@/$* should still expand to separate words.
This differs from quoted $@ (which does not depend on IFS) in that pathname
generation is performed and empty words are removed.

sh: Add UTF-8 support to pattern matching.

?, [...] patterns match codepoints instead of bytes. They do not match
invalid sequences. [...] patterns must not contain invalid sequences
otherwise they will not match anything. This is so that ${var#?} removes the
first codepoint, not the first byte, without putting UTF-8 knowledge into
the ${var#pattern} code. However, * continues to match any string and an
invalid sequence matches an identical invalid sequence. (This differs from
fnmatch(3).)

sh: Add UTF-8 support to ${#var}.

If the current locale uses UTF-8, ${#var} counts codepoints (more precisely,
bytes b with (b & 0xc0) != 0x80).

Fix typos - remove duplicate "the".

PR: bin/154928
Submitted by: Eitan Adler <lists at eitanadler.com>
MFC after: 3 days

sh: Remove comment mentioning herefd, which is gone.

sh: Don't do optimized command substitution if expansions have side effects.

Before considering to execute a command substitution in the same process,
check if any of the expansions may have a side effect; if so, execute it in
a new process just like happens if it is not a single simple command.

Although the check happens at run time, it is a static check that does not
depend on current state. It is triggered by:
- expanding $! (which may cause the job to be remembered)
- ${var=value} default value assignment
- assignment operators in arithmetic
- parameter substitutions in arithmetic except ${#param}, $$, $# and $?
- command substitutions in arithmetic

This means that $((v+1)) does not prevent optimized command substitution,
whereas $(($v+1)) does, because $v might expand to something containing
assignment operators.

Scripts should not depend on these exact details for correctness. It is also
imaginable to have the shell fork if and when a side effect is encountered
or to create a new temporary namespace for variables.

Due to the $! change, the construct $(jobs $!) no longer works. The value of
$! should be stored in a variable outside command substitution first.

sh: Allow arbitrary large numbers in CHECKSTRSPACE.
Reduce "stack string" API somewhat and simplify code.
Add a check for integer overflow of the "stack string" length (probably
incomplete).

Remove dead code.

c is assigned 0 and *loc is pointing to NULL, so c!=0 cannot be true,
and dereferencing loc would be a bad idea anyway.

Coverity Prevent: CID 5113
Reviewed by: jilles

sh: Fix corruption of command substitutions with special chars after newline

The CTLESC byte to protect a special character was output before instead of
after a newline directly preceding the special character.

The special handling of newlines is because command substitutions discard
all trailing newlines.

sh: Remove the herefd hack.

The herefd hack wrote out partial here documents while expanding them. It
seems unnecessary complication given that other expansions just allocate
memory. It causes bugs because the stack is also used for intermediate
results such as arithmetic expressions. Such places should disable herefd
for the duration but not all of them do, and I prefer removing the need for
disabling herefd to disabling it everywhere needed.

Here documents larger than 1024 bytes will use a bit more CPU time and
memory.

Additionally this allows a later change to expand here documents in the
current shell environment. (This is faster for small here documents but also
changes behaviour.)

Obtained from: dash

sh: Replace some macros and repeated code in expand.c with functions.

No functional change is intended, but the binary is about 1K smaller on
i386.

sh: Code size optimizations to "stack string" memory allocation:
* Prefer one CHECKSTRSPACE with multiple USTPUTC to multiple STPUTC.
* Add STPUTS macro (based on function) and use it instead of loops that add
nul-terminated strings to the stack string.

No functional change is intended, but code size is about 1K less on i386.

sh: Code size optimizations to buffered output.

This is mainly less use of the outc macro.

No functional change is intended, but code size is about 2K less on i386.

sh: Fix some issues with CTL* bytes and ${var#pat}.

subevalvar() incorrectly assumed that CTLESC bytes were present iff the
expansion was quoted. However, they are present iff various processing such
as word splitting is to be done later on.

Example:
v=@$e@$e@$e@
y="${v##*"$e"}"
echo "$y"
failed if $e contained the magic CTLESC byte.

Exp-run done by: pav (with some other sh(1) changes)

sh: Do IFS splitting on word in ${v+word} and ${v-word}.

The code is inspired by NetBSD sh somewhat, but different because we
preserve the old Almquist/Bourne/Korn ability to have an unquoted part in a
quoted ${v+word}. For example, "${v-"*"}" expands to $v as a single field if
v is set, but generates filenames otherwise.

Note that this is the only place where we split text literally from the
script (the similar ${v=word} assigns to v and then expands $v). The parser
must now add additional markers to allow the expansion code to know whether
arbitrary characters in substitutions are quoted.

Example:
for i in ${$+a b c}; do echo $i; done

Exp-run done by: pav (with some other sh(1) changes)

A new jail(8) with a configuration file, to replace the work currently done
by /etc/rc.d/jail.

In the spirit of r90111, depend on c89 and remove the "STATIC" macro
and its usage.

Make DEBUG traces 64-bit clean:
- Use %t to print ptrdiff_t values.
- Cast a ptrdiff_t value explicitly to int for a field width specifier.

While here, sort includes.

Submitted by: Garrett Cooper

Consistently use "STATIC" for all functions in order to be able to set
breakpoints with in a debugger. And use naked "static" for variables.

Noticed by: bde

sh: Improve comments in expand.c.

sh: Remove remnants of '!!' to negate pattern.

This Almquist extension was disabled long ago.

In pathname generation, components starting with '!!' were treated as
containing wildcards, causing unnecessary readdir (which could fail, causing
pathname generation to fail while it should not).

sh: Fix heap-based buffer overflow in pathname generation.

The buffer for generated pathnames could be too small in some cases. It
happened to be always at least PATH_MAX long, so there was never an overflow
if the resulting pathnames would be usable.

This bug may be abused if a script subjects input from an untrusted source
to pathname generation, which a bad idea anyhow. Most shell scripts do not
work on untrusted data. secteam@ says no advisory is necessary.

PR: bin/148733
Reported by: Changming Sun snnn119 at gmail com
MFC after: 10 days

sh: Forget about terminated background processes sooner.

Unless $! has been referenced for a particular job or $! still contains that
job's pid, forget about it after it has terminated. If $! has been
referenced, remember the job until the wait builtin has reported its
completion (either with the pid as parameter or without parameters).

In interactive mode, jobs are forgotten after termination has been reported,
which happens before primary prompts and through the jobs builtin. Even
then, though, remember a job if $! has been referenced.

This is similar to what is suggested by POSIX and should fix most memory
leaks (which also tend to cause sh to use more CPU time) with long running
scripts that start background jobs.

Caveats:
* Repeatedly referencing $! without ever doing 'wait', like
while :; do foo & echo started foo: $!; sleep 60; done
will still use a lot of memory and CPU time in the long run.
* The jobs and jobid builtins do not cause a job to be remembered for longer
like expanding $! does.

PR: bin/55346

MFC r207944: sh: Fix pathname expansion with quoted slashes like *\/.

These are git commits 36f0fa8fcbc8c7b2b194addd29100fb40e73e4e9 and
d6d06ff5c2ea0fa44becc5ef4340e5f2f15073e4 in dash.

Because this is the first code I'm importing from dash to expand.c, add the
Herbert Xu copyright notice which is in dash's expand.c.

When pathname expanding *\/, the CTLESC representing the quoted state was
erroneously taken as part of the * pathname component. This CTLESC was then
seen by the pattern matching code as escaping the '\0' terminating the
string.

The code is slightly different because dash converts the CTLESC characters
to backslashes and removes all the other CTL* characters to allow
substituting glob(3).

The effect of the bug was also slightly different from dash (where nothing
matched at all). Because a CTLESC can escape a '\0' in some way, whether
files were included despite the bug depended on memory that should not be
read. In particular, on many machines /*\/ expanded to a strict subset of
what /*/ expanded to.

Example:
echo /*"/null"

This should print /dev/null, not /*/null.

PR: bin/146378
Obtained from: dash

sh: Fix pathname expansion with quoted slashes like *\/.

These are git commits 36f0fa8fcbc8c7b2b194addd29100fb40e73e4e9 and
d6d06ff5c2ea0fa44becc5ef4340e5f2f15073e4 in dash.

Because this is the first code I'm importing from dash to expand.c, add the
Herbert Xu copyright notice which is in dash's expand.c.

When pathname expanding *\/, the CTLESC representing the quoted state was
erroneously taken as part of the * pathname component. This CTLESC was then
seen by the pattern matching code as escaping the '\0' terminating the
string.

The code is slightly different because dash converts the CTLESC characters
to backslashes and removes all the other CTL* characters to allow
substituting glob(3).

The effect of the bug was also slightly different from dash (where nothing
matched at all). Because a CTLESC can escape a '\0' in some way, whether
files were included despite the bug depended on memory that should not be
read. In particular, on many machines /*\/ expanded to a strict subset of
what /*/ expanded to.

Example:
echo /*"/null"

This should print /dev/null, not /*/null.

PR: bin/146378
Obtained from: dash

sh: Use stalloc for arith variable names.

This is simpler than the custom memory tracker I added earlier, and is also
needed by the dash arith code I plan to import.

sh: Do tilde expansion in substitutions.

This applies to word in ${v-word}, ${v+word}, ${v=word}, ${v?word} (which
inherits quoting from the outside) and in ${v%word}, ${v%%word}, ${v#word},
${v##word} (which does not inherit any quoting).

In all cases tilde expansion is only attempted at the start of word, even if
word contains spaces. This agrees with POSIX and other shells.

This is the last part of the patch tested in the exp-run.

Exp-run done by: erwin (with some other sh(1) changes)

sh: Allow quoting pattern match characters in ${v%pat} and ${v#pat}.

Note that this depends on r206145 for allowing pattern match characters to
have their special meaning inside a double-quoted expansion like "${v%pat}".

PR: bin/117748
Exp-run done by: erwin (with some other sh(1) changes)

Create the altix project branch. The altix project will add support
for the SGI Altix 350 to FreeBSD/ia64. The hardware used for porting
is a two-module system, consisting of a base compute module and a
CPU expansion module. SGI's NUMAFlex architecture can be an excellent
platform to test CPU affinity and NUMA-aware features in FreeBSD.

sh: Fix some bugs with backquoted builtins:
- correctly handle error output in $(builtin 2>&1), clarify out1/out2 vs
output/errout in the code
- treat all builtins as regular builtins so errors do not abort the shell
and variable assignments do not persist
- respect the caller's INTOFF

Some bugs still exist:
- expansion errors may still abort the shell
- some side effects of expansions and builtins persist

sh: Various warning fixes (from WARNS=6 NO_WERROR=1):
- const
- initializations to silence -Wuninitialized (it was safe anyway)
- remove nested extern declarations
- rename "index" locals to "idx"

sh: Do not consider a tilde-prefix with expansions in it.

That is, do not do tilde expansion if any of the CTL* bytes (\201-\210), not
only CTLESC and CTLQUOTEMARK, are encountered. Such an expansion would look
up a user name with sh's internal representation.

The parser does not currently distinguish between backslashed and
unbackslashed \201-\210, so tilde expansion of user names with these bytes
in them is not so easy to fix.

sh: Constify various strings.

Most of this is adding const keywords, but setvar() in var.c had to be
changed somewhat more.

sh: Exempt $@ and $* from set -u

This seems more useful and will likely be in the next POSIX standard.

Also document more precisely in the man page what set -u does (note that
$@, $* and $! are the only special parameters that can ever be unset, all
the others are always set, although they may be empty).

Fix some weirdnesses in the NetBSD IFS code,
in particular "$@"$ifschar if the final positional parameter is empty.
With the NetBSD code, adding the $ifschar removes a parameter.

PR: standards/79067
Approved by: ed (mentor) (implicit)

Improve IFS expansion using code from NetBSD.

We now pass the ifs.sh testsuite.

PR: standards/79067
Approved by: ed (mentor) (implicit)
Obtained from: NetBSD

- Import the HEAD csup code which is the basis for the cvsmode work.

Expand $LINENO to the current line number. This is required by SUSv3's "User
Portability Utilities" option.

Often configure scripts generated by the autotools test if $LINENO works and
refuse to use /bin/sh if not.

Package test run by: pav

Sigh, when reapplying the patch to HEAD, I somehow forgot to commit this file.

Reported by: Jaakko Heinonen

Fix expanding of quoted positional parameters in case patterns.

Obtained from: NetBSD (expand.c 1.58 and 1.59)
Submitted by: Paul Jarc
PR: 56147

When parsing an invalid parameter expansion (eg. ${} or ${foo@bar}) do not
issue a syntax error immediately but save the information that it is erroneous
for later when the parameter expansion is actually done. This means eg. "false
&& ${}" will not generate an error which seems to be required by POSIX.
Include the invalid parameter expansion in the error message (sometimes
abbreviated with ... because recovering it would require a lot of code).

PR: 105078
Submitted by: emaste

Remove some white space at EOL.

Various small code cleanups resulting from a code reviewing
and linting procedure:

1. Remove useless sub-expression:

- if (*start || (!ifsspc && start > string && (nulonly || 1))) {
+ if (*start || (!ifsspc && start > string)) {

The sub-expression "(nulonly || 1)" always evaluates to true and
according to CVS logs seems to be just a left-over from some
debugging and introduced by accident. Removing the sub-expression
doesn't change semantics and a code inspection showed that the
variable "nulonly" is also not necessary here in any way (and the
expression would require fixing instead of removing).

2. Remove dead code:

- if (backslash && c == '\\') {
- if (read(STDIN_FILENO, &c, 1) != 1) {
- status = 1;
- break;
- }
- STPUTC(c, p);
- } else if (ap[1] != NULL && strchr(ifs, c) != NULL) {
+ if (ap[1] != NULL && strchr(ifs, c) != NULL) {

Inspection of the control and data flow showed that variable
"backslash" is always false (0) when the "if"-expression is
evaluated, hence the whole block is effectively dead code.
Additionally, the skipping of characters after a backslash is already
performed correctly a few lines above, so this code is also not
needed at all. According to the CVS logs and the ASH 0.2 sources,
this code existed in this way already since its early days.

3. Cleanup Style:

- ! trap[signo][0] == '\0' &&
+ ! (trap[signo][0] == '\0') &&

The expression wants to ensure the trap is not assigned the empty
string. But the "!" operator has higher precedence than "==", so the
comparison should be put into parenthesis to form the intended way of
expression. Nevertheless the code was effectively not really broken
as both particular NUL comparisons are semantically equal, of course.
But the parenthesized version is a lot more intuitive.

4. Remove shadowing variable declaration:

- char *q;

The declaration of symbol "q" hides another identical declaration of
"q" in the same context. As the other "q" is already reused multiple
times and also can be reused again without negative side-effects,
just remove the shadowing declaration.

5. Just small cosmetics:

- if (ifsset() != 0)
+ if (ifsset())

The ifsset() macro is already coded by returning the boolean result
of a comparison operator, so no need to compare this boolean result
again against a numerical value. This also aligns the macros usage to
the remaining existing code.

Reviewed by: stefanf@

Fix a bug when shell expansion is done against dangling symlinks, by
converting the stat() call to a lstat() call, which will cover the
situation. One can exercise this bug by referring a dangling link with
something like */the-link.

Approved by: re (scottl)
Submitted by: Simon 'corecode' Schubert [corecode fs ei tum de]
Obtained from: NetBSD via DragonFlyBSD (NetBSD rev. 1.51 and DragonFly
rev. 1.6)
MFC After: 3 days

Remove clause 3 from the UCB licenses.

OK'ed by: imp, core

Remove collate_range_cmp() stabilization, it conflicts with ranges

Changes following CScout analysis:

- Removed dead declarations
- Made objects that should have been declared as static, static.

The changes use STATIC instead of static, following the existing
convention in the rest of the code.

Approved by: schweikh (mentor)
MFC after: 2 weeks

Instead of eating trailing newlines after inserting them into the
output buffer, don't insert them at all. This prevents a buffer
*underrun* when the substitution consists completely of newlines
(e.g. `echo`) and the byte before the source buffer to which p
points is a '\n', in which case more characters would be removed
from the output buffer than were inserted.

This fixes certain port builds on sparc64.

Approved by: re (scottl)
Reviewed by: des, tjr

Fixed (soon might be fatal) -Wformat warnings.

Do not strip CTL* escapes from redirection filenames in exptilde(),
expari(), expbackq() and evalvar(). Similar to revision 1.39.
Patch from Tor Egge.

PR: 45349
MFC after: 2 weeks

Add the "wordexp" shell built-in command which will be used to implement
the POSIX wordexp() function.

Do not strip CTL* escapes from redirection filenames in argstr(); they
are later stripped with rmescapes() in expandarg(). If the filename has
already been unescaped, doing it again in rmescapes() can walk off the
end of the string, leading to memory corruption and eventually SIGSEGV.

Noticed by: kris

Replace various spellings with FALLTHROUGH which is lint()able

Consistently use FBSDID

Implement the -u (-o nounset) option, which gives an error message if
an unset variable is expanded.

Obtained from: NetBSD (bjh21, christos)

o __P has been reoved
o Old-style K&R declarations have been converted to new C89 style
o register has been removed
o prototype for main() has been removed (gcc3 makes it an error)
o int main(int argc, char *argv[]) is the preferred main definition.
o Attempt to not break style(9) conformance for declarations more than
they already are.
o Change
int
foo() {
...
to
int
foo(void)
{
...

Don't check uninitialized memory for having the shell control character
value CTLARI since this might break expansion of arithmetic expressions.

Don't access memory below start of stackblock.

Problem analyzed by hunt@iprg.nokia.com, slightly different patch applied.

PR: 24443
Submitted by: hunt@iprg.nokia.com

BASESYNTAX, DQSYNTAX, SQSYNTAX and ARISYNTAX handles negative indexes.
Allow those to be used to properly quote characters in the shell
control character range.

Fix environment passung to eval'ed commands.

PR: bin/6577
Submitted by: Anatoly Vorobey <mellon@pobox.com>
Approved by: silence amoung other sh experts

First round of 8-bit fixes.

Fix "subscript has type `char'" warnings by casting to int, as
discussed on -arch.

$Id$ -> $FreeBSD$

Various spelling/formatting changes.

Submitted by: Philippe Charnier <charnier@xp11.frmug.org>

Next approach to make loops in interactive interruptable.

PR: bin/9173

During variable expansion, the internal representation of the expression
might be relocated. Handle this case.
PR: 7059

When a variable expansion is enclosed in double quotes, the internal
representation of the expression is quoted. Take care of this when
doing pattern matching in conjunction with trimming.

#!/bin/sh
c=d:e; echo "${c%:e}"

PR: NetBSD PR#7231
Noticed by: Havard Eidnes <Havard.Eidnes@runit.sintef.no>

Be more consistent with handling of quote mark control character.
Don't output double-quotes inside variable expansion/arithmetic
expansion region in here-documents. When leaving the arithmetic
expansion syntax mode, adjust the dblquote flag according to
previous syntax, in order to avoid splitting of quoted variables.

Better handling of word splitting. Don't record the same region
multiple times when performing nested variable expansion, and
preserve some quoting information in order to avoid removing
apparently empty expansion result.

Add rcsid. Spelling.

Back out previous fix - this bug's got diplomatic immunity as a registered
political issue.

>Number: 3780
>Category: bin
>Synopsis: WEXITSTATUS() may return nagative value, which causes sh to generate bad $?

PR: 3780
Submitted by: sanewo@ba2.so-net.or.jp

Now [^abc] means the same as [!abc] like bash and *csh already does

Use the __unused attribute where warranted.

Sync with NetBSD's revision 1.29 of this file.

Obtained from: NetBSD

Revert $FreeBSD$ to $Id$

Fix a expansion bug that caused the result of `echo $((1 << 30))`
to get truncated.

Submitted by: bde

Make the long-awaited change from $Id$ to $FreeBSD$

This will make a number of things easier in the future, as well as (finally!)
avoiding the Id-smashing problem which has plagued developers for so long.

Boy, I'm glad we're not using sup anymore. This update would have been
insane otherwise.

This doesn't change any behavior, but may be a slight
optimization. (num-- > 0) --> (--num >= 0).

Obtained from: NetBSD

Merge in NetBSD mods and -Wall cleaning.

Obtained from: NetBSD, me

1) define STATIC as static and not empty
2) replace collate_range_cmp call with its code

Ok, lets try this again, shall we? It was definatly my mistake, not
Steve's.. :-]

ack! back these out so I can see what I did wrong. It looks like a
patch-by-hand botch, but it sig-11's during make world.

Fix for PR#1248, sh doesn't expand past ${9}

Submitted by: Steve Price <sprice@hiwaay.net>

Merge of 4.4-Lite2 sh source, plus some gcc -Wall cleaning. This is a
merge of parallel duplicate work by Steve Price and myself. :-]

There are some changes to the build that are my fault... mkinit.c was
trying (poorly) to duplicate some of the work that make(1) is designed to
do. The Makefile hackery is my fault too, the depend list was incomplete
because of some explicit OBJS+= entries, so mkdep wasn't picking up their
source file #includes.

This closes a pile of /bin/sh PR's, but not all of them..

Submitted by: Steve Price <steve@bonsai.hiwaay.net>, peter

Convert to newly added collate compare function

Localize it

Import the 4.4BSD-Lite2 /bin/sh sources

Requested by: joerg

(Note, this is mostly going to be conflicts, which is expected. Our entire
sh source has a mainline, so this should not change anything except for
a few new files appearing. I dont think they are a problem)

Remove trailing whitespace.

Reviewed by: phk

Fix the deletion of trailing newlines with backquote expansion.
Reviewed by:
Submitted by:
Obtained from:

Added $Id$

BSD 4.4 Lite bin Sources