improvements in malloc/free handling.

Disable attempts to handle EINTR and non-blocking I/O by default. It is
confusing to other programs and unexpected behavior. Reported by Ingo Schwarze.
This behavior is now controlled with EL_SAFEREAD.

PR/54399: S�ren Tempel: Uninitialized memory access in libedit history.
Initialize the buffer using calloc. While here change all malloc(a * sizeof(b))
to calloc(a, sizeof(b)). XXX: should fix realloc similarly.

From Yuichiro Naito (FreeBSD):

hrs@ says that
(cbp >= MB_LEN_MAX) condition is necessary for checking invalid
byte sequences. If malicious input was given, libedit would read
byte sequences forever.

Remove utf-8 requirement (Yuichiro NAITO)

branches: 1.103.4; 1.103.6;
remove unused variable

branches: 1.102.6;
PR/51706: Amir Plivatsky: Fix memory leak

branches: 1.101.2;
abstract read code to a single function (Ingo Schwarze)

el_map.alt can't be NULL here (Ingo Schwarze)

remove debug read (Ingo Schwarze)

From Ingo Schwarze:

Reduce obfuscation of errno handling. There is only one purpose
non-local errno handling is needed for: Inside el_wgets(), several
functions call down indirectly to el_wgetc(), many of them via the
dispatch table. When el_wgetc() fails, it does properly report
failure, but then various cleanup is done which may clobber errno.
But when returning due to failure, el_wgets() wants to have errno
set to the reason of the original read failure, not to the reason
of some subsequent failure of some cleanup operation. So el_wgetc()
needs to save errno, and if it's non-zero, el_wgets() needs to
restore it on failure.

This core logic is currently obscured by the fact that el_errno
is set and inspected at some additional places where it isn't needed.
Besides, since el_wgetc() and and el_wgets() are both in read.c,
el_errno does not need to be in struct editline, it can and should
be local to read.c in struct el_read_t.

Let's look at what can be simplified.

1. keymacro_get() abuses el_errno instead of having a proper
error return code. Adding that error return code is easy
because node_trav() already detects the condition and an
adequate code is already defined. Returning it, testing
for it in read_getcmd(), and returning with error from there
removes the need to inspect el_errno from el_wgets() after
calling read_getcmd().
Note that resetting lastchar and cursor and clearing buffer[0]
is irrelevant. The code returns from el_wgets() right afterwards.
Outside el_wgets(), these variables are no longer relevant.
When el_wgets() is called the next time, it will call ch_reset()
anyway, resetting the two pointers. And as long as lastchar
points to the beginning of the buffer, the contents of the
buffer won't be used for anything.

2. read_getcmd() doesn't need to set el_errno again after el_wgetc()
failure since el_wgetc() already did so. While here, remove
the silly "if EOF or error" comments from the el_wgetc()
return value tests. It's a public interface documented in a
manual, so people working on the implementation can obviously
be expected to know how it works. It's a case of

count++; /* Increment count. */

3. In the two code paths of el_wgets() that lead up to "goto noedit",
there is no need to save the errno because nothing that might
change it happens before returning.

For clarity, since el_wgets() is the function restoring the errno,
also move initializing it to the same function.

Finally, note that restoring errno when the saved value is zero is
wrong. No library code is ever allowed to clear a previously set
value of errno. Only application programs are allowed to do that,
and even they usually don't need to do so, except when using certain
ill-designed interfaces like strtol(3).

I tested that the behaviour remains sane in the following cases,
all during execution of el_wgets(3) and with a signal handler
for USR1 installed without SA_RESTART.

* Enter some text and maybe move around a bit.
Then send a USR1 signal.
The signal gets processed, then read_char() resumes reading.
Send another USR1 signal.
Now el_wgets() sets errno=EINTR and returns -1.

* Press Ctrl-V to activate ed-quoted-insert.
Then send a USR1 signal.
The signal gets processed, then read_char() resumes reading.
Send another USR1 signal.
ed_quoted_insert() returns ed_end_of_file(), i.e. CC_EOF,
and el_wgets() returns 0.

* Press a key starting a keyboard macro.
Then send a USR1 signal.
The signal gets processed, then read_char() resumes reading.
Send another USR1 signal.
Now el_wgets() sets errno=EINTR and returns -1.

* Press : to enter builtin command mode.
Start typing a command.
Then send a USR1 signal.
The signal gets processed, then read_char() resumes reading.
Send another USR1 signal.
Now c_gets() returns -1, ed_command() beeps and returns CC_REFRESH,
and el_wgets() resumes operation as it should.

I also tested with "el_set(el, EL_EDITMODE, 0)", and it returns
the right value and sets errno correctly.

Stop the read module from poking the el_chared.c_macro data structure
currently belonging to the chared module. The read module does so
from three of its functions, while no other module uses the macro
data, not even the chared module itself. That's quite logical
because macros are a feature of input handling, all of which is
done by the read module, and none by the chared module. So move
the data into the read modules's own opaque data structure, struct
el_read_t.

That simplifies internal interfaces in several respects: The
semi-public chared.h has one fewer struct, one fewer #define, and
one fewer member in struct el_chared_t; all three move to one single
C file, read.c, and are now module-local. And the internal interface
function ch_reset() needs one fewer argument, making the code of many
functions in various modules more readable.

The price is one additional internal interface function, read_end(),
10 lines long including comments, called publicly from exactly one
place: el_end() in el.c. That's hardly an increase in complexity
since most other modules already have their *_end() function, read.c
was the odd one out not having one.

From Ingo Schwarze

s/protected/libedit_private/g

From Ingo Schwarze:
- Put the data type el_rfunc_t into the public header <histedit.h>.
- Make el_read in struct editline an opaque pointer rather
than an embedded struct.
- Do not include "read.h" everywhere, but only in the two files
needing access to el_read, read.c and el.c.
- To functions that don't need more, pass the struct el_read_t *
rather than the full EditLine *.
- Of course, that means that read_init() can now fail from
memory exhaustion, but it's easy to clean up after that.

From Ingo Schwarze:
* Replace fcns.c by a shorter and simpler func.h
and include it only in the one file needing it, map.c.
* Combine help.h and help.c into a simplified help.h
and include it only in the one file needing it, map.c.
* Check the very simple, static files editline.c, historyn.c, and
tokenizern.c into CVS rather than needlessly generating them.
* So we no longer autogenerate any C files. :-)
* Shorten and simplify makelist by deleting the options -n, -e, -bc,
and -m; the latter was unused and useless in the first place.
* Move the declaration of el_func_t from fcns.h to the header
actually needing it, map.h. Since that header is already
included by el.h for unrelated reasons, that makes el_func_t
just as globally available as before.
* No longer include the simplified fcns.h into el.h,
include it directly into the *.c files needing it.

FIONREAD takes int as an argument (Ingo Schwarze)

From Ingo Schwarze:

* Delete the stubs of the XK_EXE mechanism that was never implemented.
From a security, stability, and simplicity perspective, i would
consider implementing it a truly terrible idea, so let's better
get rid of it.

* Do not use the local variable "num" in el_wgets() alternately for
two completely different purposes. Only use it for the number
of characters read, as stated in the comment (or -1 as long as
that number is still unknown), not for the (more or less boolean)
return value of read_getcmd(). Actually, there is no need at
all to save the latter return value after testing it once.

* The function read_getcmd() has very unusual return values:
It returns -1 for success and 0 for EOF/error. Switch that around
to 0 for success and -1 for EOF/error to be less confusing, and
get rid of the OKCMD preprocessor macro.

* Get rid of one #ifdef section in el_wgets() by using
el->el_chared.c_macro directly at the only place
where it is used.

* Delete the unused MIN() macro.

Get rid of private/public; keep protected (Ingo Schwarze)

Char -> wchar_t from Ingo Schwarze.

more macro WIDECHAR undoing from Ingo Schwarze.

More WIDECHAR elimination (Ingo Schwarze)

Start removing the WIDECHAR ifdefs; building without it has stopped working
anyway. (Ingo Schwarze)

PR/50880: David Binderman: Remove redundant code.
While here, fix all debugging formats.

Tuck in mbstate_t to the wide char version only to avoid exposing the zeroing
hack and doing it in the narrow case.

Make the read_char function always take a wchar_t * argument (Ingo Schwarze)

Get split el_getc and el_wgetc completely and call el_wgetc internally.
Change some character constants to they wide versions. (Ingo Schwarze)

whitespace and header sorting changes (Ingo Schwarze). No functional changes.

More header cleanups from Ingo Schwarze.

From Ingo Scharze:
Let "el.h" include everything needed for struct editline,
and don't include that stuff multiple times. That also improves
consistency, also avoids circular inclusions, and also makes it
easier to follow what is going on, even though not quite as nice.
But it seems like the best we can do...

cleanup chartype.h includes (Ingo Schwarze)

cleanup inclusion of histedit.h (Ingo Schwarze)

From Ingo Schwartze:

Next step: Remove #ifdef'ing in read_char(), in the same style
as we did for setlocale(3) in el.c.

A few remarks are required to explain the choices made.

* On first sight, handling mbrtowc(3) seems a bit less trivial
than handling setlocale(3) because its prototype uses the data
type mbstate_t from <wchar.h>. However, it turns out that
"histedit.h" already includes <wchar.h> unconditionally (i don't
like headers including other headers, but that ship has sailed,
people are by now certainly used to the fact that including
"histedit.h" doesn't require including <wchar.h> before), and
"histedit.h" is of course included all over the place. So from
that perspective, there is no problem with using mbrtowc(3)
unconditionally ever for !WIDECHAR.

* However, <wchar.h> also defines the mbrtowc(3) prototype,
so we cannot just #define mbrtowc away, or including the header
will break. It would also be a bad idea to porovide a local
implementation of mbrtowc() and hope that it overrides the one
in libc. Besides, the required prototype is subtly different:
While mbrtowc(3) takes "wchar_t *" as its first argument, we
need a function that takes "Char *". So unfortunately, we have
to keep a ct_mbrtowc #define, at least until we can maybe get
rid of "Char *" in the more remote future.

* After getting rid of the #else clause in read_char(), we can
pull "return 1;" into the default: clause. After that, we can
get rid of the ugly "goto again_lastbyte;" and just "break;".
As a bonus, that also gets rid of the ugly CONSTCOND.

* While here, delete the unused ct_mbtowc() from chartype.h.

GC IGNORE_EXTCHARS and simplify code (Ingo Schwarze)

From Ingo Schwarze:

If CHARSET_IS_UTF8 is not set, read_char() is broken in a large
number of ways:

1. The isascii(3) check can yield false positives. If a string in
an arbitrary encoding contains a byte in the range 0..127,
that does not at all imply that it forms a character all by
itself, and even less that it represents the same character
as in ASCII. Consequently, read_char() may return characters
the user never typed.
Even if the encoding is not state dependent, the assumption that
bytes in the range 0..127 represent ASCII characters is broken.
Consider UTF-16, for example.

2. The reverse problem can also occur. In an arbitrary encoding,
there is no guarantee that a character that can be represented
by ASCII is represented by a seven-bit byte, and even less by
the same byte as in ASCII.
Even for single-byte encodings, these assumptions are broken.
Consider the ISO 646 national variants, for example.
Consequently, the current code is insufficient to keep ASCII
characters working even for single-byte encodings.

3. The condition "++cbp != 1" can never trigger (because initially,
cbp is 0, and the code can only go back up via the final goto,
which has another cbp = 0 right before it) and it has no effect
(because cbp isn't used afterwards).

4. bytes = ct_mbtowc(cp, cbuf, cbp) is broken. If this returns -1,
the code assumes that is can just call mbtowc(3) again for later
input bytes. In some implementations, that may even be broken
for state-independent encodings, but trying again after mbtowc(3)
failure certainly produces completely erratic and meaningless
results in state-dependent encodings.

5. The assignment "*cp = (Char)(unsigned char)cbuf[0]" is
completely bogus. Even if the byte cbuf[0] represents a
character all by itself, which it usually will not, whether
or not the cast produces the desired result depends on the
internal representation of wchar_t in the C library, which
the application program can know nothing about. Even for ASCII
in the C/POSIX locale, an ASCII character other than '\0' ==
L'\0' == 0 need not have the same numeric value as a char and
as a wchar_t.

To summarize, this code only works if all of the following
conditions hold:

- The encoding is a single-byte encoding.
- ASCII is a subset of the encoding.
- The implementation of mbtowc(3) in the C library does not
require re-initialization after encoding errors.
- The implementation of wchar_t in the C library uses the
same numerical values as ASCII.

Otherwise, it silently produces wrong results.

The simplest way to fix this is to just use the same code as for
UTF-8 (right above). Of course, that causes functional changes
but that shouldn't matter since current behaviour is undefined.

The patch below provides the following improvements:

- It works for all stateless single-byte encodings, no matter
whether they are somehow related to ASCII, no matter how
mb[r]towc(3) are internally implemented, and no matter how
wchar_t is internally represented.
- Instead of producing unpredictable and definitely wrong
results for non-UTF-8 multibyte characters, it behaves in
a well-defined way: It aborts input processing, sets errno,
and returns failure.
Note that short of providing full support for arbitrary locales,
it is impossible to do better. We cannot know whether a given
unsupported locale is state-dependent, and for a state-dependent
locale, it makes no sense to retry parsing after an encoding
error, so the best we can do is abort processing for *any*
unsupported multi-byte character.
- Note that single-byte characters in arbitrary state-independent
locales still work, even in locales that may potentially also
contain multibyte characters, as long as those don't occur in
input. I'm not sure whether any such locales exist in practice...

Tested with UTF-8 and C/POSIX on OpenBSD. Also tested that in the
C/POSIX locale, non-ASCII bytes get through unmangled. You may
wish to test with ISO-LATIN on NetBSD if NetBSD supports that.

----
Also use a constant for meta to avoid warnings.

- Add some more Char casts
- reduce ifdefs by providing empty defs for nls functions (Ingo Schwarze)

Remove utf8_islead() mbrtowc() handles this just fine (Ingo Schwarze)

UTF-8 fixes from Ingo Schwarze:

1. Assume that errno is non-zero when entering read_char()
and that read(2) returns 0 (indicating end of file).
Then, the code will clear errno before returning.
(Obviously, the statement "errno = 0" is almost always
a bug unless there is save_errno = errno right before it
and the previous value is properly restored later,
in all reachable code paths.)

2. When encountering an invalid byte sequence, the code discards
all following bytes until MB_LEN_MAX overflows; consider, for
example, 0xc2 immediately followed by a few valid ASCII bytes.
Three of those ASCII bytes will be discarded.

3. On a POSIX system, EILSEQ will always be set after reading a
valid (yes, valid, not invalid!) UTF-8 character. The reason
is that mbtowc(3) will first be called with a length limit
(third argument) of 1, which will fail, return -1, and - on
a POSIX system - set errno to EILSEQ.
This third bug is mitigated a bit because i couldn't find any
system that actually conforms to POSIX in this respect: None
of OpenBSD, NetBSD, FreeBSD, Solaris 11, and glibc set errno
when an incomplete character is passed to mbtowc(3), even though
that is required by POSIX.
Anyway, that mbtowc(3) bug will be fixed at least in OpenBSD
after release unlock, so it would be good to fix this bug in
libedit before fixing the bug in mbtowc(3).

How can these three bugs be fixed?

1. As far as i understand it, the intention of the bogus errno = 0
is to undo the effects of failing system calls in el_wset(),
sig_set(), and read__fixio() if the subsequent read(2) indicates
end of file. So, restoring errno has to be moved right after
read__fixio(). Of course, neither 0 nor e is the right value
to restore: 0 is wrong if errno happened to be set on entry, e
would be wrong because if one read(2) fails but a second attempt
succeeds after read__fixio(), errno should not be touched. So,
the errno to be restored in this case has to be saved before
calling read(2) for the first time.

2. Solving the second issue requires distinguishing invalid and
incomplete characters, but that is impossible with the function
mbtowc(3) because it returns -1 in both cases and sets errno
to EILSEQ in both cases (once properly implemented).

It is vital that each input character is processed right away.
It is not acceptable to wait for the next input character before
processing the previous one because this is an interactive
library, not a batch system. Consequently, the only situation
where it is acceptable to wait for the next byte without first
processing the previous one(s) is when the previous one(s) form
an incomplete sequence that can be continued to form a valid
character.

Consequently, short of reimplementing a full UTF-8 state machine
by hand, the only correct way forward is to use mbrtowc(3).
Even then, care is needed to always have the state object
properly initialized before using it, and to not discard a valid
ASCII or UTF-8 lead byte if it happens to follow an invalid
sequence.

3. Fortunately, solution 2. also solves issue 3. as a side effect,
by no longer using mbtowc(3) in the first place.

Bounds search for reallocated index, from OpenBSD via Andreas Fett

branches: 1.70.4;
Test early for EOF to avoid infinite loop in the wide char case. From
Linas Vepstas: linasvepstas at gmail dot com

branches: 1.69.2;
return !OKCMD on error.

PR/46935: Steffen Nurpmeso: editline(3) (libedit): faulty errno handling,
faulty reuse of val in wrong context

branches: 1.67.2;
re-enable -Wconversion

pass -Wconversion

KNF return (\1); -> return \1;

kill ptr_t and ioctl_t, add * sizeof(*foo) to all allocations.

Rename key to keymacro to avoid conflicts with term.h. The renaming of term
to terminal was again to avoid conflicts with term.h. term.h is a moving
namespace violation.

term -> terminal
fix wide function confusion

make obvious what we are trying to do...

Try to fix build of port where "char" is unsigned.

Support other non-latin1 single byte character sets.
From: Alexander Barkov and Nirbhay Choubey at oracle dot com

PR/44599: Steven Vernon: libedit acts as if no data read if editmode is turned
off because nread is uninitialized in the edit disabled case.

branches: 1.57.2;
refresh only on SIGCONT not SIGWINCH from Edward Sheldrake

retry the read after sigwinch too, from Edward Sheldrake

https://bugzilla.redhat.com/show_bug.cgi?id=575383
Handle EINTR properly.

- Document and enable wide character support.
- Fix read function compatibility.

Wide character support (UTF-8) from Johny Mattsson; currently disabled.

Always initialize nread since it is an out param.
From Michael Cook mcook at bbn dot com

decrement the number of levels after the loop (Julien Torres)

from Julien Torres, flip the order we pop in the macro array.

make el_gets set the count to -1 on error to distinguish between EOF and
error.

remove VEOF test. the tty is in cooked mode when we are not editing and
the tty driver does the check for us.

use the VEOF character from the terminal, instead of hard-coding 4.

reset and redraw on sigcont. From Anon Ymous.

pass lint on _LP64.

don't restart on EINTR, instead return NULL immediately. From Anon Ymous

branches: 1.43.2;
match documentation in el_push

fix -Wsign-compare issues

Allow a single process to control multiple ttys (for pthreads using _REENTRANT)
using multiple EditLine objects. Mostly from Preston A. Elder.

branches: 1.40.12;
Fix bug with multiple pending el_pushes. Reported by Julien Torres.

On a fatal error, we want to stop processing the macro buffers.

Add missing second argument to another call of ch_reset().

Don't reset the macro strings each time we enter el_gets(), otherwise
el_push() is unusable programmatically.

Don't use non-standard uint or u_int.

Make sure we flush after we prepare when we are unbuffered otherwise the
prompt will not appear immediately.

PR/23486: Andreas Gustafsson: gdb no longer works with emacs
- make sure that we keep previous contents of the buffer in unbuffered mode.
- when turning editing on and off keep tty consistent.

branches: 1.33.2;
Better fix for rl_prep_terminal() from Gerry Swislow.

Separate tty separation from the prompt printing function. From Gerry Swislow.

portability fixes.

change allocation policy in el_push to allocate the string itself.
fix issues with strdup.

PR/23107: Nathan Williams: ^D as the first char on the command line does
not DTRT in readline compatibility mode

Implement enough of readline's 4.0 async mode to make gdb happy. This is
not complete yet, but it seems to work...
This required to introduce an unbuffered mode to el_gets(), but that was
a minor change.

Fix something that's been annoying me for a while...
Pull in <fcntl.h>, so that the fcntl #defines are present, and we build in the
automatic reset of non-blocking mode, rather than beeping like mad.

Move UCB-licensed code from 4-clause to 3-clause licence.

Patches provided by Joel Baker in PR 22280, verified by myself.

From michael@moria.de:
- use __attribute__((__unused__)) in arguments where appropriate.
- some int -> size_t and char * to const char * conversions.

Fix problem with previous patches that broke vi history.
- c_gets() was usually returning a length, but sometimes
one of the CC_xxx values (which are small +ve integers)!
- fixed c_gets() by putting a ' ' under the cursor.
From David Laight.

PR/18995: David Laight: libedit fixes for posix conformant sh

The posix 'sh' specification defines vi-mode editing quite tightly.
The netbsd libedit code (used by sh to do this) was missing several
features, there were also minor errors in others.

Compare netbsd sh to the definition available from:
http://www.opengroup.org/onlinepubs/007904975/utilities/sh.html
In particular the following were not supported:
U - undo all changes to line
| - goto column
Y - yank to end of line
y - yank
# - comment out current line
@ - take input from shell alias [1]
G - goto numbered line in history buffer
v - edit history line with vi
_ - append word from last input line
. - redo last command
Other minor changes have also been made.

[1] This needs the shell to define an appropriate routine to
return the text of the alias. There is no requirement that
such a function exist.

vi mode and memory fixes from david laight.

- constify; passes all gcc and lint strict checks.
- add config.h [Jason Evans], to create a portable version of libedit that
can be easily compiled on other OS's.

PR/14067: Anthony Mallet: Provide a programmatic way to set the read_char
function via a new el_set() operation.
Thanks, nicely done :-)

Enlarge editline buffers as needed to support arbitrary length lines.
This also addresses lib/9712 by Phil Nelson.

- add support for home and end keys.
- improve debugging support

convert to new style guide, which includes:
- ansi prototypes & features (such as stdargs)
- 8 space indents

PR/9243: Kevin Schoedel: libedit ignores repeat count

remove some lint

branches: 1.14.2;
minor tweak to previous fix: don't spuriously truncate final newline
under emacs.

Fix PR7685 (gdb under emacs prints spurious ^M and messes up terminal)
plus a few bogons noted along the way:
1) Set EDIT_DISABLED if terminal type is emacs.
2) fix bug in NO_TTY mode which caused it to not notice CR or LF
3) implement EDIT_DISABLED within libedit to be somewhat like NO_TTY,
except that a prompt is printed first.

More trailing white space.

In userland, pull in <errno.h> instead of <sys/errno.h> for the declaration
of errno.

delint

* implement CC_REFRESH_BEEP; as per CC_REFRESH but beep as well. this
is useful in completion when a partial completion is found
* remove entry in BUGS about el_parse(); that was fixed a while ago

fix compile errors if FIONREAD is defined. noted by
David Holland <dholland@bordeaux.eecs.harvard.edu> in [bin/4012].

Small optimization. Don't call isatty() on every invocation of el_gets, but
remember if the tty setup failed... Also trim the input line of trailing \r's.

Make el_gets() work when the input is not a tty.

Fix compiler warnings.

Portability fixes:
__const -> const
BADSIG -> SIG_ERR
int flags -> u_int flags
#if __STDC__ -> #ifdef __STDC__

Implement CC_REDISPLAY, which (unlink CC_REFRESH) redraws the entire input
line (a la ^R). This is useful if the binding outputs information and
mucks up the input line. To be used in ``list-choices'' bindings (refer
to the ^D binding in csh when filec is set)

RCSid police
editline first appeared in 4.4BSD not NetBSD1.0

branches: 1.1.1;
Initial revision

Disable attempts to handle EINTR and non-blocking I/O by default. It is
confusing to other programs and unexpected behavior. Reported by Ingo Schwarze.
This behavior is now controlled with EL_SAFEREAD.

PR/54399: S�ren Tempel: Uninitialized memory access in libedit history.
Initialize the buffer using calloc. While here change all malloc(a * sizeof(b))
to calloc(a, sizeof(b)). XXX: should fix realloc similarly.

From Yuichiro Naito (FreeBSD):

hrs@ says that
(cbp >= MB_LEN_MAX) condition is necessary for checking invalid
byte sequences. If malicious input was given, libedit would read
byte sequences forever.

Remove utf-8 requirement (Yuichiro NAITO)

branches: 1.103.4; 1.103.6;
remove unused variable

branches: 1.102.6;
PR/51706: Amir Plivatsky: Fix memory leak

branches: 1.101.2;
abstract read code to a single function (Ingo Schwarze)

el_map.alt can't be NULL here (Ingo Schwarze)

remove debug read (Ingo Schwarze)

From Ingo Schwarze:

Reduce obfuscation of errno handling. There is only one purpose
non-local errno handling is needed for: Inside el_wgets(), several
functions call down indirectly to el_wgetc(), many of them via the
dispatch table. When el_wgetc() fails, it does properly report
failure, but then various cleanup is done which may clobber errno.
But when returning due to failure, el_wgets() wants to have errno
set to the reason of the original read failure, not to the reason
of some subsequent failure of some cleanup operation. So el_wgetc()
needs to save errno, and if it's non-zero, el_wgets() needs to
restore it on failure.

This core logic is currently obscured by the fact that el_errno
is set and inspected at some additional places where it isn't needed.
Besides, since el_wgetc() and and el_wgets() are both in read.c,
el_errno does not need to be in struct editline, it can and should
be local to read.c in struct el_read_t.

Let's look at what can be simplified.

1. keymacro_get() abuses el_errno instead of having a proper
error return code. Adding that error return code is easy
because node_trav() already detects the condition and an
adequate code is already defined. Returning it, testing
for it in read_getcmd(), and returning with error from there
removes the need to inspect el_errno from el_wgets() after
calling read_getcmd().
Note that resetting lastchar and cursor and clearing buffer[0]
is irrelevant. The code returns from el_wgets() right afterwards.
Outside el_wgets(), these variables are no longer relevant.
When el_wgets() is called the next time, it will call ch_reset()
anyway, resetting the two pointers. And as long as lastchar
points to the beginning of the buffer, the contents of the
buffer won't be used for anything.

2. read_getcmd() doesn't need to set el_errno again after el_wgetc()
failure since el_wgetc() already did so. While here, remove
the silly "if EOF or error" comments from the el_wgetc()
return value tests. It's a public interface documented in a
manual, so people working on the implementation can obviously
be expected to know how it works. It's a case of

count++; /* Increment count. */

3. In the two code paths of el_wgets() that lead up to "goto noedit",
there is no need to save the errno because nothing that might
change it happens before returning.

For clarity, since el_wgets() is the function restoring the errno,
also move initializing it to the same function.

Finally, note that restoring errno when the saved value is zero is
wrong. No library code is ever allowed to clear a previously set
value of errno. Only application programs are allowed to do that,
and even they usually don't need to do so, except when using certain
ill-designed interfaces like strtol(3).

I tested that the behaviour remains sane in the following cases,
all during execution of el_wgets(3) and with a signal handler
for USR1 installed without SA_RESTART.

* Enter some text and maybe move around a bit.
Then send a USR1 signal.
The signal gets processed, then read_char() resumes reading.
Send another USR1 signal.
Now el_wgets() sets errno=EINTR and returns -1.

* Press Ctrl-V to activate ed-quoted-insert.
Then send a USR1 signal.
The signal gets processed, then read_char() resumes reading.
Send another USR1 signal.
ed_quoted_insert() returns ed_end_of_file(), i.e. CC_EOF,
and el_wgets() returns 0.

* Press a key starting a keyboard macro.
Then send a USR1 signal.
The signal gets processed, then read_char() resumes reading.
Send another USR1 signal.
Now el_wgets() sets errno=EINTR and returns -1.

* Press : to enter builtin command mode.
Start typing a command.
Then send a USR1 signal.
The signal gets processed, then read_char() resumes reading.
Send another USR1 signal.
Now c_gets() returns -1, ed_command() beeps and returns CC_REFRESH,
and el_wgets() resumes operation as it should.

I also tested with "el_set(el, EL_EDITMODE, 0)", and it returns
the right value and sets errno correctly.

Stop the read module from poking the el_chared.c_macro data structure
currently belonging to the chared module. The read module does so
from three of its functions, while no other module uses the macro
data, not even the chared module itself. That's quite logical
because macros are a feature of input handling, all of which is
done by the read module, and none by the chared module. So move
the data into the read modules's own opaque data structure, struct
el_read_t.

That simplifies internal interfaces in several respects: The
semi-public chared.h has one fewer struct, one fewer #define, and
one fewer member in struct el_chared_t; all three move to one single
C file, read.c, and are now module-local. And the internal interface
function ch_reset() needs one fewer argument, making the code of many
functions in various modules more readable.

The price is one additional internal interface function, read_end(),
10 lines long including comments, called publicly from exactly one
place: el_end() in el.c. That's hardly an increase in complexity
since most other modules already have their *_end() function, read.c
was the odd one out not having one.

From Ingo Schwarze

s/protected/libedit_private/g

From Ingo Schwarze:
- Put the data type el_rfunc_t into the public header <histedit.h>.
- Make el_read in struct editline an opaque pointer rather
than an embedded struct.
- Do not include "read.h" everywhere, but only in the two files
needing access to el_read, read.c and el.c.
- To functions that don't need more, pass the struct el_read_t *
rather than the full EditLine *.
- Of course, that means that read_init() can now fail from
memory exhaustion, but it's easy to clean up after that.

From Ingo Schwarze:
* Replace fcns.c by a shorter and simpler func.h
and include it only in the one file needing it, map.c.
* Combine help.h and help.c into a simplified help.h
and include it only in the one file needing it, map.c.
* Check the very simple, static files editline.c, historyn.c, and
tokenizern.c into CVS rather than needlessly generating them.
* So we no longer autogenerate any C files. :-)
* Shorten and simplify makelist by deleting the options -n, -e, -bc,
and -m; the latter was unused and useless in the first place.
* Move the declaration of el_func_t from fcns.h to the header
actually needing it, map.h. Since that header is already
included by el.h for unrelated reasons, that makes el_func_t
just as globally available as before.
* No longer include the simplified fcns.h into el.h,
include it directly into the *.c files needing it.

FIONREAD takes int as an argument (Ingo Schwarze)

From Ingo Schwarze:

* Delete the stubs of the XK_EXE mechanism that was never implemented.
From a security, stability, and simplicity perspective, i would
consider implementing it a truly terrible idea, so let's better
get rid of it.

* Do not use the local variable "num" in el_wgets() alternately for
two completely different purposes. Only use it for the number
of characters read, as stated in the comment (or -1 as long as
that number is still unknown), not for the (more or less boolean)
return value of read_getcmd(). Actually, there is no need at
all to save the latter return value after testing it once.

* The function read_getcmd() has very unusual return values:
It returns -1 for success and 0 for EOF/error. Switch that around
to 0 for success and -1 for EOF/error to be less confusing, and
get rid of the OKCMD preprocessor macro.

* Get rid of one #ifdef section in el_wgets() by using
el->el_chared.c_macro directly at the only place
where it is used.

* Delete the unused MIN() macro.

Get rid of private/public; keep protected (Ingo Schwarze)

Char -> wchar_t from Ingo Schwarze.

more macro WIDECHAR undoing from Ingo Schwarze.

More WIDECHAR elimination (Ingo Schwarze)

Start removing the WIDECHAR ifdefs; building without it has stopped working
anyway. (Ingo Schwarze)

PR/50880: David Binderman: Remove redundant code.
While here, fix all debugging formats.

Tuck in mbstate_t to the wide char version only to avoid exposing the zeroing
hack and doing it in the narrow case.

Make the read_char function always take a wchar_t * argument (Ingo Schwarze)

Get split el_getc and el_wgetc completely and call el_wgetc internally.
Change some character constants to they wide versions. (Ingo Schwarze)

whitespace and header sorting changes (Ingo Schwarze). No functional changes.

More header cleanups from Ingo Schwarze.

From Ingo Scharze:
Let "el.h" include everything needed for struct editline,
and don't include that stuff multiple times. That also improves
consistency, also avoids circular inclusions, and also makes it
easier to follow what is going on, even though not quite as nice.
But it seems like the best we can do...

cleanup chartype.h includes (Ingo Schwarze)

cleanup inclusion of histedit.h (Ingo Schwarze)

From Ingo Schwartze:

Next step: Remove #ifdef'ing in read_char(), in the same style
as we did for setlocale(3) in el.c.

A few remarks are required to explain the choices made.

* On first sight, handling mbrtowc(3) seems a bit less trivial
than handling setlocale(3) because its prototype uses the data
type mbstate_t from <wchar.h>. However, it turns out that
"histedit.h" already includes <wchar.h> unconditionally (i don't
like headers including other headers, but that ship has sailed,
people are by now certainly used to the fact that including
"histedit.h" doesn't require including <wchar.h> before), and
"histedit.h" is of course included all over the place. So from
that perspective, there is no problem with using mbrtowc(3)
unconditionally ever for !WIDECHAR.

* However, <wchar.h> also defines the mbrtowc(3) prototype,
so we cannot just #define mbrtowc away, or including the header
will break. It would also be a bad idea to porovide a local
implementation of mbrtowc() and hope that it overrides the one
in libc. Besides, the required prototype is subtly different:
While mbrtowc(3) takes "wchar_t *" as its first argument, we
need a function that takes "Char *". So unfortunately, we have
to keep a ct_mbrtowc #define, at least until we can maybe get
rid of "Char *" in the more remote future.

* After getting rid of the #else clause in read_char(), we can
pull "return 1;" into the default: clause. After that, we can
get rid of the ugly "goto again_lastbyte;" and just "break;".
As a bonus, that also gets rid of the ugly CONSTCOND.

* While here, delete the unused ct_mbtowc() from chartype.h.

GC IGNORE_EXTCHARS and simplify code (Ingo Schwarze)

From Ingo Schwarze:

If CHARSET_IS_UTF8 is not set, read_char() is broken in a large
number of ways:

1. The isascii(3) check can yield false positives. If a string in
an arbitrary encoding contains a byte in the range 0..127,
that does not at all imply that it forms a character all by
itself, and even less that it represents the same character
as in ASCII. Consequently, read_char() may return characters
the user never typed.
Even if the encoding is not state dependent, the assumption that
bytes in the range 0..127 represent ASCII characters is broken.
Consider UTF-16, for example.

2. The reverse problem can also occur. In an arbitrary encoding,
there is no guarantee that a character that can be represented
by ASCII is represented by a seven-bit byte, and even less by
the same byte as in ASCII.
Even for single-byte encodings, these assumptions are broken.
Consider the ISO 646 national variants, for example.
Consequently, the current code is insufficient to keep ASCII
characters working even for single-byte encodings.

3. The condition "++cbp != 1" can never trigger (because initially,
cbp is 0, and the code can only go back up via the final goto,
which has another cbp = 0 right before it) and it has no effect
(because cbp isn't used afterwards).

4. bytes = ct_mbtowc(cp, cbuf, cbp) is broken. If this returns -1,
the code assumes that is can just call mbtowc(3) again for later
input bytes. In some implementations, that may even be broken
for state-independent encodings, but trying again after mbtowc(3)
failure certainly produces completely erratic and meaningless
results in state-dependent encodings.

5. The assignment "*cp = (Char)(unsigned char)cbuf[0]" is
completely bogus. Even if the byte cbuf[0] represents a
character all by itself, which it usually will not, whether
or not the cast produces the desired result depends on the
internal representation of wchar_t in the C library, which
the application program can know nothing about. Even for ASCII
in the C/POSIX locale, an ASCII character other than '\0' ==
L'\0' == 0 need not have the same numeric value as a char and
as a wchar_t.

To summarize, this code only works if all of the following
conditions hold:

- The encoding is a single-byte encoding.
- ASCII is a subset of the encoding.
- The implementation of mbtowc(3) in the C library does not
require re-initialization after encoding errors.
- The implementation of wchar_t in the C library uses the
same numerical values as ASCII.

Otherwise, it silently produces wrong results.

The simplest way to fix this is to just use the same code as for
UTF-8 (right above). Of course, that causes functional changes
but that shouldn't matter since current behaviour is undefined.

The patch below provides the following improvements:

- It works for all stateless single-byte encodings, no matter
whether they are somehow related to ASCII, no matter how
mb[r]towc(3) are internally implemented, and no matter how
wchar_t is internally represented.
- Instead of producing unpredictable and definitely wrong
results for non-UTF-8 multibyte characters, it behaves in
a well-defined way: It aborts input processing, sets errno,
and returns failure.
Note that short of providing full support for arbitrary locales,
it is impossible to do better. We cannot know whether a given
unsupported locale is state-dependent, and for a state-dependent
locale, it makes no sense to retry parsing after an encoding
error, so the best we can do is abort processing for *any*
unsupported multi-byte character.
- Note that single-byte characters in arbitrary state-independent
locales still work, even in locales that may potentially also
contain multibyte characters, as long as those don't occur in
input. I'm not sure whether any such locales exist in practice...

Tested with UTF-8 and C/POSIX on OpenBSD. Also tested that in the
C/POSIX locale, non-ASCII bytes get through unmangled. You may
wish to test with ISO-LATIN on NetBSD if NetBSD supports that.

----
Also use a constant for meta to avoid warnings.

- Add some more Char casts
- reduce ifdefs by providing empty defs for nls functions (Ingo Schwarze)

Remove utf8_islead() mbrtowc() handles this just fine (Ingo Schwarze)

UTF-8 fixes from Ingo Schwarze:

1. Assume that errno is non-zero when entering read_char()
and that read(2) returns 0 (indicating end of file).
Then, the code will clear errno before returning.
(Obviously, the statement "errno = 0" is almost always
a bug unless there is save_errno = errno right before it
and the previous value is properly restored later,
in all reachable code paths.)

2. When encountering an invalid byte sequence, the code discards
all following bytes until MB_LEN_MAX overflows; consider, for
example, 0xc2 immediately followed by a few valid ASCII bytes.
Three of those ASCII bytes will be discarded.

3. On a POSIX system, EILSEQ will always be set after reading a
valid (yes, valid, not invalid!) UTF-8 character. The reason
is that mbtowc(3) will first be called with a length limit
(third argument) of 1, which will fail, return -1, and - on
a POSIX system - set errno to EILSEQ.
This third bug is mitigated a bit because i couldn't find any
system that actually conforms to POSIX in this respect: None
of OpenBSD, NetBSD, FreeBSD, Solaris 11, and glibc set errno
when an incomplete character is passed to mbtowc(3), even though
that is required by POSIX.
Anyway, that mbtowc(3) bug will be fixed at least in OpenBSD
after release unlock, so it would be good to fix this bug in
libedit before fixing the bug in mbtowc(3).

How can these three bugs be fixed?

1. As far as i understand it, the intention of the bogus errno = 0
is to undo the effects of failing system calls in el_wset(),
sig_set(), and read__fixio() if the subsequent read(2) indicates
end of file. So, restoring errno has to be moved right after
read__fixio(). Of course, neither 0 nor e is the right value
to restore: 0 is wrong if errno happened to be set on entry, e
would be wrong because if one read(2) fails but a second attempt
succeeds after read__fixio(), errno should not be touched. So,
the errno to be restored in this case has to be saved before
calling read(2) for the first time.

2. Solving the second issue requires distinguishing invalid and
incomplete characters, but that is impossible with the function
mbtowc(3) because it returns -1 in both cases and sets errno
to EILSEQ in both cases (once properly implemented).

It is vital that each input character is processed right away.
It is not acceptable to wait for the next input character before
processing the previous one because this is an interactive
library, not a batch system. Consequently, the only situation
where it is acceptable to wait for the next byte without first
processing the previous one(s) is when the previous one(s) form
an incomplete sequence that can be continued to form a valid
character.

Consequently, short of reimplementing a full UTF-8 state machine
by hand, the only correct way forward is to use mbrtowc(3).
Even then, care is needed to always have the state object
properly initialized before using it, and to not discard a valid
ASCII or UTF-8 lead byte if it happens to follow an invalid
sequence.

3. Fortunately, solution 2. also solves issue 3. as a side effect,
by no longer using mbtowc(3) in the first place.

Bounds search for reallocated index, from OpenBSD via Andreas Fett

branches: 1.70.4;
Test early for EOF to avoid infinite loop in the wide char case. From
Linas Vepstas: linasvepstas at gmail dot com

branches: 1.69.2;
return !OKCMD on error.

PR/46935: Steffen Nurpmeso: editline(3) (libedit): faulty errno handling,
faulty reuse of val in wrong context

branches: 1.67.2;
re-enable -Wconversion

pass -Wconversion

KNF return (\1); -> return \1;

kill ptr_t and ioctl_t, add * sizeof(*foo) to all allocations.

Rename key to keymacro to avoid conflicts with term.h. The renaming of term
to terminal was again to avoid conflicts with term.h. term.h is a moving
namespace violation.

term -> terminal
fix wide function confusion

make obvious what we are trying to do...

Try to fix build of port where "char" is unsigned.

Support other non-latin1 single byte character sets.
From: Alexander Barkov and Nirbhay Choubey at oracle dot com

PR/44599: Steven Vernon: libedit acts as if no data read if editmode is turned
off because nread is uninitialized in the edit disabled case.

branches: 1.57.2;
refresh only on SIGCONT not SIGWINCH from Edward Sheldrake

retry the read after sigwinch too, from Edward Sheldrake

https://bugzilla.redhat.com/show_bug.cgi?id=575383
Handle EINTR properly.

- Document and enable wide character support.
- Fix read function compatibility.

Wide character support (UTF-8) from Johny Mattsson; currently disabled.

Always initialize nread since it is an out param.
From Michael Cook mcook at bbn dot com

decrement the number of levels after the loop (Julien Torres)

from Julien Torres, flip the order we pop in the macro array.

make el_gets set the count to -1 on error to distinguish between EOF and
error.

remove VEOF test. the tty is in cooked mode when we are not editing and
the tty driver does the check for us.

use the VEOF character from the terminal, instead of hard-coding 4.

reset and redraw on sigcont. From Anon Ymous.

pass lint on _LP64.

don't restart on EINTR, instead return NULL immediately. From Anon Ymous

branches: 1.43.2;
match documentation in el_push

fix -Wsign-compare issues

Allow a single process to control multiple ttys (for pthreads using _REENTRANT)
using multiple EditLine objects. Mostly from Preston A. Elder.

branches: 1.40.12;
Fix bug with multiple pending el_pushes. Reported by Julien Torres.

On a fatal error, we want to stop processing the macro buffers.

Add missing second argument to another call of ch_reset().

Don't reset the macro strings each time we enter el_gets(), otherwise
el_push() is unusable programmatically.

Don't use non-standard uint or u_int.

Make sure we flush after we prepare when we are unbuffered otherwise the
prompt will not appear immediately.

PR/23486: Andreas Gustafsson: gdb no longer works with emacs
- make sure that we keep previous contents of the buffer in unbuffered mode.
- when turning editing on and off keep tty consistent.

branches: 1.33.2;
Better fix for rl_prep_terminal() from Gerry Swislow.

Separate tty separation from the prompt printing function. From Gerry Swislow.

portability fixes.

change allocation policy in el_push to allocate the string itself.
fix issues with strdup.

PR/23107: Nathan Williams: ^D as the first char on the command line does
not DTRT in readline compatibility mode

Implement enough of readline's 4.0 async mode to make gdb happy. This is
not complete yet, but it seems to work...
This required to introduce an unbuffered mode to el_gets(), but that was
a minor change.

Fix something that's been annoying me for a while...
Pull in <fcntl.h>, so that the fcntl #defines are present, and we build in the
automatic reset of non-blocking mode, rather than beeping like mad.

Move UCB-licensed code from 4-clause to 3-clause licence.

Patches provided by Joel Baker in PR 22280, verified by myself.

From michael@moria.de:
- use __attribute__((__unused__)) in arguments where appropriate.
- some int -> size_t and char * to const char * conversions.

Fix problem with previous patches that broke vi history.
- c_gets() was usually returning a length, but sometimes
one of the CC_xxx values (which are small +ve integers)!
- fixed c_gets() by putting a ' ' under the cursor.
From David Laight.

PR/18995: David Laight: libedit fixes for posix conformant sh

The posix 'sh' specification defines vi-mode editing quite tightly.
The netbsd libedit code (used by sh to do this) was missing several
features, there were also minor errors in others.

Compare netbsd sh to the definition available from:
http://www.opengroup.org/onlinepubs/007904975/utilities/sh.html
In particular the following were not supported:
U - undo all changes to line
| - goto column
Y - yank to end of line
y - yank
# - comment out current line
@ - take input from shell alias [1]
G - goto numbered line in history buffer
v - edit history line with vi
_ - append word from last input line
. - redo last command
Other minor changes have also been made.

[1] This needs the shell to define an appropriate routine to
return the text of the alias. There is no requirement that
such a function exist.

vi mode and memory fixes from david laight.

- constify; passes all gcc and lint strict checks.
- add config.h [Jason Evans], to create a portable version of libedit that
can be easily compiled on other OS's.

PR/14067: Anthony Mallet: Provide a programmatic way to set the read_char
function via a new el_set() operation.
Thanks, nicely done :-)

Enlarge editline buffers as needed to support arbitrary length lines.
This also addresses lib/9712 by Phil Nelson.

- add support for home and end keys.
- improve debugging support

convert to new style guide, which includes:
- ansi prototypes & features (such as stdargs)
- 8 space indents

PR/9243: Kevin Schoedel: libedit ignores repeat count

remove some lint

branches: 1.14.2;
minor tweak to previous fix: don't spuriously truncate final newline
under emacs.

Fix PR7685 (gdb under emacs prints spurious ^M and messes up terminal)
plus a few bogons noted along the way:
1) Set EDIT_DISABLED if terminal type is emacs.
2) fix bug in NO_TTY mode which caused it to not notice CR or LF
3) implement EDIT_DISABLED within libedit to be somewhat like NO_TTY,
except that a prompt is printed first.

More trailing white space.

In userland, pull in <errno.h> instead of <sys/errno.h> for the declaration
of errno.

delint

* implement CC_REFRESH_BEEP; as per CC_REFRESH but beep as well. this
is useful in completion when a partial completion is found
* remove entry in BUGS about el_parse(); that was fixed a while ago

fix compile errors if FIONREAD is defined. noted by
David Holland <dholland@bordeaux.eecs.harvard.edu> in [bin/4012].

Small optimization. Don't call isatty() on every invocation of el_gets, but
remember if the tty setup failed... Also trim the input line of trailing \r's.

Make el_gets() work when the input is not a tty.

Fix compiler warnings.

Portability fixes:
__const -> const
BADSIG -> SIG_ERR
int flags -> u_int flags
#if __STDC__ -> #ifdef __STDC__

Implement CC_REDISPLAY, which (unlink CC_REFRESH) redraws the entire input
line (a la ^R). This is useful if the binding outputs information and
mucks up the input line. To be used in ``list-choices'' bindings (refer
to the ^D binding in csh when filec is set)

RCSid police
editline first appeared in 4.4BSD not NetBSD1.0

branches: 1.1.1;
Initial revision

PR/54399: S�ren Tempel: Uninitialized memory access in libedit history.
Initialize the buffer using calloc. While here change all malloc(a * sizeof(b))
to calloc(a, sizeof(b)). XXX: should fix realloc similarly.

From Yuichiro Naito (FreeBSD):

hrs@ says that
(cbp >= MB_LEN_MAX) condition is necessary for checking invalid
byte sequences. If malicious input was given, libedit would read
byte sequences forever.

Remove utf-8 requirement (Yuichiro NAITO)

branches: 1.103.4; 1.103.6;
remove unused variable

branches: 1.102.6;
PR/51706: Amir Plivatsky: Fix memory leak

branches: 1.101.2;
abstract read code to a single function (Ingo Schwarze)

el_map.alt can't be NULL here (Ingo Schwarze)

remove debug read (Ingo Schwarze)

From Ingo Schwarze:

Reduce obfuscation of errno handling. There is only one purpose
non-local errno handling is needed for: Inside el_wgets(), several
functions call down indirectly to el_wgetc(), many of them via the
dispatch table. When el_wgetc() fails, it does properly report
failure, but then various cleanup is done which may clobber errno.
But when returning due to failure, el_wgets() wants to have errno
set to the reason of the original read failure, not to the reason
of some subsequent failure of some cleanup operation. So el_wgetc()
needs to save errno, and if it's non-zero, el_wgets() needs to
restore it on failure.

This core logic is currently obscured by the fact that el_errno
is set and inspected at some additional places where it isn't needed.
Besides, since el_wgetc() and and el_wgets() are both in read.c,
el_errno does not need to be in struct editline, it can and should
be local to read.c in struct el_read_t.

Let's look at what can be simplified.

1. keymacro_get() abuses el_errno instead of having a proper
error return code. Adding that error return code is easy
because node_trav() already detects the condition and an
adequate code is already defined. Returning it, testing
for it in read_getcmd(), and returning with error from there
removes the need to inspect el_errno from el_wgets() after
calling read_getcmd().
Note that resetting lastchar and cursor and clearing buffer[0]
is irrelevant. The code returns from el_wgets() right afterwards.
Outside el_wgets(), these variables are no longer relevant.
When el_wgets() is called the next time, it will call ch_reset()
anyway, resetting the two pointers. And as long as lastchar
points to the beginning of the buffer, the contents of the
buffer won't be used for anything.

2. read_getcmd() doesn't need to set el_errno again after el_wgetc()
failure since el_wgetc() already did so. While here, remove
the silly "if EOF or error" comments from the el_wgetc()
return value tests. It's a public interface documented in a
manual, so people working on the implementation can obviously
be expected to know how it works. It's a case of

count++; /* Increment count. */

3. In the two code paths of el_wgets() that lead up to "goto noedit",
there is no need to save the errno because nothing that might
change it happens before returning.

For clarity, since el_wgets() is the function restoring the errno,
also move initializing it to the same function.

Finally, note that restoring errno when the saved value is zero is
wrong. No library code is ever allowed to clear a previously set
value of errno. Only application programs are allowed to do that,
and even they usually don't need to do so, except when using certain
ill-designed interfaces like strtol(3).

I tested that the behaviour remains sane in the following cases,
all during execution of el_wgets(3) and with a signal handler
for USR1 installed without SA_RESTART.

* Enter some text and maybe move around a bit.
Then send a USR1 signal.
The signal gets processed, then read_char() resumes reading.
Send another USR1 signal.
Now el_wgets() sets errno=EINTR and returns -1.

* Press Ctrl-V to activate ed-quoted-insert.
Then send a USR1 signal.
The signal gets processed, then read_char() resumes reading.
Send another USR1 signal.
ed_quoted_insert() returns ed_end_of_file(), i.e. CC_EOF,
and el_wgets() returns 0.

* Press a key starting a keyboard macro.
Then send a USR1 signal.
The signal gets processed, then read_char() resumes reading.
Send another USR1 signal.
Now el_wgets() sets errno=EINTR and returns -1.

* Press : to enter builtin command mode.
Start typing a command.
Then send a USR1 signal.
The signal gets processed, then read_char() resumes reading.
Send another USR1 signal.
Now c_gets() returns -1, ed_command() beeps and returns CC_REFRESH,
and el_wgets() resumes operation as it should.

I also tested with "el_set(el, EL_EDITMODE, 0)", and it returns
the right value and sets errno correctly.

Stop the read module from poking the el_chared.c_macro data structure
currently belonging to the chared module. The read module does so
from three of its functions, while no other module uses the macro
data, not even the chared module itself. That's quite logical
because macros are a feature of input handling, all of which is
done by the read module, and none by the chared module. So move
the data into the read modules's own opaque data structure, struct
el_read_t.

That simplifies internal interfaces in several respects: The
semi-public chared.h has one fewer struct, one fewer #define, and
one fewer member in struct el_chared_t; all three move to one single
C file, read.c, and are now module-local. And the internal interface
function ch_reset() needs one fewer argument, making the code of many
functions in various modules more readable.

The price is one additional internal interface function, read_end(),
10 lines long including comments, called publicly from exactly one
place: el_end() in el.c. That's hardly an increase in complexity
since most other modules already have their *_end() function, read.c
was the odd one out not having one.

From Ingo Schwarze

s/protected/libedit_private/g

From Ingo Schwarze:
- Put the data type el_rfunc_t into the public header <histedit.h>.
- Make el_read in struct editline an opaque pointer rather
than an embedded struct.
- Do not include "read.h" everywhere, but only in the two files
needing access to el_read, read.c and el.c.
- To functions that don't need more, pass the struct el_read_t *
rather than the full EditLine *.
- Of course, that means that read_init() can now fail from
memory exhaustion, but it's easy to clean up after that.

From Ingo Schwarze:
* Replace fcns.c by a shorter and simpler func.h
and include it only in the one file needing it, map.c.
* Combine help.h and help.c into a simplified help.h
and include it only in the one file needing it, map.c.
* Check the very simple, static files editline.c, historyn.c, and
tokenizern.c into CVS rather than needlessly generating them.
* So we no longer autogenerate any C files. :-)
* Shorten and simplify makelist by deleting the options -n, -e, -bc,
and -m; the latter was unused and useless in the first place.
* Move the declaration of el_func_t from fcns.h to the header
actually needing it, map.h. Since that header is already
included by el.h for unrelated reasons, that makes el_func_t
just as globally available as before.
* No longer include the simplified fcns.h into el.h,
include it directly into the *.c files needing it.

FIONREAD takes int as an argument (Ingo Schwarze)

From Ingo Schwarze:

* Delete the stubs of the XK_EXE mechanism that was never implemented.
From a security, stability, and simplicity perspective, i would
consider implementing it a truly terrible idea, so let's better
get rid of it.

* Do not use the local variable "num" in el_wgets() alternately for
two completely different purposes. Only use it for the number
of characters read, as stated in the comment (or -1 as long as
that number is still unknown), not for the (more or less boolean)
return value of read_getcmd(). Actually, there is no need at
all to save the latter return value after testing it once.

* The function read_getcmd() has very unusual return values:
It returns -1 for success and 0 for EOF/error. Switch that around
to 0 for success and -1 for EOF/error to be less confusing, and
get rid of the OKCMD preprocessor macro.

* Get rid of one #ifdef section in el_wgets() by using
el->el_chared.c_macro directly at the only place
where it is used.

* Delete the unused MIN() macro.

Get rid of private/public; keep protected (Ingo Schwarze)

Char -> wchar_t from Ingo Schwarze.

more macro WIDECHAR undoing from Ingo Schwarze.

More WIDECHAR elimination (Ingo Schwarze)

Start removing the WIDECHAR ifdefs; building without it has stopped working
anyway. (Ingo Schwarze)

PR/50880: David Binderman: Remove redundant code.
While here, fix all debugging formats.

Tuck in mbstate_t to the wide char version only to avoid exposing the zeroing
hack and doing it in the narrow case.

Make the read_char function always take a wchar_t * argument (Ingo Schwarze)

Get split el_getc and el_wgetc completely and call el_wgetc internally.
Change some character constants to they wide versions. (Ingo Schwarze)

whitespace and header sorting changes (Ingo Schwarze). No functional changes.

More header cleanups from Ingo Schwarze.

From Ingo Scharze:
Let "el.h" include everything needed for struct editline,
and don't include that stuff multiple times. That also improves
consistency, also avoids circular inclusions, and also makes it
easier to follow what is going on, even though not quite as nice.
But it seems like the best we can do...

cleanup chartype.h includes (Ingo Schwarze)

cleanup inclusion of histedit.h (Ingo Schwarze)

From Ingo Schwartze:

Next step: Remove #ifdef'ing in read_char(), in the same style
as we did for setlocale(3) in el.c.

A few remarks are required to explain the choices made.

* On first sight, handling mbrtowc(3) seems a bit less trivial
than handling setlocale(3) because its prototype uses the data
type mbstate_t from <wchar.h>. However, it turns out that
"histedit.h" already includes <wchar.h> unconditionally (i don't
like headers including other headers, but that ship has sailed,
people are by now certainly used to the fact that including
"histedit.h" doesn't require including <wchar.h> before), and
"histedit.h" is of course included all over the place. So from
that perspective, there is no problem with using mbrtowc(3)
unconditionally ever for !WIDECHAR.

* However, <wchar.h> also defines the mbrtowc(3) prototype,
so we cannot just #define mbrtowc away, or including the header
will break. It would also be a bad idea to porovide a local
implementation of mbrtowc() and hope that it overrides the one
in libc. Besides, the required prototype is subtly different:
While mbrtowc(3) takes "wchar_t *" as its first argument, we
need a function that takes "Char *". So unfortunately, we have
to keep a ct_mbrtowc #define, at least until we can maybe get
rid of "Char *" in the more remote future.

* After getting rid of the #else clause in read_char(), we can
pull "return 1;" into the default: clause. After that, we can
get rid of the ugly "goto again_lastbyte;" and just "break;".
As a bonus, that also gets rid of the ugly CONSTCOND.

* While here, delete the unused ct_mbtowc() from chartype.h.

GC IGNORE_EXTCHARS and simplify code (Ingo Schwarze)

From Ingo Schwarze:

If CHARSET_IS_UTF8 is not set, read_char() is broken in a large
number of ways:

1. The isascii(3) check can yield false positives. If a string in
an arbitrary encoding contains a byte in the range 0..127,
that does not at all imply that it forms a character all by
itself, and even less that it represents the same character
as in ASCII. Consequently, read_char() may return characters
the user never typed.
Even if the encoding is not state dependent, the assumption that
bytes in the range 0..127 represent ASCII characters is broken.
Consider UTF-16, for example.

2. The reverse problem can also occur. In an arbitrary encoding,
there is no guarantee that a character that can be represented
by ASCII is represented by a seven-bit byte, and even less by
the same byte as in ASCII.
Even for single-byte encodings, these assumptions are broken.
Consider the ISO 646 national variants, for example.
Consequently, the current code is insufficient to keep ASCII
characters working even for single-byte encodings.

3. The condition "++cbp != 1" can never trigger (because initially,
cbp is 0, and the code can only go back up via the final goto,
which has another cbp = 0 right before it) and it has no effect
(because cbp isn't used afterwards).

4. bytes = ct_mbtowc(cp, cbuf, cbp) is broken. If this returns -1,
the code assumes that is can just call mbtowc(3) again for later
input bytes. In some implementations, that may even be broken
for state-independent encodings, but trying again after mbtowc(3)
failure certainly produces completely erratic and meaningless
results in state-dependent encodings.

5. The assignment "*cp = (Char)(unsigned char)cbuf[0]" is
completely bogus. Even if the byte cbuf[0] represents a
character all by itself, which it usually will not, whether
or not the cast produces the desired result depends on the
internal representation of wchar_t in the C library, which
the application program can know nothing about. Even for ASCII
in the C/POSIX locale, an ASCII character other than '\0' ==
L'\0' == 0 need not have the same numeric value as a char and
as a wchar_t.

To summarize, this code only works if all of the following
conditions hold:

- The encoding is a single-byte encoding.
- ASCII is a subset of the encoding.
- The implementation of mbtowc(3) in the C library does not
require re-initialization after encoding errors.
- The implementation of wchar_t in the C library uses the
same numerical values as ASCII.

Otherwise, it silently produces wrong results.

The simplest way to fix this is to just use the same code as for
UTF-8 (right above). Of course, that causes functional changes
but that shouldn't matter since current behaviour is undefined.

The patch below provides the following improvements:

- It works for all stateless single-byte encodings, no matter
whether they are somehow related to ASCII, no matter how
mb[r]towc(3) are internally implemented, and no matter how
wchar_t is internally represented.
- Instead of producing unpredictable and definitely wrong
results for non-UTF-8 multibyte characters, it behaves in
a well-defined way: It aborts input processing, sets errno,
and returns failure.
Note that short of providing full support for arbitrary locales,
it is impossible to do better. We cannot know whether a given
unsupported locale is state-dependent, and for a state-dependent
locale, it makes no sense to retry parsing after an encoding
error, so the best we can do is abort processing for *any*
unsupported multi-byte character.
- Note that single-byte characters in arbitrary state-independent
locales still work, even in locales that may potentially also
contain multibyte characters, as long as those don't occur in
input. I'm not sure whether any such locales exist in practice...

Tested with UTF-8 and C/POSIX on OpenBSD. Also tested that in the
C/POSIX locale, non-ASCII bytes get through unmangled. You may
wish to test with ISO-LATIN on NetBSD if NetBSD supports that.

----
Also use a constant for meta to avoid warnings.

- Add some more Char casts
- reduce ifdefs by providing empty defs for nls functions (Ingo Schwarze)

Remove utf8_islead() mbrtowc() handles this just fine (Ingo Schwarze)

UTF-8 fixes from Ingo Schwarze:

1. Assume that errno is non-zero when entering read_char()
and that read(2) returns 0 (indicating end of file).
Then, the code will clear errno before returning.
(Obviously, the statement "errno = 0" is almost always
a bug unless there is save_errno = errno right before it
and the previous value is properly restored later,
in all reachable code paths.)

2. When encountering an invalid byte sequence, the code discards
all following bytes until MB_LEN_MAX overflows; consider, for
example, 0xc2 immediately followed by a few valid ASCII bytes.
Three of those ASCII bytes will be discarded.

3. On a POSIX system, EILSEQ will always be set after reading a
valid (yes, valid, not invalid!) UTF-8 character. The reason
is that mbtowc(3) will first be called with a length limit
(third argument) of 1, which will fail, return -1, and - on
a POSIX system - set errno to EILSEQ.
This third bug is mitigated a bit because i couldn't find any
system that actually conforms to POSIX in this respect: None
of OpenBSD, NetBSD, FreeBSD, Solaris 11, and glibc set errno
when an incomplete character is passed to mbtowc(3), even though
that is required by POSIX.
Anyway, that mbtowc(3) bug will be fixed at least in OpenBSD
after release unlock, so it would be good to fix this bug in
libedit before fixing the bug in mbtowc(3).

How can these three bugs be fixed?

1. As far as i understand it, the intention of the bogus errno = 0
is to undo the effects of failing system calls in el_wset(),
sig_set(), and read__fixio() if the subsequent read(2) indicates
end of file. So, restoring errno has to be moved right after
read__fixio(). Of course, neither 0 nor e is the right value
to restore: 0 is wrong if errno happened to be set on entry, e
would be wrong because if one read(2) fails but a second attempt
succeeds after read__fixio(), errno should not be touched. So,
the errno to be restored in this case has to be saved before
calling read(2) for the first time.

2. Solving the second issue requires distinguishing invalid and
incomplete characters, but that is impossible with the function
mbtowc(3) because it returns -1 in both cases and sets errno
to EILSEQ in both cases (once properly implemented).

It is vital that each input character is processed right away.
It is not acceptable to wait for the next input character before
processing the previous one because this is an interactive
library, not a batch system. Consequently, the only situation
where it is acceptable to wait for the next byte without first
processing the previous one(s) is when the previous one(s) form
an incomplete sequence that can be continued to form a valid
character.

Consequently, short of reimplementing a full UTF-8 state machine
by hand, the only correct way forward is to use mbrtowc(3).
Even then, care is needed to always have the state object
properly initialized before using it, and to not discard a valid
ASCII or UTF-8 lead byte if it happens to follow an invalid
sequence.

3. Fortunately, solution 2. also solves issue 3. as a side effect,
by no longer using mbtowc(3) in the first place.

Bounds search for reallocated index, from OpenBSD via Andreas Fett

branches: 1.70.4;
Test early for EOF to avoid infinite loop in the wide char case. From
Linas Vepstas: linasvepstas at gmail dot com

branches: 1.69.2;
return !OKCMD on error.

PR/46935: Steffen Nurpmeso: editline(3) (libedit): faulty errno handling,
faulty reuse of val in wrong context

branches: 1.67.2;
re-enable -Wconversion

pass -Wconversion

KNF return (\1); -> return \1;

kill ptr_t and ioctl_t, add * sizeof(*foo) to all allocations.

Rename key to keymacro to avoid conflicts with term.h. The renaming of term
to terminal was again to avoid conflicts with term.h. term.h is a moving
namespace violation.

term -> terminal
fix wide function confusion

make obvious what we are trying to do...

Try to fix build of port where "char" is unsigned.

Support other non-latin1 single byte character sets.
From: Alexander Barkov and Nirbhay Choubey at oracle dot com

PR/44599: Steven Vernon: libedit acts as if no data read if editmode is turned
off because nread is uninitialized in the edit disabled case.

branches: 1.57.2;
refresh only on SIGCONT not SIGWINCH from Edward Sheldrake

retry the read after sigwinch too, from Edward Sheldrake

https://bugzilla.redhat.com/show_bug.cgi?id=575383
Handle EINTR properly.

- Document and enable wide character support.
- Fix read function compatibility.

Wide character support (UTF-8) from Johny Mattsson; currently disabled.

Always initialize nread since it is an out param.
From Michael Cook mcook at bbn dot com

decrement the number of levels after the loop (Julien Torres)

from Julien Torres, flip the order we pop in the macro array.

make el_gets set the count to -1 on error to distinguish between EOF and
error.

remove VEOF test. the tty is in cooked mode when we are not editing and
the tty driver does the check for us.

use the VEOF character from the terminal, instead of hard-coding 4.

reset and redraw on sigcont. From Anon Ymous.

pass lint on _LP64.

don't restart on EINTR, instead return NULL immediately. From Anon Ymous

branches: 1.43.2;
match documentation in el_push

fix -Wsign-compare issues

Allow a single process to control multiple ttys (for pthreads using _REENTRANT)
using multiple EditLine objects. Mostly from Preston A. Elder.

branches: 1.40.12;
Fix bug with multiple pending el_pushes. Reported by Julien Torres.

On a fatal error, we want to stop processing the macro buffers.

Add missing second argument to another call of ch_reset().

Don't reset the macro strings each time we enter el_gets(), otherwise
el_push() is unusable programmatically.

Don't use non-standard uint or u_int.

Make sure we flush after we prepare when we are unbuffered otherwise the
prompt will not appear immediately.

PR/23486: Andreas Gustafsson: gdb no longer works with emacs
- make sure that we keep previous contents of the buffer in unbuffered mode.
- when turning editing on and off keep tty consistent.

branches: 1.33.2;
Better fix for rl_prep_terminal() from Gerry Swislow.

Separate tty separation from the prompt printing function. From Gerry Swislow.

portability fixes.

change allocation policy in el_push to allocate the string itself.
fix issues with strdup.

PR/23107: Nathan Williams: ^D as the first char on the command line does
not DTRT in readline compatibility mode

Implement enough of readline's 4.0 async mode to make gdb happy. This is
not complete yet, but it seems to work...
This required to introduce an unbuffered mode to el_gets(), but that was
a minor change.

Fix something that's been annoying me for a while...
Pull in <fcntl.h>, so that the fcntl #defines are present, and we build in the
automatic reset of non-blocking mode, rather than beeping like mad.

Move UCB-licensed code from 4-clause to 3-clause licence.

Patches provided by Joel Baker in PR 22280, verified by myself.

From michael@moria.de:
- use __attribute__((__unused__)) in arguments where appropriate.
- some int -> size_t and char * to const char * conversions.

Fix problem with previous patches that broke vi history.
- c_gets() was usually returning a length, but sometimes
one of the CC_xxx values (which are small +ve integers)!
- fixed c_gets() by putting a ' ' under the cursor.
From David Laight.

PR/18995: David Laight: libedit fixes for posix conformant sh

The posix 'sh' specification defines vi-mode editing quite tightly.
The netbsd libedit code (used by sh to do this) was missing several
features, there were also minor errors in others.

Compare netbsd sh to the definition available from:
http://www.opengroup.org/onlinepubs/007904975/utilities/sh.html
In particular the following were not supported:
U - undo all changes to line
| - goto column
Y - yank to end of line
y - yank
# - comment out current line
@ - take input from shell alias [1]
G - goto numbered line in history buffer
v - edit history line with vi
_ - append word from last input line
. - redo last command
Other minor changes have also been made.

[1] This needs the shell to define an appropriate routine to
return the text of the alias. There is no requirement that
such a function exist.

vi mode and memory fixes from david laight.

- constify; passes all gcc and lint strict checks.
- add config.h [Jason Evans], to create a portable version of libedit that
can be easily compiled on other OS's.

PR/14067: Anthony Mallet: Provide a programmatic way to set the read_char
function via a new el_set() operation.
Thanks, nicely done :-)

Enlarge editline buffers as needed to support arbitrary length lines.
This also addresses lib/9712 by Phil Nelson.

- add support for home and end keys.
- improve debugging support

convert to new style guide, which includes:
- ansi prototypes & features (such as stdargs)
- 8 space indents

PR/9243: Kevin Schoedel: libedit ignores repeat count

remove some lint

branches: 1.14.2;
minor tweak to previous fix: don't spuriously truncate final newline
under emacs.

Fix PR7685 (gdb under emacs prints spurious ^M and messes up terminal)
plus a few bogons noted along the way:
1) Set EDIT_DISABLED if terminal type is emacs.
2) fix bug in NO_TTY mode which caused it to not notice CR or LF
3) implement EDIT_DISABLED within libedit to be somewhat like NO_TTY,
except that a prompt is printed first.

More trailing white space.

In userland, pull in <errno.h> instead of <sys/errno.h> for the declaration
of errno.

delint

* implement CC_REFRESH_BEEP; as per CC_REFRESH but beep as well. this
is useful in completion when a partial completion is found
* remove entry in BUGS about el_parse(); that was fixed a while ago

fix compile errors if FIONREAD is defined. noted by
David Holland <dholland@bordeaux.eecs.harvard.edu> in [bin/4012].

Small optimization. Don't call isatty() on every invocation of el_gets, but
remember if the tty setup failed... Also trim the input line of trailing \r's.

Make el_gets() work when the input is not a tty.

Fix compiler warnings.

Portability fixes:
__const -> const
BADSIG -> SIG_ERR
int flags -> u_int flags
#if __STDC__ -> #ifdef __STDC__

Implement CC_REDISPLAY, which (unlink CC_REFRESH) redraws the entire input
line (a la ^R). This is useful if the binding outputs information and
mucks up the input line. To be used in ``list-choices'' bindings (refer
to the ^D binding in csh when filec is set)

RCSid police
editline first appeared in 4.4BSD not NetBSD1.0

branches: 1.1.1;
Initial revision

From Yuichiro Naito (FreeBSD):

hrs@ says that
(cbp >= MB_LEN_MAX) condition is necessary for checking invalid
byte sequences. If malicious input was given, libedit would read
byte sequences forever.

Remove utf-8 requirement (Yuichiro NAITO)

branches: 1.103.4;
remove unused variable

branches: 1.102.6;
PR/51706: Amir Plivatsky: Fix memory leak

branches: 1.101.2;
abstract read code to a single function (Ingo Schwarze)

el_map.alt can't be NULL here (Ingo Schwarze)

remove debug read (Ingo Schwarze)

From Ingo Schwarze:

Reduce obfuscation of errno handling. There is only one purpose
non-local errno handling is needed for: Inside el_wgets(), several
functions call down indirectly to el_wgetc(), many of them via the
dispatch table. When el_wgetc() fails, it does properly report
failure, but then various cleanup is done which may clobber errno.
But when returning due to failure, el_wgets() wants to have errno
set to the reason of the original read failure, not to the reason
of some subsequent failure of some cleanup operation. So el_wgetc()
needs to save errno, and if it's non-zero, el_wgets() needs to
restore it on failure.

This core logic is currently obscured by the fact that el_errno
is set and inspected at some additional places where it isn't needed.
Besides, since el_wgetc() and and el_wgets() are both in read.c,
el_errno does not need to be in struct editline, it can and should
be local to read.c in struct el_read_t.

Let's look at what can be simplified.

1. keymacro_get() abuses el_errno instead of having a proper
error return code. Adding that error return code is easy
because node_trav() already detects the condition and an
adequate code is already defined. Returning it, testing
for it in read_getcmd(), and returning with error from there
removes the need to inspect el_errno from el_wgets() after
calling read_getcmd().
Note that resetting lastchar and cursor and clearing buffer[0]
is irrelevant. The code returns from el_wgets() right afterwards.
Outside el_wgets(), these variables are no longer relevant.
When el_wgets() is called the next time, it will call ch_reset()
anyway, resetting the two pointers. And as long as lastchar
points to the beginning of the buffer, the contents of the
buffer won't be used for anything.

2. read_getcmd() doesn't need to set el_errno again after el_wgetc()
failure since el_wgetc() already did so. While here, remove
the silly "if EOF or error" comments from the el_wgetc()
return value tests. It's a public interface documented in a
manual, so people working on the implementation can obviously
be expected to know how it works. It's a case of

count++; /* Increment count. */

3. In the two code paths of el_wgets() that lead up to "goto noedit",
there is no need to save the errno because nothing that might
change it happens before returning.

For clarity, since el_wgets() is the function restoring the errno,
also move initializing it to the same function.

Finally, note that restoring errno when the saved value is zero is
wrong. No library code is ever allowed to clear a previously set
value of errno. Only application programs are allowed to do that,
and even they usually don't need to do so, except when using certain
ill-designed interfaces like strtol(3).

I tested that the behaviour remains sane in the following cases,
all during execution of el_wgets(3) and with a signal handler
for USR1 installed without SA_RESTART.

* Enter some text and maybe move around a bit.
Then send a USR1 signal.
The signal gets processed, then read_char() resumes reading.
Send another USR1 signal.
Now el_wgets() sets errno=EINTR and returns -1.

* Press Ctrl-V to activate ed-quoted-insert.
Then send a USR1 signal.
The signal gets processed, then read_char() resumes reading.
Send another USR1 signal.
ed_quoted_insert() returns ed_end_of_file(), i.e. CC_EOF,
and el_wgets() returns 0.

* Press a key starting a keyboard macro.
Then send a USR1 signal.
The signal gets processed, then read_char() resumes reading.
Send another USR1 signal.
Now el_wgets() sets errno=EINTR and returns -1.

* Press : to enter builtin command mode.
Start typing a command.
Then send a USR1 signal.
The signal gets processed, then read_char() resumes reading.
Send another USR1 signal.
Now c_gets() returns -1, ed_command() beeps and returns CC_REFRESH,
and el_wgets() resumes operation as it should.

I also tested with "el_set(el, EL_EDITMODE, 0)", and it returns
the right value and sets errno correctly.

Stop the read module from poking the el_chared.c_macro data structure
currently belonging to the chared module. The read module does so
from three of its functions, while no other module uses the macro
data, not even the chared module itself. That's quite logical
because macros are a feature of input handling, all of which is
done by the read module, and none by the chared module. So move
the data into the read modules's own opaque data structure, struct
el_read_t.

That simplifies internal interfaces in several respects: The
semi-public chared.h has one fewer struct, one fewer #define, and
one fewer member in struct el_chared_t; all three move to one single
C file, read.c, and are now module-local. And the internal interface
function ch_reset() needs one fewer argument, making the code of many
functions in various modules more readable.

The price is one additional internal interface function, read_end(),
10 lines long including comments, called publicly from exactly one
place: el_end() in el.c. That's hardly an increase in complexity
since most other modules already have their *_end() function, read.c
was the odd one out not having one.

From Ingo Schwarze

s/protected/libedit_private/g

From Ingo Schwarze:
- Put the data type el_rfunc_t into the public header <histedit.h>.
- Make el_read in struct editline an opaque pointer rather
than an embedded struct.
- Do not include "read.h" everywhere, but only in the two files
needing access to el_read, read.c and el.c.
- To functions that don't need more, pass the struct el_read_t *
rather than the full EditLine *.
- Of course, that means that read_init() can now fail from
memory exhaustion, but it's easy to clean up after that.

From Ingo Schwarze:
* Replace fcns.c by a shorter and simpler func.h
and include it only in the one file needing it, map.c.
* Combine help.h and help.c into a simplified help.h
and include it only in the one file needing it, map.c.
* Check the very simple, static files editline.c, historyn.c, and
tokenizern.c into CVS rather than needlessly generating them.
* So we no longer autogenerate any C files. :-)
* Shorten and simplify makelist by deleting the options -n, -e, -bc,
and -m; the latter was unused and useless in the first place.
* Move the declaration of el_func_t from fcns.h to the header
actually needing it, map.h. Since that header is already
included by el.h for unrelated reasons, that makes el_func_t
just as globally available as before.
* No longer include the simplified fcns.h into el.h,
include it directly into the *.c files needing it.

FIONREAD takes int as an argument (Ingo Schwarze)

From Ingo Schwarze:

* Delete the stubs of the XK_EXE mechanism that was never implemented.
From a security, stability, and simplicity perspective, i would
consider implementing it a truly terrible idea, so let's better
get rid of it.

* Do not use the local variable "num" in el_wgets() alternately for
two completely different purposes. Only use it for the number
of characters read, as stated in the comment (or -1 as long as
that number is still unknown), not for the (more or less boolean)
return value of read_getcmd(). Actually, there is no need at
all to save the latter return value after testing it once.

* The function read_getcmd() has very unusual return values:
It returns -1 for success and 0 for EOF/error. Switch that around
to 0 for success and -1 for EOF/error to be less confusing, and
get rid of the OKCMD preprocessor macro.

* Get rid of one #ifdef section in el_wgets() by using
el->el_chared.c_macro directly at the only place
where it is used.

* Delete the unused MIN() macro.

Get rid of private/public; keep protected (Ingo Schwarze)

Char -> wchar_t from Ingo Schwarze.

more macro WIDECHAR undoing from Ingo Schwarze.

More WIDECHAR elimination (Ingo Schwarze)

Start removing the WIDECHAR ifdefs; building without it has stopped working
anyway. (Ingo Schwarze)

PR/50880: David Binderman: Remove redundant code.
While here, fix all debugging formats.

Tuck in mbstate_t to the wide char version only to avoid exposing the zeroing
hack and doing it in the narrow case.

Make the read_char function always take a wchar_t * argument (Ingo Schwarze)

Get split el_getc and el_wgetc completely and call el_wgetc internally.
Change some character constants to they wide versions. (Ingo Schwarze)

whitespace and header sorting changes (Ingo Schwarze). No functional changes.

More header cleanups from Ingo Schwarze.

From Ingo Scharze:
Let "el.h" include everything needed for struct editline,
and don't include that stuff multiple times. That also improves
consistency, also avoids circular inclusions, and also makes it
easier to follow what is going on, even though not quite as nice.
But it seems like the best we can do...

cleanup chartype.h includes (Ingo Schwarze)

cleanup inclusion of histedit.h (Ingo Schwarze)

From Ingo Schwartze:

Next step: Remove #ifdef'ing in read_char(), in the same style
as we did for setlocale(3) in el.c.

A few remarks are required to explain the choices made.

* On first sight, handling mbrtowc(3) seems a bit less trivial
than handling setlocale(3) because its prototype uses the data
type mbstate_t from <wchar.h>. However, it turns out that
"histedit.h" already includes <wchar.h> unconditionally (i don't
like headers including other headers, but that ship has sailed,
people are by now certainly used to the fact that including
"histedit.h" doesn't require including <wchar.h> before), and
"histedit.h" is of course included all over the place. So from
that perspective, there is no problem with using mbrtowc(3)
unconditionally ever for !WIDECHAR.

* However, <wchar.h> also defines the mbrtowc(3) prototype,
so we cannot just #define mbrtowc away, or including the header
will break. It would also be a bad idea to porovide a local
implementation of mbrtowc() and hope that it overrides the one
in libc. Besides, the required prototype is subtly different:
While mbrtowc(3) takes "wchar_t *" as its first argument, we
need a function that takes "Char *". So unfortunately, we have
to keep a ct_mbrtowc #define, at least until we can maybe get
rid of "Char *" in the more remote future.

* After getting rid of the #else clause in read_char(), we can
pull "return 1;" into the default: clause. After that, we can
get rid of the ugly "goto again_lastbyte;" and just "break;".
As a bonus, that also gets rid of the ugly CONSTCOND.

* While here, delete the unused ct_mbtowc() from chartype.h.

GC IGNORE_EXTCHARS and simplify code (Ingo Schwarze)

From Ingo Schwarze:

If CHARSET_IS_UTF8 is not set, read_char() is broken in a large
number of ways:

1. The isascii(3) check can yield false positives. If a string in
an arbitrary encoding contains a byte in the range 0..127,
that does not at all imply that it forms a character all by
itself, and even less that it represents the same character
as in ASCII. Consequently, read_char() may return characters
the user never typed.
Even if the encoding is not state dependent, the assumption that
bytes in the range 0..127 represent ASCII characters is broken.
Consider UTF-16, for example.

2. The reverse problem can also occur. In an arbitrary encoding,
there is no guarantee that a character that can be represented
by ASCII is represented by a seven-bit byte, and even less by
the same byte as in ASCII.
Even for single-byte encodings, these assumptions are broken.
Consider the ISO 646 national variants, for example.
Consequently, the current code is insufficient to keep ASCII
characters working even for single-byte encodings.

3. The condition "++cbp != 1" can never trigger (because initially,
cbp is 0, and the code can only go back up via the final goto,
which has another cbp = 0 right before it) and it has no effect
(because cbp isn't used afterwards).

4. bytes = ct_mbtowc(cp, cbuf, cbp) is broken. If this returns -1,
the code assumes that is can just call mbtowc(3) again for later
input bytes. In some implementations, that may even be broken
for state-independent encodings, but trying again after mbtowc(3)
failure certainly produces completely erratic and meaningless
results in state-dependent encodings.

5. The assignment "*cp = (Char)(unsigned char)cbuf[0]" is
completely bogus. Even if the byte cbuf[0] represents a
character all by itself, which it usually will not, whether
or not the cast produces the desired result depends on the
internal representation of wchar_t in the C library, which
the application program can know nothing about. Even for ASCII
in the C/POSIX locale, an ASCII character other than '\0' ==
L'\0' == 0 need not have the same numeric value as a char and
as a wchar_t.

To summarize, this code only works if all of the following
conditions hold:

- The encoding is a single-byte encoding.
- ASCII is a subset of the encoding.
- The implementation of mbtowc(3) in the C library does not
require re-initialization after encoding errors.
- The implementation of wchar_t in the C library uses the
same numerical values as ASCII.

Otherwise, it silently produces wrong results.

The simplest way to fix this is to just use the same code as for
UTF-8 (right above). Of course, that causes functional changes
but that shouldn't matter since current behaviour is undefined.

The patch below provides the following improvements:

- It works for all stateless single-byte encodings, no matter
whether they are somehow related to ASCII, no matter how
mb[r]towc(3) are internally implemented, and no matter how
wchar_t is internally represented.
- Instead of producing unpredictable and definitely wrong
results for non-UTF-8 multibyte characters, it behaves in
a well-defined way: It aborts input processing, sets errno,
and returns failure.
Note that short of providing full support for arbitrary locales,
it is impossible to do better. We cannot know whether a given
unsupported locale is state-dependent, and for a state-dependent
locale, it makes no sense to retry parsing after an encoding
error, so the best we can do is abort processing for *any*
unsupported multi-byte character.
- Note that single-byte characters in arbitrary state-independent
locales still work, even in locales that may potentially also
contain multibyte characters, as long as those don't occur in
input. I'm not sure whether any such locales exist in practice...

Tested with UTF-8 and C/POSIX on OpenBSD. Also tested that in the
C/POSIX locale, non-ASCII bytes get through unmangled. You may
wish to test with ISO-LATIN on NetBSD if NetBSD supports that.

----
Also use a constant for meta to avoid warnings.

- Add some more Char casts
- reduce ifdefs by providing empty defs for nls functions (Ingo Schwarze)

Remove utf8_islead() mbrtowc() handles this just fine (Ingo Schwarze)

UTF-8 fixes from Ingo Schwarze:

1. Assume that errno is non-zero when entering read_char()
and that read(2) returns 0 (indicating end of file).
Then, the code will clear errno before returning.
(Obviously, the statement "errno = 0" is almost always
a bug unless there is save_errno = errno right before it
and the previous value is properly restored later,
in all reachable code paths.)

2. When encountering an invalid byte sequence, the code discards
all following bytes until MB_LEN_MAX overflows; consider, for
example, 0xc2 immediately followed by a few valid ASCII bytes.
Three of those ASCII bytes will be discarded.

3. On a POSIX system, EILSEQ will always be set after reading a
valid (yes, valid, not invalid!) UTF-8 character. The reason
is that mbtowc(3) will first be called with a length limit
(third argument) of 1, which will fail, return -1, and - on
a POSIX system - set errno to EILSEQ.
This third bug is mitigated a bit because i couldn't find any
system that actually conforms to POSIX in this respect: None
of OpenBSD, NetBSD, FreeBSD, Solaris 11, and glibc set errno
when an incomplete character is passed to mbtowc(3), even though
that is required by POSIX.
Anyway, that mbtowc(3) bug will be fixed at least in OpenBSD
after release unlock, so it would be good to fix this bug in
libedit before fixing the bug in mbtowc(3).

How can these three bugs be fixed?

1. As far as i understand it, the intention of the bogus errno = 0
is to undo the effects of failing system calls in el_wset(),
sig_set(), and read__fixio() if the subsequent read(2) indicates
end of file. So, restoring errno has to be moved right after
read__fixio(). Of course, neither 0 nor e is the right value
to restore: 0 is wrong if errno happened to be set on entry, e
would be wrong because if one read(2) fails but a second attempt
succeeds after read__fixio(), errno should not be touched. So,
the errno to be restored in this case has to be saved before
calling read(2) for the first time.

2. Solving the second issue requires distinguishing invalid and
incomplete characters, but that is impossible with the function
mbtowc(3) because it returns -1 in both cases and sets errno
to EILSEQ in both cases (once properly implemented).

It is vital that each input character is processed right away.
It is not acceptable to wait for the next input character before
processing the previous one because this is an interactive
library, not a batch system. Consequently, the only situation
where it is acceptable to wait for the next byte without first
processing the previous one(s) is when the previous one(s) form
an incomplete sequence that can be continued to form a valid
character.

Consequently, short of reimplementing a full UTF-8 state machine
by hand, the only correct way forward is to use mbrtowc(3).
Even then, care is needed to always have the state object
properly initialized before using it, and to not discard a valid
ASCII or UTF-8 lead byte if it happens to follow an invalid
sequence.

3. Fortunately, solution 2. also solves issue 3. as a side effect,
by no longer using mbtowc(3) in the first place.

Bounds search for reallocated index, from OpenBSD via Andreas Fett

branches: 1.70.4;
Test early for EOF to avoid infinite loop in the wide char case. From
Linas Vepstas: linasvepstas at gmail dot com

branches: 1.69.2;
return !OKCMD on error.

PR/46935: Steffen Nurpmeso: editline(3) (libedit): faulty errno handling,
faulty reuse of val in wrong context

branches: 1.67.2;
re-enable -Wconversion

pass -Wconversion

KNF return (\1); -> return \1;

kill ptr_t and ioctl_t, add * sizeof(*foo) to all allocations.

Rename key to keymacro to avoid conflicts with term.h. The renaming of term
to terminal was again to avoid conflicts with term.h. term.h is a moving
namespace violation.

term -> terminal
fix wide function confusion

make obvious what we are trying to do...

Try to fix build of port where "char" is unsigned.

Support other non-latin1 single byte character sets.
From: Alexander Barkov and Nirbhay Choubey at oracle dot com

PR/44599: Steven Vernon: libedit acts as if no data read if editmode is turned
off because nread is uninitialized in the edit disabled case.

branches: 1.57.2;
refresh only on SIGCONT not SIGWINCH from Edward Sheldrake

retry the read after sigwinch too, from Edward Sheldrake

https://bugzilla.redhat.com/show_bug.cgi?id=575383
Handle EINTR properly.

- Document and enable wide character support.
- Fix read function compatibility.

Wide character support (UTF-8) from Johny Mattsson; currently disabled.

Always initialize nread since it is an out param.
From Michael Cook mcook at bbn dot com

decrement the number of levels after the loop (Julien Torres)

from Julien Torres, flip the order we pop in the macro array.

make el_gets set the count to -1 on error to distinguish between EOF and
error.

remove VEOF test. the tty is in cooked mode when we are not editing and
the tty driver does the check for us.

use the VEOF character from the terminal, instead of hard-coding 4.

reset and redraw on sigcont. From Anon Ymous.

pass lint on _LP64.

don't restart on EINTR, instead return NULL immediately. From Anon Ymous

branches: 1.43.2;
match documentation in el_push

fix -Wsign-compare issues

Allow a single process to control multiple ttys (for pthreads using _REENTRANT)
using multiple EditLine objects. Mostly from Preston A. Elder.

branches: 1.40.12;
Fix bug with multiple pending el_pushes. Reported by Julien Torres.

On a fatal error, we want to stop processing the macro buffers.

Add missing second argument to another call of ch_reset().

Don't reset the macro strings each time we enter el_gets(), otherwise
el_push() is unusable programmatically.