kern: tty: recanonicalize the buffer on ICANON/VEOF/VEOL changes

Before this change, we would canonicalize any partial input if the new
local mode is not ICANON, but that's about it. If we were switching
from -ICANON -> ICANON, or if VEOF/VEOL changes, then our internal canon
accounting would be wrong.

The main consequence of this is that in ICANON mode, we would
potentially hang a read(2) longer if the new VEOF/VEOL appears later in
the buffer, and FIONREAD would be similarly wrong as a result.

Reviewed by: kib
Differential Revision: https://reviews.freebsd.org/D43456

kern: tty: fix EOF handling for canonical reads

If the read(2) buffer is one byte short of an EOF, then we'll end up
reading the line into the buffer, then re-entering and seeing an EOF at
the beginning of the inq, assuming it's a zero-length line.

Fix this corner-case by searching one more byte than we have available
for an EOF. If we found it, then we'll trim it here; otherwise, we'll
limit our read to just the space we have in the out buffer and the next
read(2) will (potentially) read the remainder of the line.

Fix FIONREAD while we're here to match what an application can expect
read(2) to return -- scan for the first break character in the part of
the input that's been canonicalized, we'll never return more than that.

PR: 276220
Reviewed by: cy, imp (both previous version), kib
Differential Revision: https://reviews.freebsd.org/D43378

sys: Automated cleanup of cdefs and other formatting

Apply the following automated changes to try to eliminate
no-longer-needed sys/cdefs.h includes as well as now-empty
blank lines in a row.

Remove /^#if.*\n#endif.*\n#include\s+<sys/cdefs.h>.*\n/
Remove /\n+#include\s+<sys/cdefs.h>.*\n+#if.*\n#endif.*\n+/
Remove /\n+#if.*\n#endif.*\n+/
Remove /^#if.*\n#endif.*\n/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/types.h>/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/param.h>/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/capsicum.h>/

Sponsored by: Netflix

tty: properly check character position when handling IUTF8 backspaces

The tty_rubchar() code handling backspaces for UTF-8 characters didn't
properly check whether the beginning of the current line was reached.
This resulted in a kernel panic in ttyinq_unputchar() when prodded with
certain malformed UTF-8 sequences.

Fixes: PR 275009
Reviewed by: christos
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D42564

tty/teken: fix UTF8 sequence validation logic

This patch fixes UTF-8 sequence validation logic in
teken_utf8_bytes_to_codepoint() and fixes fallback behaviour in
ttydisc_rubchar() when an invalid UTF8 sequence is encountered. The code
previously used __bitcount() to extract sequence length information from
the leading byte. However, this assumption breaks for certain code
points that have additional bits set in the first half of the leading
byte (e.g. Cyrillic characters). This lead to incorrect behaviour when
deleting those characters using backspaces. The code now checks the
number of consecutive set bits in the leading byte starting from the
MSB, as per RFC 3629.

Reviewed by: christos
MFC after: 2 weeks
Differential Revision: https://reviews.freebsd.org/D42147

tty: fix improper backspace behaviour for UTF8 characters when in canonical mode

This patch adds additional logic in ttydisc_rubchar() to properly handle
backspace behaviour for UTF-8 characters.

Currently, typing in a backspace after a UTF8 character will delete only
one byte from the byte sequence, leaving garbled output in the tty's
output queue. With this change all of the character's bytes are deleted.
This change is only active when the IUTF8 flag is set (see
19054eb6053189144aa962b2ecc1bf5087758a3e "(s)tty: add support for IUTF8
input flag")

The code uses the teken_wcwidth() function to properly handle character
column widths for different code points, and adds the
teken_utf8_bytes_to_codepoint() function that converts a UTF-8 byte
sequence to a codepoint, as specified in RFC3629.

Reported by: christos
Reviewed by: christos, imp
MFC after: 2 weeks
Differential Revision: https://reviews.freebsd.org/D42067

sys: Remove $FreeBSD$: one-line .c pattern

Remove /^[\s*]*__FBSDID\("\$FreeBSD\$"\);?\s*\n/

spdx: The BSD-2-Clause-FreeBSD identifier is obsolete, drop -FreeBSD

The SPDX folks have obsoleted the BSD-2-Clause-FreeBSD identifier. Catch
up to that fact and revert to their recommended match of BSD-2-Clause.

Discussed with: pfg
MFC After: 3 days
Sponsored by: Netflix

Implement FLUSHO

Turn FLUSHO on/off with ^O (or whatever VDISCARD is). Honor that to
throw away output quickly. This tries to remain true to 4.4BSD
behavior (since that was the origin of this feature), with any
corrections NetBSD has done. Since the implemenations are a little
different, though, some edge conditions may be handled differently.

Reviewed by: kib, kevans
Differential Review: https://reviews.freebsd.org/D26148

tty: convert tty_lock_assert to tty_assert_locked to hide lock type

A later change, currently being iterated on in D24459, will in-fact change
the lock type to an sx so that TTY drivers can sleep on it if they need to.
Committing this ahead of time to make the review in question a little more
palatable.

tty_lock_assert() is unfortunately still needed for now in two places to
make sure that the tty lock has not been recursed upon, for those scenarios
where it's supplied by the TTY driver and possibly a mutex that is allowed
to recurse.

Suggested by: markj

Replace ttyprintf with sbuf_printf and tty drain routine

Add string variants of cnputc and tty_putchar, and use them from the tty
sbuf drain routine.

Suggested by: ed@
Sponsored by: Dell EMC Isilon

sys/kern: adoption of SPDX licensing ID tags.

Mainly focus on files that use BSD 2-Clause license, however the tool I
was using misidentified many licenses so this was mostly a manual - error
prone - task.

The Software Package Data Exchange (SPDX) group provides a specification
to make it easier for automated tools to detect and summarize well known
opensource licenses. We are gradually adopting the specification, noting
that the tags are considered only advisory and do not, in any way,
superceed or replace the license texts.

Don't clear the software flow control flag before draining for last
close or assert the bug that it is clear when leaving.

Remove an unrelated rotted comment that was attached to the buggy
clearing.

Since draining is not done in more cases, flushing is needed in more
cases, so start fixing flushing:
- do a full flush in ttydisc_close(). State what POSIX requires more
clearly. This was missing ttydevsw_pktnotify() calls to tell the
devsw layer to flush. Hardware tty drivers don't actually flush
since they don't understand this API.
- fix 2 missing wakeups in tty_flush(). Most of the wakeups here are
unnecessary for last close. But ttydisc_close() did one of the
missing ones.

This flow control bug ameliorated the design bug of requiring
potentially unbounded waits in draining. Software flow control is the
easiest way to get an unbounded wait, and a long wait is sometimes
actually useful. Users can type the xoff character on the receiver
and (if ixon is set on the sender) expect the output to be held until
the user is ready for more.

Hardware flow control can also give the unbounded wait, and this bug
didn't affect hardware flow control. Unbounded waits from hardware
flow control take a more unusual configuration. E.g., a terminal
program that controls the modem status lines, or unplugging the cable
in a configuration where this doesn't break the connection.

The design bug is still ameliorated by a newer bug in draining for
last close -- the 1 second timeout. E.g., if the user types the
xoff character and the sender reaches last close, then output is
not resumed and the wait times out after just 1 second. This is
broken, but preferable to an unbounded wait. Before this change,
the output was resumed immediately and usually completed.

Submitted by: bde
MFC after: 2 weeks

Correct SIGTTIN handling.

In the old TTY layer, SIGTTIN was correctly handled like this:

while (data should be read) {
send SIGTTIN if not foreground process group
read data
}

In the new TTY layer, however, this behaviour was changed, based on a
false interpretation of the standard:

send SIGTTIN if not foreground process group
while (data should be read) {
read data
}

Correct this by pushing tty_wait_background() into the ttydisc_read_*()
functions.

Reported by: koitsu
PR: kern/173010
MFC after: 2 weeks

Fix faulty error code handling in read(2) on TTYs.

When performing a non-blocking read(2), on a TTY while no data is
available, we should return EAGAIN. But if there's a modem disconnect,
we should return 0. Right now we only return 0 when doing a blocking
read, which is wrong.

MFC after: 1 month

Fix found places where uio_resid is truncated to int.

Add the sysctl debug.iosize_max_clamp, enabled by default. Setting the
sysctl to zero allows to perform the SSIZE_MAX-sized i/o requests from
the usermode.

Discussed with: bde, das (previous versions)
MFC after: 1 month

Fix whitespace inconsistencies in TTY code.

Fix whitespace inconsistencies in the TTY layer and its drivers owned by me.

A new jail(8) with a configuration file, to replace the work currently done
by /etc/rc.d/jail.

MFC r198185:

Print backspaces after echoing an EOF.

Applications like shells expect EOF to give no graphical output, while
our implementation prints ^D by default (tunable with stty echoctl).
Make the new implementation behave like the old TTY code. Print two
backspaces afterwards.

I totally forgot to MFC this, because the 8.0 freeze took a little
longer than I expected.

Reminded by: koitsu

Print backspaces after echoing an EOF.

Applications like shells expect EOF to give no graphical output, while
our implementation prints ^D by default (tunable with stty echoctl).
Make the new implementation behave like the old TTY code. Print two
backspaces afterwards.

Reported by: koitsu
MFC after: 1 month

Add ttydisc_rint_simple().

I noticed several drivers in our tree don't actually care about parity
and framing, such as pts(4), snp(4) (and my partially finished console
driver). Instead of duplicating a lot of code, I think we'd better add a
utility function for those drivers to quickly process a buffer of input.

Also change pts(4) and snp(4) to use this function.

Fix two bugs related to TTY input:

- fix write() on pseudo-terminal masters to return the amount of bytes
passed to the TTY, not the amount of bytes read from user.

- fix ttydisc_rint_bypass() to set the high watermark when it cannot
write all input, just like ttydisc_rint() itself.

Approved by: re (kib)

If we have a regular rint handler, never go into rint_bypass mode.

It turns out if we called cfmakeraw() on a TTY with only a rint handler
in place, it could inject data into the TTY, even though it should be
redirected. Always take a look at the hooks before looking at the
termios flags.

Use unsigned longs for the TTY's sysctl stats.

Spotted by: clang

Don't forget to relock the TTY after uiomove() returns an error.

Peter Holm just discovered this funny bug inside the TTY code: if
uiomove() in ttydisc_write() returns an error, we forget to relock the
TTY before jumping out of ttydisc_write(). Fix it by placing
tty_unlock() and tty_lock() around uiomove().

Submitted by: pho

- Import the HEAD csup code which is the basis for the cvsmode work.

Introduce a hooks layer for the MPSAFE TTY layer.

One of the features that prevented us from fixing some of the TTY
consumers to work once again, was an interface that allowed consumers to
do the following:

- `Sniff' incoming data, which is used by the snp(4) driver.

- Take direct control of the input and output paths of a TTY, which is
used by ng_tty(4), ppp(4), sl(4), etc.

There's no practical advantage in committing a hooks layer without
having any consumers. In P4 there is a preliminary port of snp(4) and
thompsa@ is busy porting ng_tty(4) to this interface. I already want to
have it in the tree, because this may stimulate others to work on the
remaining modules.

Discussed with: thompsa
Obtained from: //depot/projects/mpsafetty/...

Fix style(9) issue in TTY header files: document function argument names.

According to style(9), function argument names should only be omitted
for prototypes that are exported to userspace. This means we should
document the function arguments in the TTY header files, because they
are only used in userspace.

While there, change the type of the buffer argument of
ttydisc_rint_bypass() to `const void *' instead of `char *'.

Requested by: attilio
Obtained from: //depot/projects/mpsafetty/...

Make sure the TTY has not disappeared out from under us before calling
ttydevsw_outwakeup(). This should fix panics which occur after remote
login sessions timeout during moderate TTY activity. An example of
where this might occur is where a pending write to the terminal is
occurring while sshd(8) is shutting down the TTY after a TCP timeout.

Submitted by: ed

Use size_t to store the return value of ttydisc_getc().

The ttydisc_getc() routine obtains a read length from ttyoutq_read().
For no valid reason, the current code stores this value in an int, and
returns a size_t. There is no need to perform this useless conversion.

Obtained from: //depot/projects/mpsafetty/...

Backport two small fixes from the MPSAFE TTY branch in Perforce:

- Implement IMAXBEL. It turned out the IMAXBEL termios switch was marked
as supported, while it had not been implemented.

- Don't go into the high watermark when in canonical mode, no data has
been canonicalized and the input buffer is full. This caused the
terminal to lock up. This prevented users from pressing
backspace/^U/etc in such cases.

This could easily be simulated by pasting a very big amount of data in
a shell with sh(1) in canonical mode.

Obtained from: //depot/projects/mpsafetty/...

Integrate the new MPSAFE TTY layer to the FreeBSD operating system.

The last half year I've been working on a replacement TTY layer for the
FreeBSD kernel. The new TTY layer was designed to improve the following:

- Improved driver model:

The old TTY layer has a driver model that is not abstract enough to
make it friendly to use. A good example is the output path, where the
device drivers directly access the output buffers. This means that an
in-kernel PPP implementation must always convert network buffers into
TTY buffers.

If a PPP implementation would be built on top of the new TTY layer
(still needs a hooks layer, though), it would allow the PPP
implementation to directly hand the data to the TTY driver.

- Improved hotplugging:

With the old TTY layer, it isn't entirely safe to destroy TTY's from
the system. This implementation has a two-step destructing design,
where the driver first abandons the TTY. After all threads have left
the TTY, the TTY layer calls a routine in the driver, which can be
used to free resources (unit numbers, etc).

The pts(4) driver also implements this feature, which means
posix_openpt() will now return PTY's that are created on the fly.

- Improved performance:

One of the major improvements is the per-TTY mutex, which is expected
to improve scalability when compared to the old Giant locking.
Another change is the unbuffered copying to userspace, which is both
used on TTY device nodes and PTY masters.

Upgrading should be quite straightforward. Unlike previous versions,
existing kernel configuration files do not need to be changed, except
when they reference device drivers that are listed in UPDATING.

Obtained from: //depot/projects/mpsafetty/...
Approved by: philip (ex-mentor)
Discussed: on the lists, at BSDCan, at the DevSummit
Sponsored by: Snow B.V., the Netherlands
dcons(4) fixed by: kan