Remove $FreeBSD$: one-line .c pattern

Remove /^[\s*]*__FBSDID\("\$FreeBSD\$"\);?\s*\n/

Allow bootstrapping localdef on non-FreeBSD systems

The current localedef simply assumes that the locale headers on build system
are compatible with those on the target system which is not necessarily true.
It generally works on FreeBSD (as long as we don't change the locale headers),
but Linux and macOS provide completely different locale headers.

This change adds new bootstrap headers that namespace certain xlocale
structures defined or used by in the headers that localdef needs.
This is required since system headers *must* be able to include the "real"
locale headers for printf(), etc., but we also want to access the target
systems's internal locale structures.

Reviewed By: yuripv, brooks
Differential Revision: https://reviews.freebsd.org/D25229

localedef: define characters in "space" class also as "print", except
for the known conflicts ("control" characters can't be "print"able).
POSIX doesn't explicitly forbid this, and actually includes <space>
character in "print".

PR: 225692
Reviewed by: bapt, cem (previous version), pfg (previous version)
Approved by: kib (mentor)
Differential Revision: https://reviews.freebsd.org/D17467

Add -b/-l options to localedef(1) to specify output endianness and use
it appropriately when building share/ctypedef and share/colldef.

This makes the resulting locale data in EL->EB (amd64->powerpc64) cross
build and in the native EB build match. Revert the changes done to libc
in r308170 as they are no longer needed.

PR: 231965
Reviewed by: bapt, emaste, sbruno, 0mp
Approved by: kib (mentor)
Differential Revision: https://reviews.freebsd.org/D17603

userland: Fix several typos and minor errors

- duplicate words
- typos
- references to old versions of FreeBSD

Reviewed by: imp, benno

localedef(1): Add comment markings for license.

localedef(1): Fix mismatch.

Obtained from: illumos
X-MFC with: r314974

localedef(1): Fix small coverity issues.

- Operands don't affect result (CONSTANT_EXPRESSION_RESULT)
- Buffer not null terminated (BUFFER_SIZE_WARNING)

CID: 1338557, 1338565

Obtained from: illumos
MFC after: 5 days

Revert r314969, r314961:

The localdef(1) changes are breaking world:

00:18:40.750 /usr/src/share/colldef/af_ZA.UTF-8.src: 2421: error: Bad file
descriptor

I will fix them offline.

Reported by: lwshu and many others

localedef(1): Fix for memory leaks reported by coverity.

Also some small cleanups to match better current illumos.

CID: 1338540, 1338541, 1338557, 1338566

Obtained from: illumos
Discussed with: Yuri Pankov (@Nexenta)
MFC after: 5 days

localedef: Fix ctype dump (fixed wide spread errors)

This commit is from John Marino in dragonfly with the following commit log:

====
This was a CTYPE encoding error involving consecutive points of the same
ctype. It was reported by myself to Illumos over a year ago but I was
unsure if it was only happening on BSD. Given the cause, the bug is also
present on Illumos.

Basically, if consecutive points were of the exact same ctype, they would
be defined as a range regardless. For example, all of these would be
considered equivalent:

<A> ... <C>, <H> (converts to <A> .. <H>)
<A>, <B>, <H> (converts to <A> .. <H>)
<A>, <J> ... <H> (converts to <A> .. <H>)

So all the points that shouldn't have been defined got "bridged" by the
extreme points.

The effects were recently reported to FreeBSD on PR 213013. There are
countless places were the ctype flags are misdefined, so this is a major
fix that has to be MFC'd.
====

This reveals a bad change I did on the testsuite: while 0x07FF is a valid
unicode it is not used yet (reserved for future use)

PR: 213013
Submitted by: marino@
Reported by: Kurtis Rader <krader@skepticism.us>
Obtained from: Dragonfly
MFC after: 1 month

localedef(1): minor sorting to match Illumos.

Illumos recently included space in 'print' class. We already had
this but the code had slight sorting differences. Move it some
lines up to reduce diffs with Illumos.

No functional change.

Reference:
https://illumos.org/issues/5227

Fix build of localedef(1) on arm where wchar_t is an unsigned int

Actually only T_ISDIGIT should be flagged as _E4

With regard to ctype, digits (e.g. 0 to 9) and xdigits (the 0 to 9 portion
of hexidecimal numbers) are all considered "numbers". (Note that while
all digits are numbers, not all numbers are digits).

Enhance localedef to automatically set the "number" characteristic when
it encounters a digit or xdigit definition. This fixes malfunctionning
isalnum(3)

Obtained from: DragonflyBSD

eliminate need for "print" definition

By having space automatically classified as "print" type, we can
eliminate the print section from ctype src files completely (they
are just "graph" plus "<space>".

Obtained from: Dragonfly

Commit log from Dragonfly:

FreeBSD extended ctypes to include numbers (e.g. isnumber()) but never
actually implemented it. The isnumber() function was equivalent to the
isdigit() function in every case.

Now that DragonFly's ctype source files have number definitions, the
number ctype can finally be implemented. It's given a new flag _CTYPE_N.
The isalnum() and iswalnum() functions have been changed to use this
flag rather than the _CTYPE_D digit flag.

While isalnum(), isnumber(), and their wide equivalents now return
different values in locale cases, the ishexnumber() and iswhexnumber()
functions are unchanged. They are still aliases for isxdigit() and
iswxdigit().

Also change ctype.h for isdigit and isxdigit to use sbistype like the
other functions.

Obtained from: dragonfly

Convert localedef(1) from avl to RB trees

Prefer static generation of functions

Convert ctype generation to Red Black tree

Add localedef(1), a locale definition generator tool

The localedef tool can read entire (and unmodified) CLDR posix definition
files, and generate all 6 LC categories: LC_COLLATE, LC_CTYPE, LC_TIME,
LC_NUMERIC, LC_MONETARY and LC_MESSAGES.

This tool has a long history with Solaris. The Nexenta developers
modified it to read CLDR files and created the much richer collation
formats. The libc collation functions have to be modified to read the
new format (called "BSD-1.0") and to handle the new data structures.

The result will be that locale-sensitive tools and functions will now
properly sort multibyte and unicode strings.

Obtained from: Dragonfly