Merge r308330 by bapt:

localedef: Fix ctype dump (fixed wide spread errors)

See original commit for longer description.

Errata Notice: EN-16:21
Approved by: so

Copy stable/11@r303970 to releng/11.0 as part of the 11.0-RELEASE
cycle.

Prune svn:mergeinfo from the new branch, and rename it to RC1.

Update __FreeBSD_version.

Use the quarterly branch for the default FreeBSD.conf pkg(8) repo and
the dvd1.iso packages population.

Approved by: re (implicit)
Sponsored by: The FreeBSD Foundation

Copy head@r302406 to stable/11 as part of the 11.0-RELEASE cycle.
Prune svn:mergeinfo from the new branch, as nothing has been merged
here.

Additional commits post-branch will follow.

Approved by: re (implicit)
Sponsored by: The FreeBSD Foundation

localedef(1): minor spelling fixes on comments.

No functional change.

Small typo.

Plug memory leaks

Reported by: Coverity
CID= 1338535, 1338536, 1338542, 1338569, 1338570

localedef(1): minor sorting to match Illumos.

Illumos recently included space in 'print' class. We already had
this but the code had slight sorting differences. Move it some
lines up to reduce diffs with Illumos.

No functional change.

Reference:
https://illumos.org/issues/5227

DIRDEPS_BUILD: Regenerate without local dependencies.

These are no longer needed after the recent 'beforebuild: depend' changes
and hooking DIRDEPS_BUILD into a subset of FAST_DEPEND which supports
skipping 'make depend'.

Sponsored by: EMC / Isilon Storage Division

META MODE: Fix 'make the-lot' with recent locale changes

Sponsored by: EMC / Isilon Storage Division

lower again the warnings and remove the pragmas unsupported by gcc 4.2.1

Eliminate some gcc pragmas

Fix build of localedef(1) on arm where wchar_t is an unsigned int

Rewrite the histoty part
Fix information about "Dragonfly-style" format which on freebsd is named
BSD-style

Noted by: bdrewery

Improve localedef(1) manpage

Obtained from: DragonflyBSD

Bump warning level

Use const where needed instead of using pragmas to work around the warnings

Make bsd declaration static

Fix an off by one due to bad conversion from avl(3) to tree(3)
Readd calloc as it was not the issue just the messenger

Submitted by: dim
Found by: Address Sanitizer

Run memset only after having checked the return of malloc

Submitted by: pluknet

Workaround an issue on i386 to unbreak the build until the real issue is tracked
down

Fix build on arm64

Improve collation string and locales support

Merge collation support from Illumos and DragonflyBSD.

Locales are now generated with the new localedef(1) tool from CLDR POSIX files.
The generated files are now identified as "BSD 1.0" format.

The libc now only read "BSD 1.0" locales definitions, all other version will be
set to "C"
The localedef(1) tool has been imported from Illumos and modified to use tree(3)
instead of the CDDL avl(3)
A set of tool created by edwin@ and extended by marino@ for dragonfly has been
added to be able to generate locales and the Makefiles from the vanilla CLDR
unicode databases + a universal UTF-8 charmap (by marino@)
Update the locales to unicode v27
Given our regex(3) does not support multibyte (yet) it has been forced to always
use locale C
Remove now unused colldef(1) and mklocale(1)
Finish implementing the numeric BSD extension for ctypes
The number of supported locales has grown from 175 to 250 locales. Among the new
locales: 6 Arabic locales (AE EG JO MA QA SA), Different variations of spanish
locales.
Added new 3 components locales for mn_Cyrl_MN, sr_Cyrl_RS sr_Latn_RS,
zh_Hans_CN, zh_Hant_HK and zh_Hant_TW. Some aliases has been for 2 components
version when possible.

Thanks: Garrett D'Amore (Illumos) who made sure all his work was done under
BSD license!, Edwin Groothuis (edwin@) for the work he made on tools to be able
to generate locales definition usable in freebsd sources out of vanilla CLDR
definitions, John Marino (DragonflyBSD) who first merge the Illumos work into
Dragonfly and spent hours tracking down bugs.

Add missing header

Fix typo

libc: Fix (and improve) nl_langinfo (CODESET)

The output of "locale charmap" is identical to the result of
nl_langinfo (CODESET) for any given locale. The logic for returning the
codeset was very simplistic. It just returned portion of the locale name
after the period (e.g. en_FR.ISO8859-1 returned "ISO8859-1").

When softlinks were added to locales, this broke. e.g.:
en_US returned ""
en_FR.UTF8 returned "UTF8"
en_FR.UTF-8 returned "UTF-8"
zh_Hant_HK.Big5HKSCS returned "Big5HKSCS"
zh_Hant_TW.Big5 returned "Big5"
es_ES@euro returned ""

In order to fix this properly, the named locale cannot be used to
determine the encoding. This information was almost available in the
rune data. Unfortunately, all the single byte encodings were listed
as "NONE" encoding.

So I adjusted localedef tool to provide more information about the
encoding. For example, instead of "NONE", the LC_CTYPE used by
fr_FR.ISO8859-15 is now encoded as "NONE:ISO8859-15". The locale
handlers now check if the first four characters of the encoding is
"NONE" and if so, treats it as a single-byte encoding.

The nl_langinfo handling of CODESET was adjusting accordingly. Now the
following is returned:
en_US returns "ISO8859-1"
fr_FR.UTF8 returns "UTF-8"
fr_FR.UTF-8 returns "UTF-8"
zh_Hant_HK.Big5HKSCS returns "Big5"
zh_Hant_TW.Big5 returns "Big5"
es_ES@euro returns "ISO8859-15"

as before, "C" and "POSIX" locales return "US-ASCII". This is a big
improvement. The result of nl_langinfo can never be a zero-length
string and it will always exclusively one of the values of the
character maps of /usr/src/tools/tools/locale/etc/final-maps.

Submitted by: marino
Obtained from: DragonflyBSD

Actually only T_ISDIGIT should be flagged as _E4

With regard to ctype, digits (e.g. 0 to 9) and xdigits (the 0 to 9 portion
of hexidecimal numbers) are all considered "numbers". (Note that while
all digits are numbers, not all numbers are digits).

Enhance localedef to automatically set the "number" characteristic when
it encounters a digit or xdigit definition. This fixes malfunctionning
isalnum(3)

Obtained from: DragonflyBSD

eliminate need for "print" definition

By having space automatically classified as "print" type, we can
eliminate the print section from ctype src files completely (they
are just "graph" plus "<space>".

Obtained from: Dragonfly

Commit log from Dragonfly:

FreeBSD extended ctypes to include numbers (e.g. isnumber()) but never
actually implemented it. The isnumber() function was equivalent to the
isdigit() function in every case.

Now that DragonFly's ctype source files have number definitions, the
number ctype can finally be implemented. It's given a new flag _CTYPE_N.
The isalnum() and iswalnum() functions have been changed to use this
flag rather than the _CTYPE_D digit flag.

While isalnum(), isnumber(), and their wide equivalents now return
different values in locale cases, the ishexnumber() and iswhexnumber()
functions are unchanged. They are still aliases for isxdigit() and
iswxdigit().

Also change ctype.h for isdigit and isxdigit to use sbistype like the
other functions.

Obtained from: dragonfly

Merge from HEAD

Pet mandoc -Tlint

Convert localedef(1) from avl to RB trees

Prefer static generation of functions

Convert ctype generation to Red Black tree

Add localedef(1), a locale definition generator tool

The localedef tool can read entire (and unmodified) CLDR posix definition
files, and generate all 6 LC categories: LC_COLLATE, LC_CTYPE, LC_TIME,
LC_NUMERIC, LC_MONETARY and LC_MESSAGES.

This tool has a long history with Solaris. The Nexenta developers
modified it to read CLDR files and created the much richer collation
formats. The libc collation functions have to be modified to read the
new format (called "BSD-1.0") and to handle the new data structures.

The result will be that locale-sensitive tools and functions will now
properly sort multibyte and unicode strings.

Obtained from: Dragonfly