localedef: add newlines to error messages

These won't be added elsewhere, so add a little bit of room to make
these messages a little easier to read. The existing set is a mixed
bag, there are somewhere in the ballpark of 45, 46 printfs to stderr and
19 of those had newlines.

Reviewed by: yuripv
Sponsored by: Klara, Inc.
Differential Revision: https://reviews.freebsd.org/D41693

Remove $FreeBSD$: one-line .c pattern

Remove /^[\s*]*__FBSDID\("\$FreeBSD\$"\);?\s*\n/

Add collation version support to querylocale(3).

Provide a way to ask for an opaque version string for a locale_t, so
that potential changes in sort order can be detected. Similar to
ICU's ucol_getVersion() and Windows' GetNLSVersionEx(), this API is
intended to allow databases to detect when text order-based indexes
might need to be rebuilt.

The CLDR version is extracted from CLDR source data by the Makefile
under tools/tools/locale, written into the machine-generated Makefile
under shared/colldef, passed to localedef -V, and then written into
LC_COLLATE file headers. The initial version is 34.0.
tools/tools/locale was recently updated to pull down 35.0, but the
output hasn't been committed under share/colldef yet, so that will
provide the first observable change when it happens. Other versioning
schemes are possible in future, because the format is unspecified.

Reviewed by: bapt, 0mp, kib, yuripv (albeit a long time ago)
Differential Revision: https://reviews.freebsd.org/D17166

Fix build with recent byacc.

Add -b/-l options to localedef(1) to specify output endianness and use
it appropriately when building share/ctypedef and share/colldef.

This makes the resulting locale data in EL->EB (amd64->powerpc64) cross
build and in the native EB build match. Revert the changes done to libc
in r308170 as they are no longer needed.

PR: 231965
Reviewed by: bapt, emaste, sbruno, 0mp
Approved by: kib (mentor)
Differential Revision: https://reviews.freebsd.org/D17603

localedef(1): remove duplicated includes.

Hinted by: DragonFlyBSD

Mark errf _Noreturn, and mark errf and warn __printflike

The _Noreturn attribute was added to placate Coverity and other static
analysis tools. The __printflike attribute was added to catch issues
with the calls related to printf(3) abuse.

- Modify the code to facilitate the __printflike attribute addition.
- Convert errf calls in to_mb(..) and to_mb_string(..) to warn(..) so
the calls will return instead of exiting, as the code suggests it
should.

Differential Revision: D10704
MFC after: 1 month
Reviewed by: pfg
Sponsored by: Dell EMC Isilon

localedef(1): Add comment markings for license.

Revert r314969, r314961:

The localdef(1) changes are breaking world:

00:18:40.750 /usr/src/share/colldef/af_ZA.UTF-8.src: 2421: error: Bad file
descriptor

I will fix them offline.

Reported by: lwshu and many others

localedef(1): Fix mismatch in previous commit.

delete_category is meant to replace fclose() and unlink().
This broke world.

Found by: kib
Pointedhat: pfg

localedef(1): Fix for memory leaks reported by coverity.

Also some small cleanups to match better current illumos.

CID: 1338540, 1338541, 1338557, 1338566

Obtained from: illumos
Discussed with: Yuri Pankov (@Nexenta)
MFC after: 5 days

Make bsd declaration static

Add localedef(1), a locale definition generator tool

The localedef tool can read entire (and unmodified) CLDR posix definition
files, and generate all 6 LC categories: LC_COLLATE, LC_CTYPE, LC_TIME,
LC_NUMERIC, LC_MONETARY and LC_MESSAGES.

This tool has a long history with Solaris. The Nexenta developers
modified it to read CLDR files and created the much richer collation
formats. The libc collation functions have to be modified to read the
new format (called "BSD-1.0") and to handle the new data structures.

The result will be that locale-sensitive tools and functions will now
properly sort multibyte and unicode strings.

Obtained from: Dragonfly