[EN-15:08] Revised: Improvements to sendmail TLS/DH interoperability.

[EN-15:09] Fix inconsistency between locale and rune locale states.

[EN-15:10] Improved iconv(3) UTF-7 support.

Approved by: so

Copy stable/10@r272459 to releng/10.1 as part of
the 10.1-RELEASE process.

Approved by: re (implicit)
Sponsored by: The FreeBSD Foundation

MFC r268272:

minor perf enhancement for UTF-8

Reduce some duplicate code.

Reference:
https://www.illumos.org/issues/628

Obtained from: Illumos

MFC r265095, r265167;

citrus: Avoid invalid code points.

The UTF-8 decoder should not accept byte sequences which decode to
unicode code positions U+D800 to U+DFFF (UTF-16 surrogates).[1]

Contrary to the original OpenBSD patch, we do pass U+FFFE and U+FFFF,
both values are valid "non-characters" [2] and must be mapped through
UTFs.

[1] http://www.cl.cam.ac.uk/~mgk25/unicode.html#utf-8
[2] http://www.unicode.org/faq/private_use.html

Reported by: Stefan Sperling [1]
Thanks to: jilles [2]
Obtained from: OpenBSD

Copy head (r256279) to stable/10 as part of the 10.0-RELEASE cycle.

Approved by: re (implicit)
Sponsored by: The FreeBSD Foundation

Replace the #define for "iconv" so it is for the function name instead of
a macro with parameters. Remove a __DECONST hack and add consts instead
for gnu libiconv API compatability. This makes it work with things like
devel/boost-libs that expects to use "iconv" as though it were a pointer.

Add libiconv based versions of *c16*() and *c32*().

I initially thought wchar_t was locale independent, but this seems to be
only the case on Linux. This means that we cannot depend on the *wc*()
routines to implement *c16*() and *c32*(). Instead, use the Citrus
libiconv that is part of libc.

I'll see if there is anything I can do to make the existing functions
somewhat useful in case the system is built without libiconv in the
nearby future. If not, I'll simply remove the broken implementations.

Reviewed by: jilles, gabor

Add <uchar.h>.

The <uchar.h> header, part of C11, adds a small number of utility
functions for 16/32-bit "universal" characters, which may or may not be
UTF-16/32. As our wchar_t is already ISO 10646, simply add light-weight
wrappers around wcrtomb() and mbrtowc().

While there, also add (non-yet-standard) _l functions, similar to the
ones we already have for the other locale-dependent functions.

Reviewed by: theraven

Document that the return type is different from 1003.1-2008.

MFC after: 1 week

mdoc: missing comma in .Dd macro.

Also, add a missing period.

Remove an extra comma.

Remove the STANDARDS section.
querylocale is not part of IEEE Std 1003.1-2008.

MFC after: 3 days

btowc(3), isblank(3): Correct prototypes for _l variants.

MFC after: 1 week

libc: Make various internal file descriptors close-on-exec.

These are obtained via fopen().

Improve style(9) compliance of function declarations.

libc: Use O_CLOEXEC for various internal file descriptors.

This fixes a race condition where another thread may fork() before CLOEXEC
is set, unintentionally passing the descriptor to the child process.

This commit only adds O_CLOEXEC flags to open() or openat() calls where no
fcntl(fd, F_SETFD, FD_CLOEXEC) follows. The separate fcntl() call still
leaves a race window so it should be fixed later.

Remove trailing whitespace.

Add more locale-specific functions to the relevant man pages and Makefile:
- lib/libc/locale/islower.3
- lib/libc/locale/ispunct.3
- lib/libc/locale/nl_langinfo.3
- lib/libc/locale/isgraph.3
- lib/libc/locale/isspace.3

Reviewed by: bz
Approved by: theraven
MFC after: 5 days

Start manpage with Dd macro and also remove a trailing whitespace
while here.

Add a new man page containing details of new locale-specific functions for
wctype.h, iswalnum_l(3). Add it and its functions to the Makefile.

Reviewed by: gavin, jilles
Approved by: theraven
MFC after: 5 days

Restore the __collate_load_error global that was accidentally removed in the
xlocale refactoring.

MFC after: 1 week

ctype_l(3): Note that not all these functions are specified by POSIX.1-2008.

The function isascii_l() is not in POSIX even though isascii() is,
probably because isascii() is marked as obsolete. The other functions,
like digittoint_l() and ishexnumber_l(), are FreeBSD-specific just like
their non-_l versions.

Fix a leak when setting the global character locale to "C" from something else.

Reported by: mm

General mdoc(7) and typo fixes.

PR: 167734
Submitted by: Nobuyuki Koganemaru (kogane!jp.freebsd.org)
MFC after: 3 days

Fix copy/paste error in lib/libc/locale/toupper.c.

Submitted by: Kohji Okuno <okuno.kohji@jp.panasonic.com>

Fix some incorrect symbol versions.

Reported by: das

mdoc: fix function type.

mdoc: fix column names, indentation, column separation within each row, and
quotation. Also make sure we have the same amount of columns in each row as
the number of columns we specify in the head arguments.

Reviewed by: brueffer

Remove trailing whitespace per mdoc lint warning

Disussed with: gavin
No objection from: doc
Approved by: joel
MFC after: 3 days

mdoc: remove unknown macro.

mdoc: remove empty lines.

mdoc: document title should be all caps.

Make sure sections are sorted into conventional order.

Remove superfluous paragraph macro.

Make __get_locale() static inline, not just inline, so when compiling libc in
debug mode (without optimisations) it doesn't generate a linker failure.

Approved by: dim (mentor)

First set of xlocale man pages. More to follow...

Approved by: dim (mentor)

More xlocale cleanups.

Approved by: dim (mentor)

Fix build of libc.so after r232620. This caused a duplicate definition
of __getCurrentRuneLocale().

Pointy hat to: me

Remove some duplicated copyright notices.

Approved by: dim (mentor)

Reapply 227753 (xlocale cleanup), plus some fixes so that it passes build
universe with gcc.

Approved by: dim (mentor)

Revert r231673 and r231682 for now, until we can run a full make
universe with them. Sorry for the breakage.

Pointy hat to: me and brooks

Fix a misplaced __NO_TLS locations, and change a GNUism to a C11ism for
consistency.

Approved by: brooks (mentor)

Cleanup of xlocale:

- Address performance regressions encountered by das@ by caching per-thread
data in TLS where available.
- Add a __NO_TLS flag to cdefs.h to indicate where not available.
- Reorganise the xlocale.h definitions into xlocale/*.h so that they can be
included from multiple places.
- Export the POSIX2008 subset of xlocale when POSIX2008 says it should be
exported, independently of whether xlocale.h is included.
- Fix the bug where programs using ctype functions always assumed ASCII unless
recompiled.
- Fix some style(9) violations.

Reviewed by: brooks (mentor)
Approved by: dim (mentor)

wctob() returns EOF and not WEOF.

Noticed by: Zhihao Yuan <lichray gmail com>
MFC after: 1 week

libc: Eliminate some relative relocations in setlocale().

libc: Eliminate 13 relative relocations in wctype().

Split sentences at period boundaries.

Fix a crash when trying to duplicate a locale that contains some implicit C locale components.

Reported by: Michael Butler
Approved by: dim (mentor)

Implement xlocale APIs from Darwin, mainly for use by libc++. This adds a
load of _l suffixed versions of various standard library functions that use
the global locale, making them take an explicit locale parameter. Also
adds support for per-thread locales. This work was funded by the FreeBSD
Foundation.

Please test any code you have that uses the C standard locale functions!

Reviewed by: das (gdtoa changes)
Approved by: dim (mentor)

Fix grammar.

PR: 140457
Submitted by: jeremyhu AT apple.com
MFC after: 2 weeks

Revert most of r210764, now that mdocml does the right
thing with empty quotation macros.

Requested by: Alex Kozlov

mdoc: drop redundant .Pp and .LP calls

They have no effect when coming in pairs, or before .Bl/.Bd

Powerpc is special here. powerpc and powerpc64 use different ABIs, so
their implementations aren't in the same files. Introduce LIBC_ARCH
and use that in preference to MACHINE_CPUARCH. Tested by amd64 and
powerpc64 builds (thanks nathanw@)

MFtbemd:

Prefer MACHNE_CPUARCH to MACHINE_ARCH in most contexts where you want
to test of all the CPUs of a given family conform.

mdoc: remove unbalanced quotes

mdoc: make sure to pass at least one argument to quotation macros

Apply a small grammar fix to {toupper,tolower}(3).

PR: docs/140458
Submitted by: Jeremy Huddleston (Jeremyhu at apple dot com)
MFC after: 5 days

mdoc: simplify quotation scheme

Reviewed by: brueffer

mdoc: remove .Pp where not needed

This trips up mdocml and can simply go away.

Reviewed by: ru
Approved by: philip, ed (mentors)

Fix grammar.

PR: 140459
Submitted by: Jeremy Huddleston <Jeremyhu@apple.com>
MFC after: 1 week

Fix the grammar in the isprint(3) description.

PR: 140456
Submitted by: Jeremy Huddleston <jeremyhu@apple.com>

Fix the grammar in the isgraph(3) description, almost as per the PR.

PR: 140455
Submitted by: Jeremy Huddleston <jeremyhu@apple.com>
MFC after: 2 weeks

Fix the grammar as in the PR, and then some.

PR: 140454
Submitted by: Jeremy Huddleston <jeremyhu@apple.com>
MFC after: 2 weeks

Modified locale(1) to be able to show the altmon_X fields and the [cxX]_fmt's.
Also modify the "-k list" option to display only fields with a certain prefix.

MFC after: 1 week

Let the armchair generals handle this one.

Document the need for a cast when passing a char to a ctype function.

MFC after: 2 weeks

Change dstp to be const and remove bogus cast.

Submitted by: christoph.mallon at gmx.de
Approved by: kib (mentor)

Introduce a local variable and use it instead of passed in parameter
to get rid of restrict qualifier discarding. This lets libc compile
cleanly in gnu99 mode.

Suggested by: kib, christoph.mallon at gmx.de
Approved by: kib (mentor)

Don't allocate the constant array "props" on the stack in wctype.

PR: 74743
Submitted by: knut st. osmundsen
Approved by: rwatson (mentor)
MFC after: 1 month

Fix longstanding mb/wc functions segfault if error occurse
inside _<encoding>_init().
Currently _EUC_init() only was affected.

Better fix for longstanding segfault. Don't touch current locale at all
on unknown encoding. Previous fix resets it to POSIX.

1) Add (void) cast to _none_init() (while I am here)
2) Fix longstanding segfault in mb/wc code when unknown encoding is specified
in the locale file (mb/wc functions becomes NULL in that case).

Introduce new encoding: "ASCII"
It differs from default C/POSIX "NONE" mainly by stricter 8bit check
for mb*towc*/wc*tomb* family, returning EILSEQ

Remove 3rd clause of license

Per request of: glenn halperin at symbian.com

- Include runetype.h for _RuneLocale_

Back out 2nd part of wrong iswascii() change in prev. commit.

Add comment explaining __mb_sb_limit trick here.

The problem is: currently our single byte ctype(3) functions are broken
for wide characters locales in the argument range >= 0x80 - they may
return false positives.

Example 1: for UTF-8 locale we currently have:
iswspace(0xA0)==1 and isspace(0xA0)==1
(because iswspace() and isspace() are the same code)
but must have
iswspace(0xA0)==1 and isspace(0xA0)==0
(because there is no such character and all others in the range
0x80..0xff for the UTF-8 locale, it keeps ASCII only in the single byte
range because our internal wchar_t representation for UTF-8 is UCS-4).

Example 2: for all wide character locales isalpha(arg) when arg > 0xFF may
return false positives (must be 0).
(because iswalpha() and isalpha() are the same code)

This change address this issue separating single byte and wide ctype
and also fix iswascii() (currently iswascii() is broken for
arguments > 0xFF).
This change is 100% binary compatible with old binaries.

Reviewied by: i18n@

- Fix typo

Approved by: keramida (mentor)
MFC after: 3 days

Use C comments since we now preprocess these files with CPP.

Per Regents of the University of Calfornia letter, remove advertising
clause.

# If I've done so improperly on a file, please let me know.

Add missing comma.

Fix a bug where, for 6-byte sequences, the top 6 bits get compared to
111111 rather than the top 7 bits being compared against 1111110 causing
illegal bytes fe and ff being treated the same as legal bytes fc and fd.

Add __collate_load_error and __collate_range_cmp to the list of
FBSDprivate locale symbols. These functions are needed by
libcompat.

Add _cleanup to the list of stdio FBSDprivate symbols. Some
third party applications use this. This will be removed and
replaced by fcloseall() once libc version is bumped.

Add _res to the list of resolv symbols.

Found by: portbuilder runs (thanks Kris!)

Add each directory's symbol map file to SYM_MAPS.

Add symbol maps and initial symbol version definitions to libc.

Reviewed by: davidxu

-mdoc sweep.

Include a couple of headers to ensure consistency between the prototype and
the function definition.

Add HISTORY section.

Add cross-reference to nextwctype(3).

Add COMPATIBILITY and HISTORY sections. Fix typo.

Remove confusing "single C char locales" phrase; arguments to these
functions and must now be either an unsigned char or EOF, regardless of
locale.

Remove confusing "single C char locales" phrase; arguments to tolower()
and toupper() must now be either an unsigned char or EOF, regardless of
locale.

Make <runefile.h> internal to libc.

Suggested by: phantom

Make the format of LC_COLLATE files architecture independent.

. Static'ize functions exported via function reference variables only.
. Replace inclusion of sys/param.h to sys/cdefs.h and sys/types.h where
appropriate.
. move _*_init() prototypes to mblocal.h, and remove these prototypes
from .c files
. use _none_init() in __setrunelocale() instead of duplicating code
. move __mb* variables from table.c to none.c allowing us to not to
export _none_*() externs, and appropriately remove them from mblocal.h

Ok'ed by: tjr

ANSI'fy prototypes

Make the format of LC_CTYPE files architecture independent by
introducing the disk formats for _RuneLocale and friends.

The disk formats do not have (useless) pointers and have 32-bit
quantities instead of rune_t and long. (htonl(3) only works
with 32-bit quantities, so there's no loss).

Bootstrap mklocale(1) when necessary. (Bootstrapping from 4.x
would be trivial (verified), but we no longer provide pre-5.3
source upgrades and this is the first commit to actually break
it.)

Fix comparisons that test if an unsigned value is < 0.

Reviewed by: tjr

Sort sections.

Markup style.

Scheduled mdoc(7) sweep.

Implement rpmatch(), a semi-standard interface (as found on AIX, Tru64,
GNU) for determining whether a string is an affirmative or negative
response to a question according to the current locale. This is done
by matching the response against nl_langinfo(3) items YESEXPR and NOEXPR.

Remove setrunelocale()

Remove the obsolete <rune.h> interface.

Remove support for the obsolete UTF2 encoding.

Prefer C99's __func__ over GCC's __FUNCTION__.

Re-word warning about the UTF2 encoding, taking care to use the word
"obsolete" instead of "deprecated".

Bump document date for previous.

Re-word warning about the rune interface, taking care to use the word
"obsolete" instead of "deprecated".

Change "deprecated" in link-time warnings about various rune functions
to "obsolete".

Re-word compatibility section, taking care to use the word "obsolete" to
describe the 4.4BSD extension of accepting characters (runes) outside of
the range of unsigned char.

/me kicks cvs update

Revert previous commit, tjr already fixed it and I was too stupid to
notice this fact.

Approved by: re (to avoid failing cvs ci)

Fix incorrect code in an example. The previous example would produce
19 column positions wide in the first line and 20 in the rest of the lines.
This fixes the example to provide the correct output.

PR: 53454
Noticed by: Kuang-che Wu <kcwu@kcwu.homeip.net>
Submitted by: Marc Silver <marcs@draenor.org>
Approved by: re (scottl)

Fix example.

Implement wcwidth() as an inline function.

Re-word the COMPATIBILITY section, taking care to use the word "deprecated"
to describe the 4.4BSD extension of accepting arguments outside the range
of unsigned char. This gives us freedom to remove this extension when we
remove the <rune.h> interface in FreeBSD 6.

Remove unnecessary #include directives.

Prefer <runetype.h> to <rune.h>, since the latter is going away soon.

Remove useless checks for characters longer than INT_MAX bytes.

Add UTF-8-specific implementations of mbsnrtowcs() and wcsnrtombs().
These convert plain ASCII characters in-line, making them only slightly
slower than the single-byte ("NONE" encoding) version when processing
ASCII strings.

Return the correct value when dst == NULL and conversion has stopped after
nwc dropping to zero.

Implement the GNU extensions of mbsnrtowcs() and wcsnrtombs(). These are
convenient when the source string isn't null-terminated.

Implement the other conversion functions (mbstowcs(), mbsrtowcs(), wcstombs(),
wcsrtombs()) in terms of these new functions.

Add fast paths for conversion of plain ASCII characters.

Add a function to iterate over all characters in a particular character
class. This is necessary in order to implement tr(1) efficiently in
multibyte locales, since the brute force method of finding all characters
in a class is infeasible with a 32-bit (or wider) wchar_t.

Markup nits.

Sort SEE ALSO references (in dictionary order, ignoring case).

Mechanically kill hard sentence breaks.

Removed trailing whitespace.

Markup, grammar, and spelling fixes.

Fixed a typo.

Prefix the names of members of _RuneLocale and its sub-structures
with ``__'' to avoid polluting the namespace. This doesn't change the
documented rune interface at all, but breaks applications that accessed
_RuneLocale directly.

Spelling fixes.

Buffer partial wide characters more efficiently: instead of storing the
multibyte representation in conversion state objects, store the
accumulated wide character, set number and number of bytes remaining
to avoid having to derive them every time mbrtowc() is called.

Scan the source string for invalid wide characters in wcsrtombs()
in the dst == NULL case.

Grab all the information we need about a character with one call to
__maskrune() instead of one direct call and one through iswprint().

Use conversion state objects to store the accumulated wide character,
low bound, and the number of bytes remaining instead of storing the
raw byte sequence and deriving them every time mbrtowc() is called.
This is much faster -- about twice as fast in some crude benchmarks.

Use a simpler and faster buffering scheme for partial multibyte characters.

Use a simpler, faster buffering scheme for partial characters in mbrtowc().

Allow encoding modules to override the default implementations of
mbsrtowcs() and wcsrtombs(). Provide a fast implementation for the
trivial "NONE" encoding.

Fix braino in previous: check that the second byte in the character
buffer is non-null when the character is two bytes long, not when
the buffer is two bytes long.

Reduce overhead by calling internal versions of the multibyte conversion
functions directly wherever possible.

Move prototypes of various encoding-related functions into a new header
file to avoid extern'ing them all over the place.

In the absence of proper validation, at least check that null bytes
do not appear as anything but the first byte of a multibyte character.

Use a binary search to find the range containing a character in
RuneRange arrays. This is much faster when there are hundreds of
ranges (as is the case in UTF-8 locales) and was inspired by a
similar change made by Apple in Darwin.

Rewrite split_lines() to operate safely

PR: 62694
Submitted by: moulin p <moulin.p@calyopea.com>

Perform some basic validation of multibyte conversion state objects.

Remove a nonsensical remark about byte order markers in UTF-8 streams.

Document the meaning of the zero return value.

Fix a typo. I was locked out for two days from my machine.

Don't cast away const qualifiers.

Spotted by: bde

Update manual pages for change to C99 mbrtowc() semantics.

Allow partial multibyte characters to accumulate in conversion state
objects passed to mbrtowc(), mbsrtowcs(), and mbrlen(), as required
by C99.

Begin conversions for sgetrune() and sputrune() in the initial
conversion state.

Prepare to handle state-dependent encodings. This mainly involves not
taking shortcuts when it comes to storing and passing around conversion
states.

Begin in the initial shift state in mbstowcs() and wcstombs().
(This change is non-functional since nothing uses states yet.)

Prepare to handle state-dependent encodings. This mainly involves not
taking shortcuts when it comes to storing and passing around conversion
states.

Remove support for emulating mbrtowc() and wcrtomb() in terms of the
old rune interface now that it is no longer needed.

Reimplement the GB18030 encoding method using the new-style (mbrtowc()/
wcrtomb()) interface.

Reimplement the deprecated UTF2 encoding method using the UTF-8 code
as a base. mbrtowc() and wcrtomb() are now implemented directly
instead of being emulatedi with sgetrune() and sputrune().

Add cross-references to isideogram(3), isphonogram(3), isrune(3),
isspecial(3) and wctype(3).

Add basic manual pages for isideogram(), isphonogram(), isrune()
and isspecial().

Trim cross-references.

Document the isnumber() and ishexnumber() functions, and explain how they
differ (at least in theory) from isdigit() and isxdigit().

Remove duplicate MLINK.

Recognize the "rune" character class in wctype().

Make consistent with the better written wcsrtombs function:
- Fix syntax
- Remove the (slightly wrong) duplicate explanation of the error condition
- Change reference to invalid multibyte character into invalid wide character

LC_ALL not always take priority over other LC_*

Obtained from: NetBSD
PR: 62047

Add reference to environ(7)

Remove unused variables and function declarations. Add missing headers.

Properly advance "x/y/z" form slash-pointers in some rare cases

PR: 60539

First byte of GBK-like sequences is 0x81, not 0x80

Set __mbrtowc and __wcrtomb correctly when changing to the C/POSIX locale.
Save __mbrtowc and __wcrtomb and restore them when changing back to
the cached locale.

Reported by: perky

Split multibyte(3) into separate manual pages for each function.
Instead of just deleting it, turn the original page into a general
overview of the multibyte character conversion functions, somewhat
similar to stdio(3).

Split the documentation for localeconv() off into a separate manual page.

Update cross references after utf2/euc move.

Remove section 4 versions of these manual pages, they have been
moved into section 5.

Install the section 5 versions of EUC and UTF2 manual pages instead of
the section 4 versions.

Update the EUC and UTF2 manual pages for their new home in section 5.
These have been repo-copied from euc.4 and utf2.4.

Fix a typo that caused mbrtowc() to always return 0.

Add one more cross-reference to gb2312(5).

Add cross-references to new gb2312(5) manual page.

Add a fairly simple manual page for the new GB2312 encoding.

Remove unused #includes.

Use __inline instead of inline.

Refer to wide characters instead of runes. Remove redundant example locale.
Catch up with renaming of "Japanese" to "ja_JP.eucJP". Comment out the
statement that EUC is provided for compatibility with UNIX-based systems;
this is not a very good opening paragraph.

Refer to wide characters instead of runes.

Add gb2312 encoding.

Implement mbrtowc() and wcrtomb() directly (sync with big5.c).

Convert the Big5, EUC, MSKanji and UTF-8 encoding methods to implement
mbrtowc() and wcrtomb() directly. GB18030, GBK and UTF2 are left
unconverted; GB18030 will be done eventually, but GBK and UTF2 may just
be removed, as they are subsets of GB18030 and UTF-8 respectively.

Remove TODO comment about creating a macro version of towctrans().
Remove unnecessary inclusion of <ctype.h>.

Allow mbrtowc() and wcrtomb() to be implemented directly, instead of
as wrappers around the deprecated 4.4BSD rune functions. This paves the
way for state-dependent encodings, which the rune API does not support.
- Add __emulated_sgetrune() and __emulated_sputrune(), which are
implementations of sgetrune() and sputrune() in terms of
mbrtowc() and wcrtomb().
- Rename the old rune-wrapper mbrtowc() and wcrtomb() functions to
__emulated_mbrtowc() and __emulated_wcrtomb().
- Add __mbrtowc and __wcrtomb function pointers, which point to the
current locale's conversion functions, or the __emulated versions.
- Implement mbrtowc() and wcrtomb() as calls to these function pointers.
- Make the "NONE" encoding implement mbrtowc() and wcrtomb() directly.

All of this emulation mess will be removed, together with rune support,
in FreeBSD 6.

Don't bother passing a freshly-zeroed mbstate to mbsrtowcs() etc.
when the current implementation won't use it, anyway. Just pass NULL.
This will need to be changed when state-dependent encodings are
supported, but there's no need to take the performance hit
in the meantime.

Implement fgetrune(), fungetrune() and fputrune() as wrappers around
fgetwc(), ungetwc() and fputwc().

Remove incomplete support for running FreeBSD userland on old NetBSD kernels
lacking the issetugid() and utrace() syscalls.

mdoc(7): Use the new feature of the .In macro.

Remove an unused and incorrect prototype for _none_init().

Fix the case of the encoding name in the ENCODING line. Names are
case-sensitive, and MSKANJI does not work.

Cross-reference gbk(5).

Cross-reference gbk(5) now that it exists. Fix a copy & paste error:
one occurrence of GB 18030 should have been 11383.

Add a fairly minimal manual page for the GBK encoding.

Add a cross reference to Unicode 3.0.

Add cross references to the new character encoding manual pages,
and to mbsinit(3) while I'm at it.

Add manual pages for the BIG5, GB18030 and MSKanji encodings. These may
need to be fleshed out a little, especially big5(5).

Implement mblen(s, n) as mbtowc(NULL, s, n) to avoid calling sgetrune()
and to simplify things. This is only valid until we start supporting
state-dependent encodings.

Implement mbstowcs() as a wrapper around mbsrtowcs(), and wcstombs()
as a wrapper around wcsrtombs().

Implement mbtowc() in terms of mbrtowc(), and wctomb() in terms of wcrtomb().

Implement btowc() in terms of mbrtowc() instead of sgetrune(), and
wctob() in terms of wcrtomb() instead of sputrune(). There should be
no functional differences, but there may be a small performance hit
because we make an extra function call.

The aim here is to have as few functions as possible calling
s{get,put}rune() to make it easier to remove them in the future.

Restore including of "collate.h", for its own prototype (mis)match detection

Remove commented out and never used code

Remove __collate_range_cmp() stabilization, it conflicts with ranges

Add support for gb18030 encoding

PR: 51729
Submitted by: Kang Liu <liukang@bjpu.edu.cn>

Add const to __setrunelocale prototype

Reorganize wrapper around setrunelocale() to mark it as deprecated
in FreeBSD 6

. style(9)
. fix/add comments (to cover changes done thru last 20 months)
. extend monetary testcase to cover int_* values

Reduce code duplication by separating _PathLocle detection code into
internal helper function.

Move _PathLocale declaration to more logical place (setlocale.c)

Catch up with _PATH_LOCALE move from rune.h to paths.h

Mark the following interfaces as OBSOLETE_IN_6:
fgetrune(), fputrune(), fungetrune(), mbrune(), mbrrune(), mbmb(),
setinvalidrune(), UTF2 encoding method.
These have been marked as being deprecated in their manual pages since 5.0,
and their use causes a linker warning.

Fixes to locale code to properly use indirect pointers in order to prevent
memory leaks (fixes bugs earlier purported to be fixed).
Submitted by: Ed Moy <emoy@apple.com>
Obtained from: Apple Computer, Inc.
MFC after: 2 weeks

Remove transition period hack

Add GBK encoding

PR: 51504
Submitted by: Statue <statue@freebsd.sinica.edu.tw>

Assorted mdoc(7) fixes.

Approved by: re (blanket)

Back out the `hiding' of strlcpy and strlcat. Several people
vocally objected to this safety belt.

`Hide' strlcpy and strlcat (using the namespace.h / __weak_reference
technique) so that we don't wind up calling into an application's
version if the application defines them.

Inspired by: qpopper's interfering and buggy version of strlcpy

When called with s == NULL, behave as if wc == L'\0' as required by the
standard.

According to C99 decimal_point can't be the empty string, mention it.

decimal_point can't be "" according to C99, so set it to standard "."
in that case.

MFp4: Implementations of the wcstof() and wcstold() functions.

Fix a bad free() call that would occur if some #if 0'd code was used.

Whack 28 unused variables.

The .Fn function

Correct typos, mostly s/ a / an / where appropriate. Some whitespace cleanup,
especially in troff files.

mdoc(7) police: Deal with self-xrefs.

mdoc(7) police: "The .Fa argument.".

mdoc(7) police: Fixed abuses of the .Ar and .Em macros.

mdoc(7) police: "The .Fn function".

Capitalize ASCII code names.

Approved by: re

mdoc(7) police: sweep.

mdoc(7) police: sweep.

libc_r wasn't so tied to libc for 22 months.

Add cross references to mbrtowc(3) and wcrtomb(3).

Don't check whether the first byte of the buffer is a null byte when
the buffer has zero length (n == 0).

Describe the `n' and `ps' arguments to mbrlen().

Typo: pointer to -> pointed to

Use wide character ctype functions directly instead of relying on
4.4BSD extensions to the single-byte ctype functions.

Add a missing return statement for the pwcs == NULL case (XSI extension).

Add two additional references to the See Also section, which contain much
better descriptions of UTF-8 and related issues.

Remove unnecessary inclusion of <rune.h> to make it obvious that this file
does not use the deprecated rune system.

Handle boundary cases more correctly; mblen(s, 0) and mbtowc(NULL, s, 0)
return -1 regardless of what s points to, mbtowc(&w, s, 1) sets w to a
null wide character when s points to a null byte. This seems to be closer
to what most other implementations do, but the C99 standard contradicts
itself for these cases.

Do not include <sys/syslimits.h> directly; it is not intended for general
consumption.

Style sweep.

Use an internal buffer for the result when the first argument is NULL.

Avoid truncating invalid wide characters that are outside the range of
'unsigned char'; signal an error instead.

FA, FB and FC are lead bytes according to recent Microsoft documentation.

Style changes. Mainly removing excessive whitespace and parens.

Cosmetic: use LCMONETARY_SIZE_{FULL,MIN} defines like in other places

Add a UTF-8 encoding method, which will eventually replace the antique
"UTF2" method. Although UTF-8 and the old UTF2 encoding are compatible
for 16-bit characters, the new UTF-8 implementation is much more strict
about rejecting malformed input and also handles the full 31 bit range
of characters.

Add support for the 6 new C99 struct lconv members dealing with formatting
international monetary values: int_p_cs_precedes, int_n_cs_precedes,
int_p_sep_by_space, int_n_sep_by_space, int_p_sign_posn, int_n_sign_posn.
This should not break existing binaries or LC_MONETARY data files.

Reviewed by: ache
MFC after: 1 month

Add a note to the Compatiblity section suggesting that these functions
only be used for byte values. Add cross-references to the wide-char
counterparts.

Remove rants/whines about the rune interface being superior to the
ISO C interface.

Remove a completely incorrect statement from the Return Values section.
Add cross-references to the restartable mulitybte functions (mbrlen(3) etc.)

Improve three instances of questionable or confusing grammar.

Add an example.

Document towlower() and towupper() in separate manual pages instead of
trying to confusingly document both on the same page. The new manual pages
are based on tolower(3) and toupper(3) instead of the old towlower(3).

Point out that although toupper() and tolower() really accept rune_t's
and not just unsigned char's, callers should use towupper() and towlower()
instead when working with wide characters if portability is a concern.

towlower() appeared twice in the synopsis; one of the occurrences should
have been towupper(). Add towupper() to the Name section while I'm at it.

Obtained from: NetBSD (junyoung)

Add an Examples section with an example of how to use the functions.

Warn when setinvalidrune() is referenced for consistency with the rest
of the rune functions (except sgetrune() and sputrune(), which are really
macros).

Add the remaining C99 wide character string to integer conversion functions.
Restrict qualifiers were added to the existing prototypes in <inttypes.h>
and the typedef for wchar_t was removed.

Deprecate the rest of the rune interface.

Mark mbmb(), mbrune(), and mbrrune() as deprecated functions. We want to
get applications to move to the ISO C interfaces as well as have the
freedom to replace the rune interfaces with ones that support stateful
conversions some time in the future.

Add wcstod() as a wrapper around strtod(). It does not handle any characters
that strtod() does not (alternate digit characters, etc. are not handled).

Use the heap instead of the stack to store temporary multibyte string
buffers; this is slower but safer for threaded programs where threads
often have relatively low stack size limits.

Correct type of second argument: it is wchar_t ** restrict,
not wchar_t * restrict.

Add an implementation of wcsftime() (wide character version of strftime()).

Add wcstol() and wcstoul(), based on strtol() and strtoul().

Style: One space between "restrict" qualifier and "*".

Set errno to EILSEQ when invalid multibyte sequences are detected
(XSI extension to 1003.1-2001).

Typo: refer to MB_LEN_MAX instead of MB_CHAR_MAX (which does not exist).

Add restrict qualifiers to the arguments of mbstowcs, mbtowc() and
wcstombs().

Implement the XSI extension which allows the destination string to be
NULL, and returns the number of bytes that would be required to store
the result of the conversion without storing anything.

PR: 17694

Split ansi.c into a separate source file for each function.

Use ntohl() to read cnains number in new format

Style fix

Prepare for switching to unlimited chains format.
Optimize chains lookup a bit.

o Merge <machine/ansi.h> and <machine/types.h> into a new header
called <machine/_types.h>.
o <machine/ansi.h> will continue to live so it can define MD clock
macros, which are only MD because of gratuitous differences between
architectures.
o Change all headers to make use of this. This mainly involves
changing:
#ifdef _BSD_FOO_T_
typedef _BSD_FOO_T_ foo_t;
#undef _BSD_FOO_T_
#endif
to:
#ifndef _FOO_T_DECLARED
typedef __foo_t foo_t;
#define _FOO_T_DECLARED
#endif

Concept by: bde
Reviewed by: jake, obrien

Add a manual page for wcwidth().

Remove wcswidth.c from here (and move it to "string")

Remove space at the end of continuation line in prev. commit

Implement wcswidth()

Use modern-style arguments declaration

Write null wide-character as L'\0' like in other places

According to SUSv2, always return 0 for null wide-character code

Move internal defines from ctype.h here

Implement the ISO C90 Amd.1 restartable wide and multibyte character
manipulation functions mbrlen(), mbrtowc(), mbsinit(), mbsrtowcs(),
wcrtomb(), wcsrtombs().

Move wcwidth() to separate file, it doesn't belong to iswctype.c at all

According to SUSv2, wcwidth() should return -1 for non-printing characters

Cosmetic - remove unneded brackets and #undef

wcwidth: fix espression to work correctly with SWIDTH0

Add iswctype wcwidth function code

Submitted by: clkao@clkao.org
Reviewed by: keichii
Obtained from: NetBSD
MFC after: 1 month

Reduce BSS size for programs which not load collate by eliminating
static buffer.

Now malloc() is fixed, remove errno hardcoding to ENOMEM

Add safeguards to never use errno == 0 as setrunelocale() error return code

Rewrite locale loading procedures, so any load failure will not affect
currently cached data. It allows a number of nice things, like: removing
fallback code from single locale loading, remove memory leak when LC_CTYPE
data loaded again and again, efficient cache use, not only for
setlocale(locale1); setlocale(locale1), but for setlocale(locale1);
setlocale("C"); setlocale(locale1) too (i.e. data file loaded only once).

Reset __mb_cur_max to 1 when "C" or "POSIX" locales loaded after multibyte one

Fix wrong address when EucInfo > "variable" size

Style fixes in preparation for rewritting

Style fixes in preparation of code rewritting

Build iswctype.c and manual pages for the functions it defines.

Add missing prototypes for extension functions to the SYNOPSIS.

Use In macro instead of Fd. Add crossref to wctype(3). Refer to 1003.1-2001
in STANDARDS section. Document functions which are extensions to the standard.

Use the In macro instead of Fd. Add crossref to wctrans(3). Refer to
1003.1-2001 in STANDARDS section.

Implement the missing <wctype.h> functions: isw*() (iswalnum() etc.),
towlower() and towupper() required by ISO C90 Amd. 1.

iswascii(), iswhexnumber(), iswideogram(), iswnumber(), iswphonogram(),
iswrune() and iswspecial() have also been implemented for consistency
with the BSD extensions in <ctype.h>.

Reject encoding > ENCODING_LEN at early stage instead of truncating it.
Use ptr == NULL instead of !ptr in few places.
Move saverr declaration to global section.

Manual pages for wide character classification (isw*) and case conversion
(tow*) functions from NetBSD, unmodified except for the addition of $FreeBSD$.

Obtained from: NetBSD

Change wctype_t to an unsigned type to avoid warnings.

Add the ISO C90 Amd. 1 wctrans(3) and towctrans(3) functions.

Add btowc(3) to SEE ALSO section.

Use errno to indicate failure reason.
Remove incomplete checks for 'name' and 'PatchLocale', they must be
already checked at this point.

Fixed some style bugs (unsorting of MLINKS, and more than 1 assignment to
MAN per section).

Rewrite loadlocale() to eliminate LOAD_CATEGORY macro to save space.

Add ERRORS section according to POSIX (no errors)

Catch empty encoding name too

Fix return codes to match what setrunelocale() returns

Preserve errno in fallback code

Add ISO C90 Amd. 1 btowc(3) and wctob(3) functions.

Correct use of Nm macro in NAME section and a broken cross reference.

Return errno provided by fopen, not always ENOENT.
Return EFTYPE instead of EINVAL for wrong locale file format.
Whitespaces.

Check encoding for ".", ".." and / inside

Return EINVAL for NULL or too long encoding, not EFAULT

Return ENAMETOOLONG for long PATH_LOCALE, not EFAULT

1) Use errno to indicate faulure reason.
2) Move incomplete check for / in locale name from env section to
loadlocale(), add check for "." and ".." too.
It allows to check any argument, not env only.
3) Redesing LOAD_CATEGORY macro to eliminate code duplication.
4) Try harder in fallback code: if old locale can't be restored,
load "C" locale
5) White space formatting, long lines, etc.

Add ISO C90 Amd. 1 wctype(3) and iswctype(3) functions.

Sligtly modify previous out-of-bounds fix: just break instead of
return(NULL) for upward compatibility with more LC_* categories may be
implemented in future.

Prevent out of bounds writting for too many slashes case.
Replace strnpy + ='\0' with strlcpy

MFC after: 1 day

Remove the hard-coded limit of 3 bytes for EUC encodings.
Satoshi NIIMI-san kindly explained that EUC does not limit the byte length to
any arbitrary number.

We now set the limit to the maximum octet length of the codeset and it is
locale-specific.

Submitted by: Yong-Jhen Hong <winard@ms11.url.com.tw>

Install digittoint.3 (forgotten in rev 1.21)

PR: docs/26451
Submitted by: Adrian Filipi-Martin <adrian@ubergeeks.com>

This was recently MFC'd, so it will appear in 4.6.

PR: 37018

Fix EUC encoding conversion for codeset 3 and 4 to comply to the specification.

PR: 28552
Submitted by: NIIMI Satoshi <sa2c@and.or.jp>

Do not use __progname directly (except in [gs]etprogname(3)).
Also, make an internal _getprogname() that is used only inside
libc. For libc, getprogname(3) is a weak symbol in case a
function of the same name is defined in userland.

Fix the style of the SCM ID's.
I believe have made all of libc .c's as consistent as possible.

Remove __P() usage.

mdoc(7) police: tidy up.

bde got caught by mdoc(7) police. :-)

mdoc(7) police: don't you notice the warnings?

o Move NTOHL() and associated macros into <sys/param.h>. These are
deprecated in favor of the POSIX-defined lowercase variants.
o Change all occurrences of NTOHL() and associated marcros in the
source tree to use the lowercase function variants.
o Add missing license bits to sparc64's <machine/endian.h>.
Approved by: jake
o Clean up <machine/endian.h> files.
o Remove unused __uint16_swap_uint32() from i386's <machine/endian.h>.
o Remove prototypes for non-existent bswapXX() functions.
o Include <machine/endian.h> in <arpa/inet.h> to define the
POSIX-required ntohl() family of functions.
o Do similar things to expose the ntohl() family in libstand, <netinet/in.h>,
and <sys/param.h>.
o Prepend underscores to the ntohl() family to help deal with
complexities associated with having MD (asm and inline) versions, and
having to prevent exposure of these functions in other headers that
happen to make use of endian-specific defines.
o Create weak aliases to the canonical function name to help deal with
third-party software forgetting to include an appropriate header.
o Remove some now unneeded pollution from <sys/types.h>.
o Add missing <arpa/inet.h> includes in userland.

Tested on: alpha, i386
Reviewed by: bde, jake, tmm

Do not try to convert to char already converted C monetary locale members.
Do this conversion on locale load stage instead.

get __time_load_locale() prototype from include file, rather than declare
own

Restore C99 standard conformance information, isblank() _is_ in final
standard document

Pointed by: "Jacques A. Vidrine" <n@nectar.cc>

Replaced bogus cross references by the usual one for the ctype family
(ctype(3)).

Removed assertion that isblank() conforms to C90 too. This assertion
is correct but less than useful. There is some uncertainty about whether
isblank() is in C99, but it is certainly not in C90. It just conforms
to C89 because it is a conforming extension.

Fixed unsorting of almost all lists in previous commit.

Removed assertion that isblank() is in C99 here too.

Fixed unsorting of MLINKS in previous commit.
Fixed unsorting of SRCS in rev.1.18.

Remove assertion that isblank() is in C99, pointed out by ache.

From the PR:

1. ctype.h defines digittoint(), isnumber() and ishexnmber(), yet
they are not documented in any of the manpages.

2. The ctype manpage references a non-existent manpage for
digittoint().

3. The isascii() manpage claims it is standards compliant, when
it isn't.

4. isblank() claims it is _not_ standards compliant, when it
is.

Fix by including the appropriate .Nm entries, and with a new digittoint.3
page.

PR: docs/26451
Submitted by: Adrian Filipi-Martin <adrian@ubergeeks.com>

Back out recent replacement of LC_MESSAGES file with directory.

Requested by: ache

Slightly re-work locale messages storage scheme. Before this commit
LC_MESSAGES related data was installed to <locale>/LC_MESSAGES file.
Now it go to <locale>/LC_MESSAGES/SYS_LC_MESSAGES file. LC_MESSAGES
directory is supposed to be storage of message catalogs of userland tools.
This should allow us to avoid many potential problems with future
libintl related functionality introduction.

Thanks for useful suggestions about correct way how to replace plain
files with directories at installworld stage to: Ruslan Ermilov <ru>

style(9)'ify

Add my e-mail to copyrights

mdoc(7) police: use no-break space.

mdoc(7) police: use no-break space, fix markup.

mdoc(7) police: use non-break space, remove whitespace at EOL, fix markup.

* Add my e-mail to copyrights
* style(9)'ify

Fix grouping string handling

Clarify ' ' space issue

Remove specific reference to ASCII space (' '), it is true for localized
spaces too

Clarify isblank range

Clarify valid isspace() range

Clarify that is[x]digit() class is the same in any locale

mdoc(7) police: Use the new .In macro for #include statements.

First appeared in 5.0, not in 4.4

mdoc(7) police:

Avoid using parenthesis enclosure macros (.Pq and .Po/.Pc) with plain text.
Not only this slows down the mdoc(7) processing significantly, but it also
has an undesired (in this case) effect of disabling hyphenation within the
entire enclosed block.

Remove whitespace at EOL.

mdoc(7) police: fixed bugs in rev. 1.19, split section headers names.

Return "" if reallocf() fails

Describe success return value

Add transition period hack allowing old locale names return proper codeset too

Handle "ASCII" and "US-ASCII" aliases

add nl_langinfo(3)

Eliminate BUGS section. No one of listed bugs is applicable to FreeBSD-current
anymore.

add manpage for nl_langinfo(3)

Reviewed by: ru

Include <unistd.h> so that read(2) and write(2) don't cause warnings.

mdoc(7) police: LIBRARY should be before SYNOPSIS.

MAN[1-9] -> MAN.

Implement D_MD_ORDER (local extension) to get month/day order from locale

Use our standard .c rcsid format.

Fix copyright breakage in rev 1.2.
We *cannot* remove clause #4 from the Univ of California's license.

Change mon_decimal_point from "." to "" (N/A>) as it is specified by POSIX for
POSIX locale.

Actually implement T_FMT_AMPM

Removed duplicate $FreeBSD$.

Fix setlocale() to conform to the ISO C and POSIX standards.
The below text is quoted from the latest POSIX draft:

: The values of locale categories shall be determined by a precedence
: order; the first condition met below determines the value:
:
: 1. If the LC_ALL environment variable is defined and is not null,
: the value of LC_ALL shall be used.
: 2. If the LC_* environment variable (LC_COLLATE, LC_CTYPE, LC_MESSAGES,
: LC_MONETARY, LC_NUMERIC, LC_TIME) is defined and is not null, the
: value of the environment variable shall be used to initialize the
: category that corresponds to the environment variable.
: 3. If the LANG environment variable is defined and is not null, the
: value of the LANG environment variable shall be used.
: 4. If the LANG environment variable is not set or is set to the empty
: string, the implementation-defined default locale shall be used.

The conditions 1 and 2 were interchanged, i.e., LC_* were looked first,
then LC_ALL, then LANG (note that LC_ALL and LANG were essentially the
same, providing the default, with LC_ALL taking precedence over LANG).
Now, LC_ALL and LANG serve the different purposes. LC_ALL overrides
any LC_*, and LANG provides the default fallback.

Testcase:

/usr/bin/env LC_ALL=C LC_TIME=de_DE.ISO_8859-1 /bin/date

Should return date in the "C" locale format.

Inspired by: date(1) reference page in the Draft

s/fstat/_fstat/

Approved by: phantom

.St -ansiC -> .St -isoC

Fix visibility of empty variable -- it should be static.

Submitted by: bde and Hartmut Brandt <brandt@fokus.gmd.de> (via PR)
PR: bin/25308

cleanup commentaries

Deal properly with "0"

CRNCYSTR: determine '.' too

Implement CRNCYSTR

Return {YES,NO}STR from locale

Approved by: phantom

catch up to __part_load_locale() interface change

add additional function parameter: bufsize_min. it's possible
to check two sizes per one function invocation now.

Suggested by: ache

Make comparsions more clear (per style(9))

Assume that "" passed as parameter also means "no grouping"
Make comparsions more clear (per style(9))

o Fix build of libc broken in revision 1.2. offsetof() requires the
inclusion of stddef.h.

Reviewed by: peter

Don't use hardcoded struct size, use offsetof() instead (make size calculations
dynamic)

Don't try to convert grouping strings in case if C or POSIX locale
was explicitly specified.

Submitted by: ache

make it possible to specify grouping number from range 0..CHAR_MAX,
not only one-digit number

Use "namespace.h" and "un-namespace.h"

Requested by: deischen

. Fix semantics of grouping (LC_MONETARY::mon_grouping,
LC_NUMERIC::grouping) values.
. Always set __XXX_changed flags then loading numeric & monetary locale
categories to allow localeconv() to use C locale also.

As temporary workaround for missing *grouping fields parser always return "no
grouping" (CHAR_MAX, '\0').
Fixme: grouping parser needs to be implemented.

According to Garrett, POSIX widely use -1 to indicate CHAR_MAX, so back out
all my "-1" -> "something" fixes and replace -1 with CHAR_MAX directly in
strtol() in cnv()

Use __XSTRING(CHAR_MAX) instead of "127" and strtol() base 0 to parse it (0x7f)

Correct myself a bit: situation is broken not for _all_ numeric LC_MONETARY,
LC_NUMERIC fields, but only for *grouping fields - other fields are converted
to a chars in localeconv(), so final change is:

"-1" -> "127"

127 here is because CHAR_MAX supposed, which is _positive_ (SUSv2 requirement),
not negative as 255. It is still a bit of hack. To find real CHAR_MAX will be
better to sprintf() it once somewhere in static buffer. *grouping parsing
still broken and missing and needs to be implemented.

NOTE: according to SUSV2 and other implementations, numeric elements in
LC_MONETARY, LC_NUMERIC are byte-arrays, not ASCII strings!

Fix "C" locale, change "-1" to {CHAR_MAX, '\0'} according to standards.

This is only partial fix - locale loading procedure remains broken as before
and load too big values for all locales. All numeric strings there should be
converted with something like atoi() and placed into bytes. Maybe I do it
later, if someone will not fix it faster.

Implement CODESET

Back out attempt to implement CRNCYSTR - require additional prefixes according
to SUSV2. Add comment explaining it instead.

Explicitly mark deprecated entries.
Return currency_symbol for CRNCYSTR
Return "%r" for T_FMT_AMPM
Remove obsoleted comment about c_fmt
Return "" for {YES,NO}STR

add lmonetary.?, lnumeric.?, ldpart.?, lmessages.?, nl_langinfo.c
remove lconv.c

Add SUSv2 compatible nl_langinfo() function. It still need some work, but
this is already usable one.

Make localeconv() actual function. Now it will use LC_MONETARY/LC_NUMERIC
information to fill return structure. Remove unused anymore stub.

Make FreeBSD locale support complete: add support for rest locale categories
LC_MONETARY, LC_NUMERIC and LC_MESSAGES. Remove stub functions since they
don't need anymore.

Reviewed by: silence on -i18n

mdoc(7) police: split punctuation characters + misc fixes.

Remove _THREAD_SAFE and make libc thread-safe by default by
adding (weak definitions to) stubs for some of the pthread
functions. If the threads library is linked in, the real
pthread functions will pulled in.

Use the following convention for system calls wrapped by the
threads library:
__sys_foo - actual system call
_foo - weak definition to __sys_foo
foo - weak definition to __sys_foo

Change all libc uses of system calls wrapped by the threads
library from foo to _foo. In order to define the prototypes
for _foo(), we introduce namespace.h and un-namespace.h
(suggested by bde). All files that need to reference these
system calls, should include namespace.h before any standard
includes, then include un-namespace.h after the standard
includes and before any local includes. <db.h> is an exception
and shouldn't be included in between namespace.h and
un-namespace.h namespace.h will define foo to _foo, and
un-namespace.h will undefine foo.

Try to eliminate some of the recursive calls to MT-safe
functions in libc/stdio in preparation for adding a mutex
to FILE. We have recursive mutexes, but would like to avoid
using them if possible.

Remove uneeded includes of <errno.h> from a few files.

Add $FreeBSD$ to a few files in order to pass commitprep.

Approved by: -arch

Prepare for mdoc(7)NG.

mdoc(7) police: Er macro usage cleanup.

Disallow '/' characters in LC_* environment variables which might
be used to point to a bad locale file. This is only believed to be a
minor security risk - the only risk is if some program uses the result
of a localized string as a format specifier in a vulnerable function
like sprintf(). No such code is believed to exist in the FreeBSD base
system, although it is possible that badly written third party code
would do that.

Submitted by: imp
Approved by: ache

The comparison against 0 should be against LC_ALL. category isn't a
boolean and it is LC_ALL that's special.

Someone submitted this to me a long time ago, but I can't find the
mail now.

fix comment

Describe agrument range correctly, according to multibyte(3)
Remove unneded comment

Megre XPG4 code into libc

mdoc related fixes:
. synchronize NAME and SYNOPSIS sections
. replace .Ev macros with .Dv / .Er / .Em macros as mdoc(7)
specification declare

Introduce ".Lb" macro to libc manpages.

More libraries manpages updates following.

Add comment after locales
Use .Li for type

Suggested-by: sheldonh

Better wording according to multibyte(3)
Better man formatting
Add reference to multibyte(3)

Back out valid argument domain change - sneak to this function by error.
Reword test condition better. Previous variant was true for negative
characters too.

Describe valid argument domain for 8-bit wide locales to prevent common error
calling ctype functions with signed char as an argument.

There is a problem in that one cannot use ctype.h at the same time as parts
of the C++ stdlib. Our ctype.h uses symbols of the form _<X> to denote the
various character classes. Our ctype.h also extends the usual ctype.h
offering by adding the "_T" (special) class. Problem is parts of the STL
also use the symbol "_T" as its parameterized type. These two uses are
incompatible.

Thus change the form of the symbols used in ctype to something that fixes
the current problem and is less likely to cause conflicts in the future.

Requested by: Tomoaki NISHIYAMA <tomoaki@biol.s.u-tokyo.ac.jp>
Ok'ed by: JKH

Simplify sytem call renaming. Instead of _foo() <-- _libc_foo <-- foo(),
just use _foo() <-- foo(). In the case of a libpthread that doesn't do
call conversion (such as linuxthreads and our upcoming libpthread), this
is adequate. In the case of libc_r, we still need three names, which are
now _thread_sys_foo() <-- _foo() <-- foo().

Convert all internal libc usage of: aio_suspend(), close(), fsync(), msync(),
nanosleep(), open(), fcntl(), read(), and write() to _foo() instead of foo().

Remove all internal libc usage of: creat(), pause(), sleep(), system(),
tcdrain(), wait(), and waitpid().

Make thread cancellation fully POSIX-compliant.

Suggested by: deischen

Although it should be obvious that the 3-digit numeric values of the
characters shown are octal, state this explicitly for the easily
misled.

Add three-tier symbol naming in support of POSIX thread cancellation
points. For library functions, the pattern is __sleep() <--
_libc_sleep() <-- sleep(). The arrows represent weak aliases. For
system calls, the pattern is _read() <-- _libc_read() <-- read().

Add .Xrefs to tolower.3 and toupper.3, respectively.

Back up following macros by functions: ishexnumber, isideogram, isnumber,
isphonogram, isrune, isspecial. Fix ordering.

Reviewed by: bde

Correct "standard compilance" notes

Reminded by: bde

Fix dead loop if locale contains / and not all categories specified

PR: 14742
Submitted by: peter@wahoo.com.tw

mdoc(7)'fy

__collate_substitute() do something non-trivial only for German. For everyone
else, it is equivalent to strdup(). So, we will check if the substitution
tables are trivial at the load time, and possibly save 2 calls to
__collate_substitute() in strcoll().

Still, __collate_substitute() should not exist.

Reduce time of __collate_substitute() from O(strlen(s)^2) to O(strlen(s)).
Other minor optimizations. I got ~30% speedup in strcoll() for 50 char strings,
~40% speedup for 100 char strings, and unmeasurable speedup for 1M strings.

Collates are still terribly slow. To make them reasonable fast,
__collate_substitute() should be killed.

$Id$ -> $FreeBSD$

Add $Id$, to make it simpler for members of the translation teams to
track.

The $Id$ line is normally at the bottom of the main comment block in the
man page, separated from the rest of the manpage by an empty comment,
like so;

.\" $Id$
.\"

If the immediately preceding comment is a @(#) format ID marker than the
the $Id$ will line up underneath it with no intervening blank lines.
Otherwise, an additional blank line is inserted.

Approved by: bde

Correct troff sequence for backslashes in manual page.

PR: docs/12322
Submitted by: Marc Ramirez <mrami@gbtb.com>

PR: 10918
Submitted by: Yung-Jen Hung <winard@u3717a.dorm.ccu.edu.tw>
Reviewed by: bearscorp.bbs@bbs.life.nthu.edu.tw
_BIG5_sgetrune() in libc doesn't work well, this commit will fix it.

Document ishexnumber, isideogram, isnumber, isphonogram, isrune,
and isspecial.

PR: docs/9854

More egcs warning fixes:
o use braces to avoid potentially ambiguous else
o don't default to type int (and also remove a useless register
modifier).
o Use parens around assignment values used as truth values.
o Remove unused function.

Reviewed by: obrien and chuckr

Fix a macro name typo that made a word disappear.

Fixed tab lossage in previous commit.

remove ctype.c - unused and confusing

fix unsigned overflow
PR: 8437

Replace memory leaking instances of realloc with non-leaking reallocf.
In some cases replace if (a == null) a = malloc(x); else a =
realloc(a, x); with simple reallocf(a, x). Per ANSI-C, this is
guaranteed to be the same thing.

I've been running these on my system here w/o ill effects for some
time. However, the CTM-express is at part 6 of 34 for the CAM
changes, so I've not been able to do a build world with the CAM in the
tree with these changes. Shouldn't impact anything, but...

Sort cross references.

Add missing #include of <sys.types.h>

I have added the support for BIG5 encoding into libc/libxpg4/mklocale.
the diff is attached below. This is done on the 3.0 source-tree.
I have test this on 2.2-stable before, but I don't have a 3.0 machine
right now.

This patch is mainly to make libc support BIG5 encoding, thus add
zh_TW.BIG5 locale to 3.0.

Submitted by: Chen Hsiung Chan <frankch@waru.life.nthu.edu.tw>

Add reference to catopen(3)

Basic support for LC_MESSAGES

Change MACHINE references to MACHINE_ARCH.

Add #ifndef __NETBSD_SYSCALLS around calls to issetugid() which
do not exist in NetBSD 1.3.

Include string.h for memcpy function prototype.

Expanded cross references.

The terminating character in strings is `NUL', not `NULL'.

Correct description of which runes are encoded as two bytes.

PR: 4555
Submitted by: Dmitrij Tejblum <tejblum@arc.hq.cti.ru>

[0x0400 - 0xffff] [bbbbbbbb.bbbbbbbb] -> 1110bbbb, 10bbbbbb, 10bbbbbb
.Ed
.Pp
If more than a single representation of a value exists (for example,
0x00; 0xC0 0x80; 0xE0 0x80 0x80) the shortest representation is always
used (but the longer ones will be correctly decoded).
.Pp
The final three encodings provided by X-Open:
.Bd -literal
[00000000.000bbbbb.bbbbbbbb.bbbbbbbb] ->
11110bbb, 10bbbbbb, 10bbbbbb, 10bbbbbb

[000000bb.bbbbbbbb.bbbbbbbb.bbbbbbbb] ->
111110bb, 10bbbbbb, 10bbbbbb, 10bbbbbb, 10bbbbbb

[0bbbbbbb.bbbbbbbb.bbbbbbbb.bbbbbbbb] ->
1111110b, 10bbbbbb, 10bbbbbb, 10bbbbbb, 10bbbbbb, 10bbbbbb
.Ed
.Pp
which provides for the entire proposed ISO-10646 31 bit standard are currently
not implemented.
.Sh "SEE ALSO"
.Xr mklocale 1 ,
.Xr setlocale 3
@


1.4
log
@Don't use hardcoded *roff font change requests. Do it
via mdoc macros instead.
@
text
@d37 1
a37 1
.Dd "June 4, 1993"
@


1.3
log
@Very minor mdoc cleanup.
@
text
@d44 2
a45 1
\fBENCODING "UTF2"\fP
@


1.2
log
@Another round of various man page cleanups.
@
text
@d65 1
a65 1
.sp
d81 1
a81 1
.sp
@


1.2.2.1
log
@YAMFC:

Commit all of the -current changes that apply to 2.2. These fall into
several categories:

- Cosmetic/mdoc changes. They don't really afect the output
at all, but having them in 2.2 will make it easier to diff the man
pages later when looking for real changes.
- Update some man pages to reflect the current 2.2 header files.
- Sort xrefs.
- A few typo fixes.
- And a few changes that actualy added text to the man page that should
be reflected in 2.2.
- Add some missing MLINKS.

Requested by: bde
@
text
@d44 1
a44 2
.Nm ENCODING
.Qq UTF2
d65 1
a65 1
.Pp
d81 1
a81 1
.Pp
@


1.2.2.2
log
@MFC: Just the locale fixes (small doc tweaks for the most part)
and the new strptime(3) call. Having added something, does this
require a version bump? Haven't we bumped once already?

There are a *LOT* of additional 3.0 changes to be merged but I'm not
entirely comfortable with some of them so I'll take the conservative
(read: cowardly :) way out and just merge this much.
@
text
@d37 1
a37 1
.Dd June 4, 1993
@


1.1
log
@Initial revision
@
text
@d41 1
a41 1
.Nm UTF2
@


1.1.1.1
log
@BSD 4.4 Lite Lib Sources
@
text
@@


1.1.1.1.6.1
log
@Phase 2 of merge - also fix things broken in phase 1.
Watch out for falling rock until phase 3 is over!

libc completely merged except for phkmalloc & rfork (don't know if David
wants that).

Some include files in sys/ had to be updated in order to bring in libc.
@
text
@d41 1
a41 1
.Nm utf2
@


1.1.1.1.6.2
log
@This 3rd mega-commit should hopefully bring us back to where we were.
I can get it to `make world' succesfully, anyway!
@
text
@d41 1
a41 1
.Nm UTF2
@

Sorted lists.

Removed the subdirectory paths from the definitions of MAN[1-9]. They
were a workaround for limitations in bsd.man.mk that were fixed about
2 years ago.

Use revived __maskrune for digittoint
Minor formatting

Move it under XPG4 define

Move MSKanji under XPG4 define

__maskrune --> __istype

Submitted by: Sin'ichiro MIYATANI / Phase One, Inc <siu@phaseone.co.jp>
Basic support for the Shift JIS encoding of japanese.
(and one tiny typo fixed in a comment)

Remove collate_range_cmp, was left for temp. backward compatibility

Fixed quoting of backslash.

Show the real revision date and not the date that this
manpage is being viewed.

Completely remove #ifdefed out 8859-1 extension, I found it break
POSIX, C locale definition, see LC_CTYPE pre-defined table there

#ifdef out C locale extension to 8859-1 encoding, it now stays to ASCII
back as designed in *BSD

Also it not violates current standards but

1) No other Unixes have this feature

2) It broke Kerberos5 (isprint) and God knows what else
(not all vendors will agree to treat FreeBSD as special case for support
since (1))

2) Give false localization sense (programs mimic to be 8859-1
localized) which prevents true localization.

Changed all paths to be relative to src/lib instead of src/lib/libc
so that all these makefiles can be used to build libc_r too.

Added .if ${LIB} == "c" tests to restrict man page builds to libc
to avoid needlessly building them with libc_r too.

Split libc Makefile into Makefile and Makefile.inc to allow the
libc_r Makefile to include Makefile.inc too.

Don't use literal semicolons in .Fn macro invocations.

Restore PATH_LOCALE functionality using issetugid() call now

FIxed arg types (mostly missing consts) in synopsis.

Revert $FreeBSD$ to $Id$

Move _PathLocale to data-only file, so setrunelocale() not pick up
whole setlocale.c module now.

Should go into 2.2

Do Andrey's homework :) before merging this into 2.2:

. add idempotency #ifdef
. avoid sloppy common-style external declaration.

Comment out PATH_LOCALE reference

Should go into 2.2

Use symbolic constants instead of hardcoded digits
Add range check for setrunelocale since it can be called
directly.
Remove _startup_setlocale compatibility function

Should go into 2.2

Fix yet another setlocale() bug.
Submitted by: Wojtek Pilorz <wpilorz@celebris.bdk.lublin.pl>

Update the comment why range checking not needed

Fix setrunelocale fail if called directly without prior setlocale
call

Should go in 2.2

Killed getenv of PATH_LOCALE per Andrey's suggestion. It was intended
for debugging locale stuff, but was rarely if ever used...and of course
just bit a big chunk out of our collective hind-ends.

Add yet one comment saying that range checking already done
to stop people "fixing" it by snprintf

Don't use hardcoded *roff font change requests. Do it
via mdoc macros instead.

Very minor mdoc cleanup.

Sort cross references.

Make the long-awaited change from $Id$ to $FreeBSD$

This will make a number of things easier in the future, as well as (finally!)
avoiding the Id-smashing problem which has plagued developers for so long.

Boy, I'm glad we're not using sup anymore. This update would have been
insane otherwise.

Add a couple of additional xrefs.

Add comment that range checking is already done at upper level
Kill snprintf left in collate.c from previous backout

Should go in 2.2

Back out rev 1.5: the overflow condition is already handled elsewhere.

Fix yet another buffer overflow. :-(

Vulnerable: all programs that use setlocale(LC_COLLATE),
setlocale(LC_CTYPE), or setlocale(LC_ALL). The only setuid/setgid
binary i've found for this is w(1).

Should go into 2.2.

Add rcsid[]

Since locale reading code not resistent against stack overflowing or
similar intruder attacks, don't allow PATH_LOCALE env variable action
for s-bit programs (non-standard locale path setting)

Add Id
Optimize/improve recently added locale restoration on failure

PATH_LOCALE: use this non-standard env variable first time only, i.e.
strdup() it to prevent unsetenv() or setenv() effects. Check its length to
not allow user to overflow internal locale buffer. Move PATH_LOCALE
handling code into one place.

POSIX: make better stub for LC_MONETARY & LC_NUMERIC, now it check
locale directory existance instead of refusing all non-C non-POSIX
locales. POSIX treats empty locale env variable as unset variable
while our old code treats it as "C" locale, fix it. Implement previous locale
restoring, if locale setting fails. Old code assumes success if some
of LC_ALL subset is successed even other fails, POSIX treats it as
failure with previous locale restoring, fix it.

Remove unneccessary length checking in currentlocale()

Rename collate_range_cmp to __collate_range_cmp for internal usage
inside libc. Add collate_range_cmp as alias to __collate_range_cmp
for temp. backward compatibility.
collate_range_cmp will be replaced with direct code for each
external program for compatibility with the rest of world

Simplify debug output
Simplify collate_range_cmp for ASCII-compatible collate we have now

Save half of space in LC_COLLATE and remove unneded code.
This change is not compatible with previous variant, however proper
error code returned in both cases.
Colldef changes will follows.

Add comment explaining what function does
Cover strcoll return 0 case too

Don't use __dead or __pure in user code. They were obfuscations
for gcc >= 2.5 and no-ops for gcc >= 2.6. Converted to use __dead2
or __pure2 where it wasn't already done, except in math.h where use
of __pure was mostly wrong.

Correctly use .Fn instead of .Nm to reference function names
in a bunch of man pages.

Use the correct .Bx (BSD UNIX) or .At (AT&T UNIX) macros
instead of explicitly specifying the version in the text
in a bunch of man pages.

Add Id
Move comment up to place

Add isblank attr to non-break space
Add comment describing how upper half of table made

Remove isgraph attr from non-break space

Add Id

simplify/speedup/extend

Remove old version hooks

There is so many places where range comparation (using collate)
needed (much more than I think initially), so I forced to add
new user-visible non-standard function to libc.

Add internal function __collcmp once instead of adding it statically
to many places in the libc

Remove some hardcode *roff font change escape codes from
this man page to prevent half of it from coming out with underlines.

This man page needs to be gone over to fully convert it to mdoc format.

This closes PR#1440.

Submitted by: Jens Schweikhardt <schweikhardt@rus.uni-stuttgart.de>

Fixed longstanding namespace convolution involving rune_t vs wchar_t.
If _ANSI_SOURCE or _POSIX_SOURCE is defined, then <ctype.h> had to
be included before <stddef.h> or <stdlib.h> to get rune_t declared.
Now rune_t is declared perfectly bogusly in all cases when <ctype.h>
is included.

This change breaks similar (but more convoluted) convolutions in the
stddef.h in gcc distributions. Ports of gcc should avoid using the
gcc headers.

Don't include <sys/types.h> when it isn't used.

This commit covers most of the ANSI library functions. Many others only
need <sys/types.h> because they use u_xxx.

Fix error in wcstombs: byte count not counted
Remove unneded casts in sgetrune/sputrune
Submitted by: wcstombs fix by Mihoko Tanaka <m_tonaka@pa.yokogawa.co.jp>

Added missing section numbers to a bunch of .Xr macros, or
converted them into .Fn macros where appropriate. Also fixed
up some minor formatting problems.

Remove half-dancing solution for signed chars to help finding
POSIXly-incorrect programs.

Remove half-dancing solution for signed chars to help finding
POSIXly-incorrect programs.

Correct a bunch of man page cross references and generally
try and silence "manck".

ncurses, rpc, and some of the gnu stuff are still a big mess, however.

Another round of various man page cleanups.

recording cvs-1.6 file death

Oops. I seem to have over-corrected with my last commit. It should be
right this time

Correct what seem to me to be some mistakes in the references and
standards sections. Also add a missing `,' to each file.

Move more stuff out to XPG4
Handle negative chars inside runetype/tolower/toupper

Treat empty encoding as "C" encoding

Optimize PathLocale handling.
Handle C/POSIX/"" properly.
Don't reset collate to C when it is unnecessary

Use fake (empty) startup_setlocale for XPG4

Reduce static binaries bloat by splitting out normally unused
setinvalidrune()

Migrate from XPG4 to XPG3 (libxpg4 will be added soon)
Remove big part of my startup_setlocale hack.
Add missing manpage links.

As above.
Oh I also put in a couple of compile-time warnings for the crypt stuff.

Propogate default table to restricted 8859-1 template
(only control/graph/punct/print tags used)
Based on: kaleb@x.org and Bruce suggestions

Forget to close file
Submitted by: SANETO Takanori sanewo@strg.sony.co.jp

Fix cut&paste error: LC_COLLATE should be LC_TIME

Add time locale loading

Implement locale-sensitive strftime () from ADO (heavily modified
by me). This probably loses for multibyte characters, but I have no
way of telling. I'll let ache decide whether to add this support to
startup_setlocale. Note that for this to make any sense at all, the
symlinks in /usr/share/locale must go. (For the moment, this doesn't
make any difference since there are no locales supplied.)

Obtained from: Arthur David Olson <ado@elsie.nci.nih.gov>

Remove trailing whitespace.

Reviewed by: ache and wollman (long ago)

isctype.c:
o The tolower() and toupper() functions duplicated too much code
and were out of date (surprise). This didn't matter because
it was difficult to call them.
o Change formatting to be more like that in <ctype.h> (with
extra parentheses as in the macros). Perhaps this file should
be machine generated or everything should be handled like
__tolower() so that no code is repeated.

nomacros.c:
o Instead of looking at _USE_CTYPE_INLINE_ to see what <ctype.h>
has done, set _EXTERNALIZE_CTYPE_INLINES_ to tell <ctype.h>
what to do, so that we don't have anything left to do. Note
that code is now generated even if inlines are used by default.
This allows users to switch to non-inline versions.

Update info about LC_COLLATE implementation

Minor optimization

Don't pick _warn module now

Add 8-bit collate stuff
Submitted by: alex@elvisti.kiev.ua

Sync with ctype.h (remove EOF handling)

Sync with ctype.h (EOF, sign extention fixes)

Make not-so-space-eaten locale version:
split modules to bring only neccessary functions,
eliminate sprintf, make reduced startup_locale version.

First crack at making libc work with the new make macros. It compiles on
my machine, and a simple static (genassym) and shared (sysctl) executable
both work. Still to be done: RPCand YP merge.

This commit was generated by cvs2svn to compensate for changes in r1573,
which included commits to RCS files with non-trunk default branches.

BSD 4.4 Lite Lib Sources