libthr: add pthread_sigqueue(3)

PR: 278459
Reviewed by: markj
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
Differential revision: https://reviews.freebsd.org/D44867

libthr: filter rather than link with libsys

The allows gcc + GNU ld to link programs with -m32 -pthread without
erroring out due to _umtx_op_err being undefined (unless -lsys is added
to the link command.

We now always link _umtx_op_err into libthr (not just when it's static)
and filter it with libsys so we call that implementation. The dynamic
implementations (at least the assembly ones) should likely become stubs
as a further refinement.

Reviewed by: kib
Differential Revision: https://reviews.freebsd.org/D43783

Remove $FreeBSD$: one-line .h pattern

Remove /^\s*\*+\s*\$FreeBSD\$.*$\n/

rtld: workaround for broken ABI

Right now, libthr does not initialize RtldLockInfo.rtli_version when calling
_rtld_thread_init(), which makes versioning the interface troublesome.

Add a workaround: if the calling object of _rtld_thread_init() exports
the "_pli_rtli_version" symbol, then consider rtli_version initialized.
Otherwise, forcibly set it to RTLI_VERSION_ONE, currently defined as
RTLI_VERSION.

Export "_pli_rtli_version" from libthr and properly initialize rtli_version.

Sponsored by: The FreeBSD Foundation
MFC after: 1 week
Differential revision: https://reviews.freebsd.org/D29633

Add pthread_getname_np() and pthread_setname_np() aliases for
pthread_get_name_np() and pthread_set_name_np().

This re-applies r361770 after compatibility fixes.

Reviewed by: antoine, jkim, markj
Tested by: antoine (exp-run)
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
Differential revision: https://reviews.freebsd.org/D25117

Revert r361770 "Add pthread_getname_np() and pthread_setname_np() aliases" for now.

It is not compatible enough with Linux.

Requested by: antoine, jkim
Sponsored by: The FreeBSD Foundation

Add pthread_getname_np() and pthread_setname_np() aliases

for pthread_get_name_np() and pthread_set_name_np(), to be
compatible with Linux.

PR: 238404
Proposed and reviewed by: markj
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
Differential revision: https://reviews.freebsd.org/D25117

Add pthread_peekjoin_np(3).

The function allows to peek at the thread exit status and even see
return value, without joining (and thus finally destroying) the target
thread.

Reviewed by: markj
Sponsored by: The FreeBSD Foundation (kib)
MFC after: 2 weeks
Differential revision: https://reviews.freebsd.org/D23676

Fix initial exec TLS mode for dynamically loaded shared objects.

If dso uses initial exec TLS mode, rtld tries to allocate TLS in
static space. If there is no space left, the dlopen(3) fails. If space
if allocated, initial content from PT_TLS segment is distributed to
all threads' pcbs, which was missed and caused un-initialized TLS
segment for such dso after dlopen(3).

The mode is auto-detected either due to the relocation used, or if the
DF_STATIC_TLS dynamic flag is set. In the later case, the TLS segment
is tried to allocate earlier, which increases chance of the dlopen(3)
to succeed. LLD was recently fixed to properly emit the flag, ld.bdf
did it always.

Initial test by: dumbbell
Tested by: emaste (amd64), ian (arm)
Tested by: Gerald Aryeetey <aryeeteygerald_rogers.com> (arm64)
Sponsored by: The FreeBSD Foundation
MFC after: 2 weeks
Differential revision: https://reviews.freebsd.org/D19072

Add pthread_get_name_np(3).

The function retrieves the thread name previously set by
pthread_set_name_np(3). The name is cached in the process memory.

Requested by: Willem Jan Withagen <wjw@digiware.nl>
Man page update: Yuri Pankov <yuripv@yuripv.net>
Reviewed by: ian (previous version)
Discussed with: arichardson, bjk (man page)
Sponsored by: The FreeBSD Foundation
MFC after: 2 weeks
Differential revision: https://reviews.freebsd.org/D16702

Add implementation of robust mutexes, hopefully close enough to the
intention of the POSIX IEEE Std 1003.1TM-2008/Cor 1-2013.

A robust mutex is guaranteed to be cleared by the system upon either
thread or process owner termination while the mutex is held. The next
mutex locker is then notified about inconsistent mutex state and can
execute (or abandon) corrective actions.

The patch mostly consists of small changes here and there, adding
neccessary checks for the inconsistent and abandoned conditions into
existing paths. Additionally, the thread exit handler was extended to
iterate over the userspace-maintained list of owned robust mutexes,
unlocking and marking as terminated each of them.

The list of owned robust mutexes cannot be maintained atomically
synchronous with the mutex lock state (it is possible in kernel, but
is too expensive). Instead, for the duration of lock or unlock
operation, the current mutex is remembered in a special slot that is
also checked by the kernel at thread termination.

Kernel must be aware about the per-thread location of the heads of
robust mutex lists and the current active mutex slot. When a thread
touches a robust mutex for the first time, a new umtx op syscall is
issued which informs about location of lists heads.

The umtx sleep queues for PP and PI mutexes are split between
non-robust and robust.

Somewhat unrelated changes in the patch:
1. Style.
2. The fix for proper tdfind() call use in umtxq_sleep_pi() for shared
pi mutexes.
3. Removal of the userspace struct pthread_mutex m_owner field.
4. The sysctl kern.ipc.umtx_vnode_persistent is added, which controls
the lifetime of the shared mutex associated with a vnode' page.

Reviewed by: jilles (previous version, supposedly the objection was fixed)
Discussed with: brooks, Martin Simmons <martin@lispworks.com> (some aspects)
Tested by: pho
Sponsored by: The FreeBSD Foundation

If libthr.so is dlopened without RTLD_GLOBAL flag, the libthr symbols
do not participate in the global symbols namespace, but rtld locks are
still replaced and functions are interposed. In particular,
__pthread_map_stacks_exec is resolved to the libc version. If a
library is loaded later, which requires adjustment of the stack
protection mode, rtld calls into libc __pthread_map_stacks_exec due to
the symbols scope. The libc version might recurse into binder and
recursively acquire rtld bind lock, causing the hang.

Make libc __pthread_map_stacks_exec() interposed, which synchronizes
rtld locks and version of the stack exec hook when libthr loaded,
regardless of the symbol scope control or symbol resolution order.

The __pthread_map_stacks_exec() symbol is removed from the private
version in libthr since libc symbol now operates correctly in presence
of libthr.

Reported and tested by: markj
Sponsored by: The FreeBSD Foundation
MFC after: 2 weeks

Fix known issues which blow up the process after dlopen("libthr.so")
(or loading a dso linked to libthr.so into process which was not
linked against threading library).

- Remove libthr interposers of the libc functions, including
__error(). Instead, functions calls are indirected through the
interposing table, similar to how pthread stubs in libc are already
done. Libc by default points either to syscall trampolines or to
existing libc implementations. On libthr load, libthr rewrites the
pointers to the cancellable implementations already in libthr. The
interposition table is separate from pthreads stubs indirection
table to not pull pthreads stubs into static binaries.

- Postpone the malloc(3) internal mutexes initialization until libthr
is loaded. This avoids recursion between calloc(3) and static
pthread_mutex_t initialization.

- Reinstall signal handlers with wrapper on libthr load. The
_rtld_is_dlopened(3) is used to avoid useless calls to sigaction(2)
when libthr is statically referenced from the main binary.

In the process, fix openat(2), swapcontext(2) and setcontext(2)
interposing. The libc symbols were exported at different versions
than libthr interposers. Export both libc and libthr versions from
libc now, with default set to the higher version from libthr.

Remove unused and disconnected swapcontext(3) userspace implementation
from libc/gen.

No objections from: deischen
Tested by: pho, antoine (exp-run) (previous versions)
Sponsored by: The FreeBSD Foundation
MFC after: 1 week

Add accept4() system call.

The accept4() function, compared to accept(), allows setting the new file
descriptor atomically close-on-exec and explicitly controlling the
non-blocking status on the new socket. (Note that the latter point means
that accept() is not equivalent to any form of accept4().)

The linuxulator's accept4 implementation leaves a race window where the new
file descriptor is not close-on-exec because it calls sys_accept(). This
implementation leaves no such race window (by using falloc() flags). The
linuxulator could be fixed and simplified by using the new code.

Like accept(), accept4() is async-signal-safe, a cancellation point and
permitted in capability mode.

Introduce a non-portable function pthread_getthreadid_np(3) to retrieve
calling thread's unique integral ID, which is similar to AIX function of
the same name. Bump __FreeBSD_version to note its introduction.

Reviewed by: kib

Implement the __pthread_map_stacks_exec() for libthr.

Stack creation code is changed to call _rtld_get_stack_prot() to get
the stack protection right. There is a race where thread is created
during dlopen() of dso that requires executable stacks. Then,
_rtld_get_stack_prot() may return PROT_READ | PROT_WRITE, but thread
is still not linked into the thread list. In this case, the callback
misses the thread stack, and rechecks the required protection
afterward.

Reviewed by: davidxu

Revert revision 214007, I realized that MySQL wants to resolve
a silly rwlock deadlock problem, the deadlock is caused by writer
waiters, if a thread has already locked a reader lock, and wants to
acquire another reader lock, it will be blocked by writer waiters,
but we had already fixed it years ago.

A new jail(8) with a configuration file, to replace the work currently done
by /etc/rc.d/jail.

sort function name.

Add pthread_rwlockattr_setkind_np and pthread_rwlockattr_getkind_np, the
functions set or get pthread_rwlock type, current supported types are:
PTHREAD_RWLOCK_PREFER_READER_NP,
PTHREAD_RWLOCK_PREFER_WRITER_NONRECURSIVE_NP,
PTHREAD_RWLOCK_PREFER_WRITER_NP,
default is PTHREAD_RWLOCK_PREFER_WRITER_NONCECURSIVE_NP, this maintains
binary compatible with old code.

To support stack unwinding for cancellation points, add -fexceptions flag
for them, two functions _pthread_cancel_enter and _pthread_cancel_leave
are added to let thread enter and leave a cancellation point, it also
makes it possible that other functions can be cancellation points in
libraries without having to be rewritten in libthr.

Update symbol.

Add wrapper for setcontext() and swapcontext(), the wrappers
unblock SIGCANCEL which is needed by thread cancellation.

On shared object unload, in __cxa_finalize, call and clear all installed
atexit and __cxa_atexit handlers that are either installed by unloaded
dso, or points to the functions provided by the dso.

Use _rtld_addr_phdr to locate segment information from the address of
private variable belonging to the dso, supplied by crtstuff.c. Provide
utility function __elf_phdr_match_addr to do the match of address against
dso executable segment.

Call back into libthr from __cxa_finalize using weak
__pthread_cxa_finalize symbol to remove any atfork handler which
function points into unloaded object.

The rtld needs private __pthread_cxa_finalize symbol to not require
resolution of the weak undefined symbol at initialization time. This
cannot work, since rtld is relocated before sym_zero is set up.

Idea by: kan
Reviewed by: kan (previous version)
MFC after: 3 weeks

Create the altix project branch. The altix project will add support
for the SGI Altix 350 to FreeBSD/ia64. The hardware used for porting
is a two-module system, consisting of a base compute module and a
CPU expansion module. SGI's NUMAFlex architecture can be an excellent
platform to test CPU affinity and NUMA-aware features in FreeBSD.

Remove extra new semaphore stubs, because libc already has them, and
ld can find the newest version which is default.

Poked by: kan@

Use umtx to implement process sharable semaphore, to make this work,
now type sema_t is a structure which can be put in a shared memory area,
and multiple processes can operate it concurrently.
User can either use mmap(MAP_SHARED) + sem_init(pshared=1) or use sem_open()
to initialize a shared semaphore.
Named semaphore uses file system and is located in /tmp directory, and its
file name is prefixed with 'SEMD', so now it is chroot or jail friendly.
In simplist cases, both for named and un-named semaphore, userland code
does not have to enter kernel to reduce/increase semaphore's count.
The semaphore is designed to be crash-safe, it means even if an application
is crashed in the middle of operating semaphore, the semaphore state is
still safely recovered by later use, there is no waiter counter maintained
by userland code.
The main semaphore code is in libc and libthr only has some necessary stubs,
this makes it possible that a non-threaded application can use semaphore
without linking to thread library.
Old semaphore implementation is kept libc to maintain binary compatibility.
The kernel ksem API is no longer used in the new implemenation.

Discussed on: threads@

Make openat(2) a cancellation point.

This is required by POSIX and matches open(2).

Reviewed by: kib, jhb
MFC after: 1 month

- Import the HEAD csup code which is the basis for the cvsmode work.

Make pthread_cleanup_push() and pthread_cleanup_pop() as a pair of macros,
use stack space to keep cleanup information, this eliminates overhead of
calling malloc() and free() in thread library.

Discussed on: thread@

Add POSIX pthread API pthread_getcpuclockid() to get a thread's cpu
time clock id.

Add more cpu affinity function's symbols.

Implement functions pthread_getaffinity_np and pthread_setaffinity_np to
get and set thread's cpu affinity mask.

Yet another pointy hat: when I zapped FBSDprivate_1.1, I forgot to move
its contents to FBSDprivate_1.0.

Convert pthread.map to the format expected by version_gen.awk, and modify
the Makefile accordingly; libthr now explicitly uses libc's Versions.def.

MFC after: 2 weeks

Remove incorrectly added FBSDprivate_1.1 namespace, and move symbols which
are new in FreeBSD 8 to the appropriate namespace.

Per discussion on -threads, rename _islocked_np() to _isowned_np().

Add pthread_mutex_islocked_np(), a cheap way to verify that a mutex is
locked. This is intended primarily to support the userland equivalent
of the various *_ASSERT_LOCKED() macros we have in the kernel.

MFC after: 2 weeks

Remove vfork() overloading, it is no longer needed.

1. Add function pthread_mutex_setspinloops_np to turn a mutex's spin
loop count.
2. Add function pthread_mutex_setyieldloops_np to turn a mutex's yield
loop count.
3. Make environment variables PTHREAD_SPINLOOPS and PTHREAD_YIELDLOOPS
to be only used for turnning PTHREAD_MUTEX_ADAPTIVE_NP mutex.

Add _pthread_mutex_init_calloc_cb() to libthr and libkse, so that malloc(3)
(part of libc) can use pthreads mutexes without causing infinite recursion
during initialization.

Enable symbol versioning by default. Use WITHOUT_SYMVER to disable it.
Warning, after symbol versioning is enabled, going back is not easy
(use WITHOUT_SYMVER at your own risk).

Change the default thread library to libthr.

There most likely still needs to be a version bump for at least the
thread libraries. If necessary, this will happen later.

Use C comments since we now preprocess these files with CPP.

1. Don't override underscore version of aio_suspend(), system(),
wait(), waitpid() and usleep(), they are internal versions and
should not be cancellation points.
2. Make wait3() as a cancellation point.
3. Move raise() and pause() into file thr_sig.c.
4. Add functions _sigsuspend, _sigwait, _sigtimedwait and _sigwaitinfo,
remove SIGCANCEL bit in wait-set for those functions, the signal is
used internally to implement thread cancellation.

Add symbol versions: FBSD_1.0 and FBSDprivate.

WARNS level 4 cleanup.

Remove mqueue and timer, now they are in librt.

1. Add SIGEV_THREAD notification for mq_notify.
2. Reuse current timer code and abstract some common code to
to support both timer and mqueue.

Follow the mistake in libpthread, the first version name in libpthread
is LIBTHREAD_1_0, but really it should be LIBPTHREAD_1_0.
Fix it so libmap.conf works again (it was broken by recent versioning
code in rtld_elf).

Add code to handle timer_delete(). The timer wrapper code is completely
rewritten, now timers created with same sigev_notify_attributes will
run in same thread, this allows user to organize which timers can
run in same thread to save some thread resource.

Add timer_create wrapper.

Put pthread_condattr_init sorted order.

Export following functions:
_pthread_mutexattr_getpshared
_pthread_mutexattr_setpshared
pthread_condattr_getpshared
pthread_condattr_setpshared
pthread_mutexattr_getpshared
pthread_mutexattr_setpshared

Sort function names.

Add function pthread_timedjoin_np, the function is similar with pthread_join
except the function will return ETIMEDOUT if target thread does not exit
before specified absolute time passes.

Add usleep to the map files.

Noticed by: davidxu

Add debugger event reporting support, current only TD_CREATE and TD_DEATH
events are reported.

Export pthread_condattr_getclock, pthread_condattr_setclock.

Import my recent 1:1 threading working. some features improved includes:
1. fast simple type mutex.
2. __thread tls works.
3. asynchronous cancellation works ( using signal ).
4. thread synchronization is fully based on umtx, mainly, condition
variable and other synchronization objects were rewritten by using
umtx directly. those objects can be shared between processes via
shared memory, it has to change ABI which does not happen yet.
5. default stack size is increased to 1M on 32 bits platform, 2M for
64 bits platform.
As the result, some mysql super-smack benchmarks show performance is
improved massivly.

Okayed by: jeff, mtm, rwatson, scottl