New futex(2) based rwlock implementation based on the mutex code.

This implementation reduces contention because threads no longer need
to spin calling sched_yield(2) before going to sleep.

Tested by many, thanks!

ok visa@, pirofti@

Move mutex, condvar, and thread-specific data routes, pthread_once, and
pthread_exit from libpthread to libc, along with low-level bits to
support them. Major bump to both libc and libpthread.

Requested by libressl team. Ports testing by naddy@
ok kettenis@

Use "volatile unsigned int" instead of _atomic_lock_t. The _atomic_lock_t
isn't the same size on all our architectures and should only be used for
spin locks.

ok visa@, mpi@

New mutex and condvar implementations based on futex(2).

Not enabled yet, it needs some SPINLOCK_SPIN_HOOK love and
some bumps.

Tested by many including sthen@ in a bulk.

ok visa@, sthen@, kettenis@, tedu@

Get rid of ticket support, replace "struct _spinlock" with "_atomic_lock_t".

ok tedu@

Remove _USING_TICKETS, it's defined as 0. No functional change.

ok tedu@ mpi@

Use a Thread Information Block in both single and multi-threaded programs.
This stores errno, the cancelation flags, and related bits for each thread
and is allocated by ld.so or libc.a. This is an ABI break from 5.9-stable!

Make libpthread dlopen'able by moving the cancelation wrappers into libc
and doing locking and fork/errno handling via callbacks that libpthread
registers when it first initializes. 'errno' *must* be declared via
<errno.h> now!

Clean up libpthread's symbol exports like libc.

On powerpc, offset the TIB/TCB/TLS data from the register per the ELF spec.

Testing by various, particularly sthen@ and patrick@
ok kettenis@

make pthread_barrier_wait behave more like it does on other platforms.
from Kari Tristan Helgason

Eliminate the need to explicitly invoke syscalls via their _thread_sys_*
aliases by using a macro REDIRECT_SYSCALL() to map the symbols. Apply
that to getthrid(), sysctl(), and issetugid() as well.

ok mpi@ beck@

Replace the malloc spinlock with a mutex. This lock is held over system calls
which run for many cycles and may even sleep. This leads to other threads
spinning for a long time waiting on the lock. Using a mutex means those
threads go to sleep and get woken up when the lock is released, which results
in a lot less CPU usage. More work is needed to improve the performance of
threaded code that suffers from malloc lock contention, but this diff makes
ports like Firefox significantly more usable.

Tested by many.

ok mpi@, guenther@, tedu@, jca@

Split the intra-thread functionality from kill(2) into its own syscall
thrkill(2), rolling the kill(2) syscall number with the ABI change to
avoid breaking binaries during during the transition. thrkill(2) includes
a 'tcb' argument that eliminates the need for locking in pthread_kill()
and simplifies pthread_cancel(). Switch __stack_smash_handler() to use
thrkill(2) and explicitly unblock SIGABRT.

Minor bump to both libc and libpthread: make sure you install a new kernel!

ok semarie@

ld.so no longer needs or uses a bind lock, so stop setting it. This
eliminates a chunk of complexity from the libpthread init and the fork
wrapper, as it was the bind lock that needed prebinding before use.

Instead of testing for __ELF__ and/or vax, leave out the bits for interfacing
with ld.so locking whenever building NOPIC

pointless use of __ELF__ noted by brad@
ok miod@

Delete the duplicated sched_{policy,param} members from the internal struct
pthread and instead use the values from the embedded struct pthread_attr.
For bonus points, pay attention to the sched_inherit attribute and possibly
set the values from the parent thread.

Problem noted by natano of bitrig.

Declare and document getthrid()

indirectly prodded by krw@

Use a flag on the pthread_t to indicate that the thread's stack was
allocated by the kernel's execve bits.

ok matthew@

Don't try to reuse _initial_thread in the fork() wrapper, as the
thread's existing handle must continue to be valid and it didn't
fully 'change' the thread handle anyway. For pthread_main_np(),
use a new flag, THREAD_ORIGINAL, to indicate that the flagged thread
is the original thread for *this* process.

Fixes some ConsoleKit failures according to aja@

shared semaphore fixes. first, eliminate the sem inside a sem indirection.
it's not needed (mindless holdover from earlier prototype). everything is
simpler and sem_init shared semaphores even work now (confirmed by sthen).
correct the einval checks to not deref a pointer before the null check.
in sem_open, if we created the semaphore, we need to initialize the spinlock.

interprocess semaphores ala sem_open. mostly following in the pattern
of shm_open. with some additions and fixes from zhuk.

Move the declaration of _thread_sys_sigprocmask() to rthread.h

THE MACHINE DEPENDENT CERROR CODE HAS HARD CODED OFFSETS INTO PTHREAD_T!
found the hard way by miod. resize and realign all the locks and
semaphores back to the way they were. sometimes i just can't win.

something's not quite right yet. ticket locks result in more CPU usage
and spinning in kernel. partially back out, but in a way that makes going
forward again easy.
seen by ajacoutot

cleanup and consolidate the spinlock_lock (what a name!) code.
it's now atomic_lock to better reflect its usage, and librthread now
features a new spinlock that's really a ticket lock.
thrlseep can handle both types of lock via a flag in the clock arg.
(temp back compat hack)
remove some old stuff that's accumulated along the way and no longer used.
some feedback from dlg, who is concerned with all things ticket lock.
(you need to boot a new kernel before installing librthread)

SEM_VALUE_MAX belongs in <limits.h> (via sys/syslimits.h)
struct sem --> struct __sem (doesn't belong in public namespace)
Zap pointless _KERNEL tests

ok deraadt@

We want to check that the dynamic linker is available at run-time, so
we should actually check for _DYNAMIC at run-time rather than checking
for __PIC__ at compile time, since the two are actually independent.

Problem and solution identified by guenther; minor tweaks by me.
ok guenther

Add pthread spinlock support.

Implementation, documentation and naive regression tests for:
- pthread_spin_init()
- pthread_spin_destroy()
- pthread_spin_lock()
- pthread_spin_trylock()
- pthread_spin_unlock()

Implementation okay guenther@, documentation okay jmc@.

Add pthread barrier support.

Implementation and documentation for:
- pthread_barrier_init()
- pthread_barrier_destroy()
- pthread_barrier_wait()
- pthread_barrierattr_init()
- pthread_barrierattr_destroy()
- pthread_barrierattr_getpshared()
- pthread_barrierattr_setpshared()

Currently only private barriers are supported.

Okay guenther@.

Remove pthread_suspend* and related functions. This is part of the
rthreads major library bump from last night. okay kettenis@

Add sem_timewait() and fix sem_wait()'s handling of signals, so
that it resumes waiting unless the thread was canceled. As part
of this, change the internal _sem_wait() function to return zero
on success and an errno value on failure instead of 1 on success
and zero on failure.

Add pthread_condattr_{get,set}clock(), requested by aja@
Add pthread_mutex_timedlock(), requested by dcoppa@

Fix previous commit: _rthread_init() was static.
Improve consistency of error naming

Fix the handling of the stackaddr, stacksize, and guardsize attributes:
don't try to merge values, round the sizes separately, and don't try to
unmap application-supplied stacks.
Copy from uthread the caching of default-sized stacks.
Have pthread_attr_init() and pthread_create() get the default attributes
from staticly allocated pthread_attr_t.
Cache the pagesize in _rthread_init() and provide a macro for rounding to it

based on suggestions from kettenis@ and tedu@, ok kettenis@

Add pthread_rwlockattr_{get,set}pshared() and
pthread_mutexattr_{get,set}kind_np()

tweaked diff from brad

Reimplement mutexes, condvars, and rwlocks to eliminate bugs,
particularly the "consume the signal you just sent" hang, and putting
the wait queues in userspace.

Do cancellation handling in pthread_cond_*wait(), pthread_join(),
and sem_wait().

Add __ prefix to thr{sleep,wakeup,exit,sigdivert}() syscalls; add
'abort" argument to thrsleep to close cancellation race; make
thr{sleep,wakeup} return errno values via *retval to avoid touching
userspace errno.

Split out the pthread_mutexattr_* functions from rthread_sync.c to
new file rthread_mutexattr.c. Add basic implementations of
pthread_mutexattr_{set,get}{protocol,prioceiling}

Requested by aja

Implement cancelation for the basic syscall cancelation points,
using previously allocated SIGTHR to interrupt in-process syscalls
and fixing the spelling of "cancelled" along the way.
Modeled on FreeBSD's libthr

Copy support for sched_get_priority_{min,max} from libpthread.
Requested by many to ease substitution of librthread for libpthread

Move <machine/spinlock.h> into rthread.h; strip out unnecessary #includes

Use __tfork, __get_tcb, and __set_tcb to have a real TCB and per-thread
errno. The ASM bits for _cerror are sketchy or missing for some archs
but that can be corrected in-tree.

Convert thrsleep() to an absolute timeout with clockid to eliminate a
race condition and prep for later support of pthread_condattr_setclock()

"get it in" deraadt@, tedu@, cheers by others

Convert thrsigdivert to (almost) be sigtimedwait by adding siginfo_t
and struct timespec * argument. sigtimedwait is just a one line
wrapper after this.

"get it in" deraadt@, tedu@, cheers by others

Change threxit() to take a pointer to a pid_t to zero out from the
kernel so that librthread can detect when a thread is completely
done with its stack without need a kqueue. The dying thread moves
itself to a GC list, other threads scan the GC list on pthread_create()
and pthread_join() and free the stack and handle once the thread's
thread id is zeroed.

"get it in" deraadt@, tedu@, cheers by others

Add f*lockfile() routines to librthread

ok kurt@

Increase default thread stack size to 256K on 32bit archs and 512K on
64bit archs to accommodate the growing number of ports that put large
buffers on the stack. Supported by many with no objections.

thrsigdivert() takes a sigset_t, not a pointer to one. Detected by
finally reading the lint warnings (sigh)

ok tedu@

Make sigwait() work correctly. In particular, it'll work when the
signal is already pending in the calling thread or the main thread
and there's no longer a race condition where the signal could be
diverted but sigwait() would still block. There were some off-by-one
errors too.

(The checking of the main thread's pending list is just until a
pending list for the process can be added to struct process. For
now, such signals end up as pending on the main thread.)

oks tedu@ and art@

- Add fork/vfork wrapper functions to reset state in the child process.
- Make an effort to protect important libc and ld.so critical areas during
the fork(2)/vfork(2) sys call.
- Add pthread_atfork(3) implementation based on Daniel Eischen's code.

Original diff by Philip Guenther <guenther at gmail.com> with some
additions and refinements by me. Positive test report from brad@ with
many kde apps. fork(2) and pthread_atfork(3) pthread regresses pass.
okay tedu@, kettenis@, marc@

add -Wstrict-prototypes -Wmissing-prototypes -Wsign-compare
Minor tweaks to compile with the above, primarily in fixing
the conflicts between semaphore.h and rthread.h
"i like the additional warnings" tedu@

Cleanup struct pthread and stack after thread exits. This version does
not use a separate reaper thread. ok tedu@ marc@

thread stack handling changes. Add guard zones and allow stack
size (and guard zone size) to be set using pthread_attr. Guard
zones are specified in bytes, but implemented in terms of a
page size.

OK Otto@

Add "print debug foo" function safe to use in signal handlers, example:
_rthread_debug(3, "foo = %s", foo);
where output is to stderr and will only be printed if the current
debug level is >= 3. Messages with a debug level of 0 are always
printed. The level can be set from the environment (not enabled yet)
or with gdb by modifying variable _rthread_debug_level.
"Fine with me" -Otto

Implement suspend/resume and creation of initially suspended threads.
With this, java seems to be operational. Also make threads_ready
non-static, which is needed for an upcoming diff. ok tedu@

use queue.h macros for thread house keeping; make some vars non-static
and fix a bug in thread creation error path. ok tedu@

prototype all the thread syscalls in rthread.h for now.
update for new thrwakeup that takes a count argument

more consistently use _rthread prefix for all not meant to be exported
interfaces that aren't static, and a few that are but which will change

Fix hppa ldcw alignment issue.

Help deraadt, tedu, kettenis
Ok tedu, kettenis

update copyright to 2005

add pthread_get/set_concurrency (useless for now)
add pthread_get/set_stack[addr] (info not used yet)

add pthread_cleanup_push and pthread_cleanup_pop

add bits for pthread_cancel. we don't really have cancellation points yet
but some of the functions are here now.

change keys to use table instead of list, makes a sane destructor implementation
possible

make the mutex unlock and sleep in pthread_cond_wait properly atomic

correct implementation of pthread_cond_signal. it doesn't raise the sem
value if there are no waiters.

add userland thread library. incomplete, but functional

Move mutex, condvar, and thread-specific data routes, pthread_once, and
pthread_exit from libpthread to libc, along with low-level bits to
support them. Major bump to both libc and libpthread.

Requested by libressl team. Ports testing by naddy@
ok kettenis@

Use "volatile unsigned int" instead of _atomic_lock_t. The _atomic_lock_t
isn't the same size on all our architectures and should only be used for
spin locks.

ok visa@, mpi@

New mutex and condvar implementations based on futex(2).

Not enabled yet, it needs some SPINLOCK_SPIN_HOOK love and
some bumps.

Tested by many including sthen@ in a bulk.

ok visa@, sthen@, kettenis@, tedu@

Get rid of ticket support, replace "struct _spinlock" with "_atomic_lock_t".

ok tedu@

Remove _USING_TICKETS, it's defined as 0. No functional change.

ok tedu@ mpi@

Use a Thread Information Block in both single and multi-threaded programs.
This stores errno, the cancelation flags, and related bits for each thread
and is allocated by ld.so or libc.a. This is an ABI break from 5.9-stable!

Make libpthread dlopen'able by moving the cancelation wrappers into libc
and doing locking and fork/errno handling via callbacks that libpthread
registers when it first initializes. 'errno' *must* be declared via
<errno.h> now!

Clean up libpthread's symbol exports like libc.

On powerpc, offset the TIB/TCB/TLS data from the register per the ELF spec.

Testing by various, particularly sthen@ and patrick@
ok kettenis@

make pthread_barrier_wait behave more like it does on other platforms.
from Kari Tristan Helgason

Eliminate the need to explicitly invoke syscalls via their _thread_sys_*
aliases by using a macro REDIRECT_SYSCALL() to map the symbols. Apply
that to getthrid(), sysctl(), and issetugid() as well.

ok mpi@ beck@

Replace the malloc spinlock with a mutex. This lock is held over system calls
which run for many cycles and may even sleep. This leads to other threads
spinning for a long time waiting on the lock. Using a mutex means those
threads go to sleep and get woken up when the lock is released, which results
in a lot less CPU usage. More work is needed to improve the performance of
threaded code that suffers from malloc lock contention, but this diff makes
ports like Firefox significantly more usable.

Tested by many.

ok mpi@, guenther@, tedu@, jca@

Split the intra-thread functionality from kill(2) into its own syscall
thrkill(2), rolling the kill(2) syscall number with the ABI change to
avoid breaking binaries during during the transition. thrkill(2) includes
a 'tcb' argument that eliminates the need for locking in pthread_kill()
and simplifies pthread_cancel(). Switch __stack_smash_handler() to use
thrkill(2) and explicitly unblock SIGABRT.

Minor bump to both libc and libpthread: make sure you install a new kernel!

ok semarie@

ld.so no longer needs or uses a bind lock, so stop setting it. This
eliminates a chunk of complexity from the libpthread init and the fork
wrapper, as it was the bind lock that needed prebinding before use.

Instead of testing for __ELF__ and/or vax, leave out the bits for interfacing
with ld.so locking whenever building NOPIC

pointless use of __ELF__ noted by brad@
ok miod@

Delete the duplicated sched_{policy,param} members from the internal struct
pthread and instead use the values from the embedded struct pthread_attr.
For bonus points, pay attention to the sched_inherit attribute and possibly
set the values from the parent thread.

Problem noted by natano of bitrig.

Declare and document getthrid()

indirectly prodded by krw@

Use a flag on the pthread_t to indicate that the thread's stack was
allocated by the kernel's execve bits.

ok matthew@

Don't try to reuse _initial_thread in the fork() wrapper, as the
thread's existing handle must continue to be valid and it didn't
fully 'change' the thread handle anyway. For pthread_main_np(),
use a new flag, THREAD_ORIGINAL, to indicate that the flagged thread
is the original thread for *this* process.

Fixes some ConsoleKit failures according to aja@

shared semaphore fixes. first, eliminate the sem inside a sem indirection.
it's not needed (mindless holdover from earlier prototype). everything is
simpler and sem_init shared semaphores even work now (confirmed by sthen).
correct the einval checks to not deref a pointer before the null check.
in sem_open, if we created the semaphore, we need to initialize the spinlock.

interprocess semaphores ala sem_open. mostly following in the pattern
of shm_open. with some additions and fixes from zhuk.

Move the declaration of _thread_sys_sigprocmask() to rthread.h

THE MACHINE DEPENDENT CERROR CODE HAS HARD CODED OFFSETS INTO PTHREAD_T!
found the hard way by miod. resize and realign all the locks and
semaphores back to the way they were. sometimes i just can't win.

something's not quite right yet. ticket locks result in more CPU usage
and spinning in kernel. partially back out, but in a way that makes going
forward again easy.
seen by ajacoutot

cleanup and consolidate the spinlock_lock (what a name!) code.
it's now atomic_lock to better reflect its usage, and librthread now
features a new spinlock that's really a ticket lock.
thrlseep can handle both types of lock via a flag in the clock arg.
(temp back compat hack)
remove some old stuff that's accumulated along the way and no longer used.
some feedback from dlg, who is concerned with all things ticket lock.
(you need to boot a new kernel before installing librthread)

SEM_VALUE_MAX belongs in <limits.h> (via sys/syslimits.h)
struct sem --> struct __sem (doesn't belong in public namespace)
Zap pointless _KERNEL tests

ok deraadt@

We want to check that the dynamic linker is available at run-time, so
we should actually check for _DYNAMIC at run-time rather than checking
for __PIC__ at compile time, since the two are actually independent.

Problem and solution identified by guenther; minor tweaks by me.
ok guenther

Add pthread spinlock support.

Implementation, documentation and naive regression tests for:
- pthread_spin_init()
- pthread_spin_destroy()
- pthread_spin_lock()
- pthread_spin_trylock()
- pthread_spin_unlock()

Implementation okay guenther@, documentation okay jmc@.

Add pthread barrier support.

Implementation and documentation for:
- pthread_barrier_init()
- pthread_barrier_destroy()
- pthread_barrier_wait()
- pthread_barrierattr_init()
- pthread_barrierattr_destroy()
- pthread_barrierattr_getpshared()
- pthread_barrierattr_setpshared()

Currently only private barriers are supported.

Okay guenther@.

Remove pthread_suspend* and related functions. This is part of the
rthreads major library bump from last night. okay kettenis@

Add sem_timewait() and fix sem_wait()'s handling of signals, so
that it resumes waiting unless the thread was canceled. As part
of this, change the internal _sem_wait() function to return zero
on success and an errno value on failure instead of 1 on success
and zero on failure.

Add pthread_condattr_{get,set}clock(), requested by aja@
Add pthread_mutex_timedlock(), requested by dcoppa@

Fix previous commit: _rthread_init() was static.
Improve consistency of error naming

Fix the handling of the stackaddr, stacksize, and guardsize attributes:
don't try to merge values, round the sizes separately, and don't try to
unmap application-supplied stacks.
Copy from uthread the caching of default-sized stacks.
Have pthread_attr_init() and pthread_create() get the default attributes
from staticly allocated pthread_attr_t.
Cache the pagesize in _rthread_init() and provide a macro for rounding to it

based on suggestions from kettenis@ and tedu@, ok kettenis@

Add pthread_rwlockattr_{get,set}pshared() and
pthread_mutexattr_{get,set}kind_np()

tweaked diff from brad

Reimplement mutexes, condvars, and rwlocks to eliminate bugs,
particularly the "consume the signal you just sent" hang, and putting
the wait queues in userspace.

Do cancellation handling in pthread_cond_*wait(), pthread_join(),
and sem_wait().

Add __ prefix to thr{sleep,wakeup,exit,sigdivert}() syscalls; add
'abort" argument to thrsleep to close cancellation race; make
thr{sleep,wakeup} return errno values via *retval to avoid touching
userspace errno.

Split out the pthread_mutexattr_* functions from rthread_sync.c to
new file rthread_mutexattr.c. Add basic implementations of
pthread_mutexattr_{set,get}{protocol,prioceiling}

Requested by aja

Implement cancelation for the basic syscall cancelation points,
using previously allocated SIGTHR to interrupt in-process syscalls
and fixing the spelling of "cancelled" along the way.
Modeled on FreeBSD's libthr

Copy support for sched_get_priority_{min,max} from libpthread.
Requested by many to ease substitution of librthread for libpthread

Move <machine/spinlock.h> into rthread.h; strip out unnecessary #includes

Use __tfork, __get_tcb, and __set_tcb to have a real TCB and per-thread
errno. The ASM bits for _cerror are sketchy or missing for some archs
but that can be corrected in-tree.

Convert thrsleep() to an absolute timeout with clockid to eliminate a
race condition and prep for later support of pthread_condattr_setclock()

"get it in" deraadt@, tedu@, cheers by others

Convert thrsigdivert to (almost) be sigtimedwait by adding siginfo_t
and struct timespec * argument. sigtimedwait is just a one line
wrapper after this.

"get it in" deraadt@, tedu@, cheers by others

Change threxit() to take a pointer to a pid_t to zero out from the
kernel so that librthread can detect when a thread is completely
done with its stack without need a kqueue. The dying thread moves
itself to a GC list, other threads scan the GC list on pthread_create()
and pthread_join() and free the stack and handle once the thread's
thread id is zeroed.

"get it in" deraadt@, tedu@, cheers by others

Add f*lockfile() routines to librthread

ok kurt@

Increase default thread stack size to 256K on 32bit archs and 512K on
64bit archs to accommodate the growing number of ports that put large
buffers on the stack. Supported by many with no objections.

thrsigdivert() takes a sigset_t, not a pointer to one. Detected by
finally reading the lint warnings (sigh)

ok tedu@

Make sigwait() work correctly. In particular, it'll work when the
signal is already pending in the calling thread or the main thread
and there's no longer a race condition where the signal could be
diverted but sigwait() would still block. There were some off-by-one
errors too.

(The checking of the main thread's pending list is just until a
pending list for the process can be added to struct process. For
now, such signals end up as pending on the main thread.)

oks tedu@ and art@

- Add fork/vfork wrapper functions to reset state in the child process.
- Make an effort to protect important libc and ld.so critical areas during
the fork(2)/vfork(2) sys call.
- Add pthread_atfork(3) implementation based on Daniel Eischen's code.

Original diff by Philip Guenther <guenther at gmail.com> with some
additions and refinements by me. Positive test report from brad@ with
many kde apps. fork(2) and pthread_atfork(3) pthread regresses pass.
okay tedu@, kettenis@, marc@

add -Wstrict-prototypes -Wmissing-prototypes -Wsign-compare
Minor tweaks to compile with the above, primarily in fixing
the conflicts between semaphore.h and rthread.h
"i like the additional warnings" tedu@

Cleanup struct pthread and stack after thread exits. This version does
not use a separate reaper thread. ok tedu@ marc@

thread stack handling changes. Add guard zones and allow stack
size (and guard zone size) to be set using pthread_attr. Guard
zones are specified in bytes, but implemented in terms of a
page size.

OK Otto@

Add "print debug foo" function safe to use in signal handlers, example:
_rthread_debug(3, "foo = %s", foo);
where output is to stderr and will only be printed if the current
debug level is >= 3. Messages with a debug level of 0 are always
printed. The level can be set from the environment (not enabled yet)
or with gdb by modifying variable _rthread_debug_level.
"Fine with me" -Otto

Implement suspend/resume and creation of initially suspended threads.
With this, java seems to be operational. Also make threads_ready
non-static, which is needed for an upcoming diff. ok tedu@

use queue.h macros for thread house keeping; make some vars non-static
and fix a bug in thread creation error path. ok tedu@

prototype all the thread syscalls in rthread.h for now.
update for new thrwakeup that takes a count argument

more consistently use _rthread prefix for all not meant to be exported
interfaces that aren't static, and a few that are but which will change

Fix hppa ldcw alignment issue.

Help deraadt, tedu, kettenis
Ok tedu, kettenis

update copyright to 2005

add pthread_get/set_concurrency (useless for now)
add pthread_get/set_stack[addr] (info not used yet)

add pthread_cleanup_push and pthread_cleanup_pop

add bits for pthread_cancel. we don't really have cancellation points yet
but some of the functions are here now.

change keys to use table instead of list, makes a sane destructor implementation
possible

make the mutex unlock and sleep in pthread_cond_wait properly atomic

correct implementation of pthread_cond_signal. it doesn't raise the sem
value if there are no waiters.

add userland thread library. incomplete, but functional