MFC: r275751

Add _NEW flag to mtx(9), sx(9), rmlock(9) and rwlock(9).
A _NEW flag passed to _init_flags() to avoid check for double-init.

MFC r285706,r303562,r303563,r303584,r303643,r303652,r303655,r303707:

(by markj)
Don't increment the spin count until after the first attempt to acquire a
rwlock read lock. Otherwise the lockstat:::rw-spin probe will fire
spuriously.

==

rwlock: s/READER/WRITER/ in wlock lockstat annotation

==

sx: increment spin_cnt before cpu_spinwait in xlock

The change is a no-op only done for consistency with the rest of the file.

==

locks: change sleep_cnt and spin_cnt types to u_int

Both variables are uint64_t, but they only count spins or sleeps.
All reasonable values which we can get here comfortably hit in 32-bit range.

==

Implement trivial backoff for locking primitives.

All current spinning loops retry an atomic op the first chance they get,
which leads to performance degradation under load.

One classic solution to the problem consists of delaying the test to an
extent. This implementation has a trivial linear increment and a random
factor for each attempt.

For simplicity, this first thouch implementation only modifies spinning
loops where the lock owner is running. spin mutexes and thread lock were
not modified.

Current parameters are autotuned on boot based on mp_cpus.

Autotune factors are very conservative and are subject to change later.

==

locks: fix up ifdef guards introduced in r303643

Both sx and rwlocks had copy-pasted ADAPTIVE_MUTEXES instead of the correct
define.

==

locks: fix compilation for KDTRACE_HOOKS && !ADAPTIVE_* case

==

locks: fix sx compilation on mips after r303643

The kernel.h header is required for the SYSINIT macro, which apparently
was present on amd64 by accident.

MFC r301157:

Microoptimize locking primitives by avoiding unnecessary atomic ops.

Inline version of primitives do an atomic op and if it fails they fallback to
actual primitives, which immediately retry the atomic op.

The obvious optimisation is to check if the lock is free and only then proceed
to do an atomic op.

MFC r306346

Make no assertions about mutex state when the scheduler is stopped.

This changes the assert path to match the lock and unlock paths.

Sponsored by: Dell EMC

MFC r303211:
Implement mtx_trylock_spin(9).

MFC r302346:
Ensure that spinlock sections are balanced even after a panic.

MFC r285663, r285664, r285667:
Ensure that locstat_nsecs() has no effect when lockstat probes are not
enabled or when the profiled lock carries the LO_NOPROFILE flag.

PR: 201642, 201517
Approved by: re (gjb)
Tested by: Jason Unovitch

MFC r284297: several lockstat improvements

MFC 272315 272757 274091 274902

for real this time

r272315
Explicitly return None for negative event indices. Prior to this,
eventat(-1) would return the next-to-last event causing the back button
to cycle back to the end of an event source instead of stopping at the
start.

r272757
Add schedgraph traces for callout handlers. Specifically, a callwheel logs
a running event each time it executes a callout function. The event
includes the function pointer, argument, and whether or not it was run from
hardware interrupt context. The callwheel is marked idle when each handler
completes. This effectively logs the duration of each callout routine in
the graph.

r274091
Bind Ctrl-Q as a global hotkey to exit. Bind Ctrl-W as a hotkey to close
dialogs.

r274902
Add a new thread state "spinning" to schedgraph and add tracepoints at the
start and stop of spinning waits in lock primitives.

Reviewed by: jhb

Revert r278650. Definite layer 8 bug.

Submitted by: dhw and Thomas Mueller <tmueller@sysgo.com>

MFC 272315 272757 274091 274902

r272315
Explicitly return None for negative event indices. Prior to this,
eventat(-1) would return the next-to-last event causing the back button
to cycle back to the end of an event source instead of stopping at the
start.

r272757
Add schedgraph traces for callout handlers. Specifically, a callwheel logs
a running event each time it executes a callout function. The event
includes the function pointer, argument, and whether or not it was run from
hardware interrupt context. The callwheel is marked idle when each handler
completes. This effectively logs the duration of each callout routine in
the graph.

r274091
Bind Ctrl-Q as a global hotkey to exit. Bind Ctrl-W as a hotkey to close
dialogs.

r274902
Add a new thread state "spinning" to schedgraph and add tracepoints at the
start and stop of spinning waits in lock primitives.

Reviewed by: jhb

MFC r285663, r285664, r285667:
Ensure that locstat_nsecs() has no effect when lockstat probes are not
enabled or when the profiled lock carries the LO_NOPROFILE flag.

PR: 201642, 201517
Approved by: re (gjb)
Tested by: Jason Unovitch

MFC r284297: several lockstat improvements

MFC 272315 272757 274091 274902

for real this time

r272315
Explicitly return None for negative event indices. Prior to this,
eventat(-1) would return the next-to-last event causing the back button
to cycle back to the end of an event source instead of stopping at the
start.

r272757
Add schedgraph traces for callout handlers. Specifically, a callwheel logs
a running event each time it executes a callout function. The event
includes the function pointer, argument, and whether or not it was run from
hardware interrupt context. The callwheel is marked idle when each handler
completes. This effectively logs the duration of each callout routine in
the graph.

r274091
Bind Ctrl-Q as a global hotkey to exit. Bind Ctrl-W as a hotkey to close
dialogs.

r274902
Add a new thread state "spinning" to schedgraph and add tracepoints at the
start and stop of spinning waits in lock primitives.

Reviewed by: jhb

Revert r278650. Definite layer 8 bug.

Submitted by: dhw and Thomas Mueller <tmueller@sysgo.com>

MFC 272315 272757 274091 274902

r272315
Explicitly return None for negative event indices. Prior to this,
eventat(-1) would return the next-to-last event causing the back button
to cycle back to the end of an event source instead of stopping at the
start.

r272757
Add schedgraph traces for callout handlers. Specifically, a callwheel logs
a running event each time it executes a callout function. The event
includes the function pointer, argument, and whether or not it was run from
hardware interrupt context. The callwheel is marked idle when each handler
completes. This effectively logs the duration of each callout routine in
the graph.

r274091
Bind Ctrl-Q as a global hotkey to exit. Bind Ctrl-W as a hotkey to close
dialogs.

r274902
Add a new thread state "spinning" to schedgraph and add tracepoints at the
start and stop of spinning waits in lock primitives.

Reviewed by: jhb