Copy stable/10@r272459 to releng/10.1 as part of
the 10.1-RELEASE process.

Approved by: re (implicit)
Sponsored by: The FreeBSD Foundation

MFC r268384:
When the lock was stolen from us, retry the whole lock sequence in kernel,
instead of returning EINTR.

Copy head (r256279) to stable/10 as part of the 10.0-RELEASE cycle.

Approved by: re (implicit)
Sponsored by: The FreeBSD Foundation

Mark MALLOC_DEFINEs static that have no corresponding MALLOC_DECLAREs.

This means that their use is restricted to a single C file.

In lf_iteratelocks_vnode, increment state->ls_threads around iterating
of the vnode advisory lock list. This prevents deallocation of state
while inside the loop.

Reported and tested by: pho
MFC after: 2 weeks

Decrement state->ls_threads when vnode appeared to be doomed.

Reported and tested by: pho

Do not leak the state->ls_lock after VI_DOOMED check introduced
in the r192683.

Reported by: pho
Submitted by: jhb

The advisory lock may be activated or activated and removed during the
sleep waiting for conditions when the lock may be granted.
To prevent lf_setlock() from accessing possibly freed memory, add reference
counting to the struct lockf_entry. Bump refcount around the sleep.

Make lf_free_lock() return non-zero when structure was freed, and use
this after the sleep to return EINTR to the caller. The error code might
need a clarification, but we cannot return success to usermode, since
the lock is not owned anymore.

Reviewed by: dfr
Tested by: pho
MFC after: 1 month

In lf_purgelocks(), assert that state->ls_pending is empty after we
weeded out threads, and clean ls_active instead of ls_pending.

Reviewed by: dfr
Tested by: pho
MFC after: 1 month

In lf_advlockasync(), recheck for doomed vnode after the state->ls_lock
is acquired. In the lf_purgelocks(), assert that vnode is doomed and set
*statep to NULL before clearing ls_pending list. Otherwise, we allow for
the thread executing lf_advlockasync() to put new pending entry after
state->ls_lock is dropped in lf_purgelocks().

Reviewed by: dfr
Tested by: pho
MFC after: 1 month

Replace the while statement with the if for clarity. The loop body
cannot be executed more then once.

Reviewed by: dfr
Tested by: pho
MFC after: 1 month

Remove unused variable.

Found with: Coverity Prevent(tm)
CID: 3664

Approved by: kib

Don't rely on the value of *statep without first taking the vnode interlock.

Reviewed by: Mike Tancsa
MFC after: 2 weeks

Re-implement the client side of rpc.lockd in the kernel. This implementation
provides the correct semantics for flock(2) style locks which are used by the
lockf(1) command line tool and the pidfile(3) library. It also implements
recovery from server restarts and ensures that dirty cache blocks are written
to the server before obtaining locks (allowing multiple clients to use file
locking to safely share data).

Sponsored by: Isilon Systems
PR: 94256
MFC after: 2 weeks

When blocking on an F_FLOCK style lock request which is upgrading a
shared lock to exclusive, drop the shared lock before deadlock
detection.

MFC after: 2 days

Fix compilation with LOCKF_DEBUG.

Move the head of byte-level advisory lock list from the
filesystem-specific vnode data to the struct vnode. Provide the
default implementation for the vop_advlock and vop_advlockasync.
Purge the locks on the vnode reclaim by using the lf_purgelocks().
The default implementation is augmented for the nfs and smbfs.
In the nfs_advlock, push the Giant inside the nfs_dolock.

Before the change, the vop_advlock and vop_advlockasync have taken the
unlocked vnode and dereferenced the fs-private inode data, racing with
with the vnode reclamation due to forced unmount. Now, the vop_getattr
under the shared vnode lock is used to obtain the inode size, and
later, in the lf_advlockasync, after locking the vnode interlock, the
VI_DOOMED flag is checked to prevent an operation on the doomed vnode.

The implementation of the lf_purgelocks() is submitted by dfr.

Reported by: kris
Tested by: kris, pho
Discussed with: jeff, dfr
MFC after: 2 weeks

Don't try to use an SX lock while holding the vnode interlock.

Sponsored by: Isilon Systems

Add the new kernel-mode NFS Lock Manager. To use it instead of the
user-mode lock manager, build a kernel with the NFSLOCKD option and
add '-k' to 'rpc_lockd_flags' in rc.conf.

Highlights include:

* Thread-safe kernel RPC client - many threads can use the same RPC
client handle safely with replies being de-multiplexed at the socket
upcall (typically driven directly by the NIC interrupt) and handed
off to whichever thread matches the reply. For UDP sockets, many RPC
clients can share the same socket. This allows the use of a single
privileged UDP port number to talk to an arbitrary number of remote
hosts.

* Single-threaded kernel RPC server. Adding support for multi-threaded
server would be relatively straightforward and would follow
approximately the Solaris KPI. A single thread should be sufficient
for the NLM since it should rarely block in normal operation.

* Kernel mode NLM server supporting cancel requests and granted
callbacks. I've tested the NLM server reasonably extensively - it
passes both my own tests and the NFS Connectathon locking tests
running on Solaris, Mac OS X and Ubuntu Linux.

* Userland NLM client supported. While the NLM server doesn't have
support for the local NFS client's locking needs, it does have to
field async replies and granted callbacks from remote NLMs that the
local client has contacted. We relay these replies to the userland
rpc.lockd over a local domain RPC socket.

* Robust deadlock detection for the local lock manager. In particular
it will detect deadlocks caused by a lock request that covers more
than one blocking request. As required by the NLM protocol, all
deadlock detection happens synchronously - a user is guaranteed that
if a lock request isn't rejected immediately, the lock will
eventually be granted. The old system allowed for a 'deferred
deadlock' condition where a blocked lock request could wake up and
find that some other deadlock-causing lock owner had beaten them to
the lock.

* Since both local and remote locks are managed by the same kernel
locking code, local and remote processes can safely use file locks
for mutual exclusion. Local processes have no fairness advantage
compared to remote processes when contending to lock a region that
has just been unlocked - the local lock manager enforces a strict
first-come first-served model for both local and remote lockers.

Sponsored by: Isilon Systems
PR: 95247 107555 115524 116679
MFC after: 2 weeks

- Fix the last of the threading bugs that were introduced as far back as
1.38 in 2001. Break out of the FOREACH_THREAD_IN_PROC loop when we've
discovered a new proc in the chain.
- Increment i and check for maxlockdepth once per matching process not
once per thread. This didn't properly terminate the loop before.
- Fix a bug which has existed potentially since rev 1.1. waitblock->lf_next
can be NULL when a thread has been woken-up but not yet scheduled. Check
for this condition rather than blindly dereferencing.

Found by: libMicro

- Relax requirements for p_numthreads, p_threads, p_swtick, and p_nice from
requiring the per-process spinlock to only requiring the process lock.
- Reflect these changes in the proc.h documentation and consumers throughout
the kernel. This is a substantial reduction in locking cost for these
fields and was made possible by recent changes to threading support.

Do not call free() while holding vnode interlock.

Reported and tested by: Peter Holm
Reviewed by: jeff
Approved by: re (kensmith)

- Remove explicit Giant protection from lockf. Use the vnode interlock
to protect this datastructure instead.
- Preallocate an extra lockf structure in case we want to split a lock
on insert or delete.
- msleep() on the vnode interlock when blocking on a lock.

Reviewed by: rwatson
Approved by: re

Commit 14/14 of sched_lock decomposition.
- Use thread_lock() rather than sched_lock for per-thread scheduling
sychronization.
- Use the per-process spinlock rather than the sched_lock for per-process
scheduling synchronization.

Tested by: kris, current@
Tested on: i386, amd64, ULE, 4BSD, libthr, libkse, PREEMPTION, etc.
Discussed with: kris, attilio, kmacy, jhb, julian, bde (small parts each)

Print name of device instead of useless major/minor numbers.

Fix a debug message to print a usable device name rather than useless
major+minor tupple.

- Make lf_print static and move its prototype into kern_lockf.c
- Protect all of the advlock code with Giant as some filesystems
may not be entering with Giant held now.

Sponsored by: Isilon Systems, Inc.

/* -> /*- for copyright notices, minor format tweaks as necessary

Remove advertising clause from University of California Regent's license,
per letter dated July 22, 1999.

Approved by: core

Use __FBSDID().

Deprecate machine/limits.h in favor of new sys/limits.h.
Change all in-tree consumers to include <sys/limits.h>

Discussed on: standards@
Partially submitted by: Craig Rodrigues <rodrigc@attbi.com>

Including <sys/stdint.h> is (almost?) universally only to be able to use
%j in printfs, so put a newsted include in <sys/systm.h> where the printf
prototype lives and save everybody else the trouble.

Back out M_* changes, per decision of the TRB.

Approved by: trb

Remove M_TRYWAIT/M_WAITOK/M_WAIT. Callers should use 0.
Merge M_NOWAIT/M_DONTWAIT into a single flag M_NOWAIT.

- Fix a bunch of casts to long which were truncating off_t's.
- Remove the comments which were justifying this by the fact
that we don't have %q in the kernel, this was probably right
back in time, but we now have %q, and we even have better to
print those types (%j).

Remove a conditional #include <sys/kernel.h>, it is already
included unconditionally before.

Submitted by: Olivier Houchard <cognet@ci0.org>

Add a #include for <sys/mount.h>

More caddr_t removal.
Change struct knote's kn_hook from caddr_t to void *.

Remove __P.

make LOCKF_DEBUG kernel option work (sorta)

Submitted by: Maxim Konovalov <maxim@macomnet.ru>
PR: kern/32267

KSE Milestone 2
Note ALL MODULES MUST BE RECOMPILED
make the kernel aware that there are smaller units of scheduling than the
process. (but only allow one thread per process at this time).
This is functionally equivalent to teh previousl -current except
that there is a thread associated with each process.

Sorry john! (your next MFC will be a doosie!)

Reviewed by: peter@freebsd.org, dillon@freebsd.org

X-MFC after: ha ha ha ha

advlock: simplify overflow checks

Cosmetique & style fixes from bde

Remove extra check unneded now

Add yet one check for SEEK_END overflow

Oops, fix my broken handling of new l_len<0 case

Originally BSD return EINVAL for l_len < 0, but now POSIX wants it too,
so implement POSIX l_len < 0 handling.

Cosmetique: correct English in comments

Pointed by: bde

Move <machine/*> after <sys/*>

Pointed by: bde

Detect off_t EOVERFLOW of start/end offsets calculations for adv. lock,
as POSIX require.

Undo part of the tangle of having sys/lock.h and sys/mutex.h included in
other "system" header files.

Also help the deprecation of lockmgr.h by making it a sub-include of
sys/lock.h and removing sys/lockmgr.h form kernel .c files.

Sort sys/*.h includes where possible in affected files.

OK'ed by: bde (with reservations)

Implement client side NFS locks.

Obtained from: BSD/os
Import Ok'd by: mckusick, jkh, motd on builder.freebsd.org

Protect p_wmesg and p_wchan with sched_lock while checking for deadlocks
with other byte range file locks.

Commit the remaining part of PR14914:

Alot of the code in sys/kern directly accesses the *Q_HEAD and *Q_ENTRY
structures for list operations. This patch makes all list operations
in sys/kern use the queue(3) macros, rather than directly accessing the
*Q_{HEAD,ENTRY} structures.

Reviewed by: phk
Submitted by: Jake Burkholder <jake@checker.org>
PR: 14914

$Id$ -> $FreeBSD$

Fixed corruption of the "blocked" list in lf_setlock() when tsleep()
returns 0 after ptrace() attach and/or detach doesn't quite quite
deliver a signal. Perhaps the process shouldn't be woken in this
case, but avoiding the problem is easy.

PR: 12247

Fixed a couple of places where mechanical fixing of compiler warnings
caused misspelling of NOLOCKF as NULL.

Fix a freelist trashing under following confitions:
- first program lock a region in a file,
- second program wait on the lock,
- first program extend the region,
- second program interrupted by a signal.

Fix warnings in preparation for adding -Wall -Wcast-qual to the
kernel compile

add #include <sys/kernel.h> where it's needed by MALLOC_DEFINE()

Fixed printf format errors.

Make the debug options new-style.

This also zaps a DPT option from lint; it wasn't referenced from
anywhere.

Don't include <sys/lock.h> in headers when only `struct simplelock' is
required. Fixed everything that depended on the pollution.

Last major round (Unless Bruce thinks of somthing :-) of malloc changes.

Distribute all but the most fundamental malloc types. This time I also
remembered the trick to making things static: Put "static" in front of
them.

A couple of finer points by: bde

Distribute and statizice a lot of the malloc M_* types.

Substantial input from: bde

Use OID_AUTO instead of a magic number for Lite2 sysctl debug.lockf_debug.

Removed unused #includes.

Back out part 1 of the MCFH that changed $Id$ to $FreeBSD$. We are not
ready for it yet.

Added some ufs #includes so that this compiles with option LOCKF_DEBUG.
Moving this all from ufs wasn't a good move. At least the debugging
routines depend on the file system.

Cleaned up the LOCKF_DEBUG #includes.

Restored some lost function return types.

This is the kernel Lite/2 commit. There are some requisite userland
changes, so don't expect to be able to run the kernel as-is (very well)
without the appropriate Lite/2 userland changes.

The system boots and can mount UFS filesystems.

Untested: ext2fs, msdosfs, NFS
Known problems: Incorrect Berkeley ID strings in some files.
Mount_std mounts will not work until the getfsent
library routine is changed.

Reviewed by: various people
Submitted by: Jeffery Hsu <hsu@freebsd.org>

Make the long-awaited change from $Id$ to $FreeBSD$

This will make a number of things easier in the future, as well as (finally!)
avoiding the Id-smashing problem which has plagued developers for so long.

Boy, I'm glad we're not using sup anymore. This update would have been
insane otherwise.

Delete bogus inconsistency check that could cause a gratuitous panic. I
had added this years ago when I didn't understand all the subtilties of
the flock code.

Fixed arg checking in if_advlock(). Invalid args were accepted in an
optimized case. Preposterous lengths weren't checked for.

Found by: NIST-PCTS

Eliminated nested include of <sys/unistd.h> in <sys/file.h> in the kernel.
Include it directly in the few places where it is used.

Reduced some #includes of <sys/file.h> to #includes of <sys/fcntl.h> or
nothing.

A Major staticize sweep. Generates a couple of warnings that I'll deal
with later.
A number of unused vars removed.
A number of unused procs removed or #ifdefed.

Remove trailing whitespace.

Changed some variable names in lf_addblock to make the code both
understandable and conform to other conventions used in the file.

While in the real world, I had a bad case of being swapped out for a lot of
cycles. While waiting there I added a lot of the extra ()'s I have, (I have
never used LISP to any extent). So I compiled the kernel with -Wall and
shut up a lot of "suggest you add ()'s", removed a bunch of unused var's
and added a couple of declarations here and there. Having a lap-top is
highly recommended. My kernel still runs, yell at me if you kernel breaks.

Made lockf advisory locking code generic (rather than ufs specific), and
use it in NFS. This is required both for diskless support and for POSIX
compliance. Note: the support in NFS is only for the local node.

Submitted by: based on work originally done by Yuval Yurom