Kill unused variable in uvm_aio_aiodone_pages().

Put back in the simplification of the aiodone daemon.

Previous "breakage" of the swap on arm64 has been found to be an issue
on one machine the rockpro/arm64 related to a deadlock built into the
sdmmc(4) stack interacting with swapping code both running under
KERNEL_LOCK().

This issue is easily reproducible on -current and entering swap when
building LLVM on a rockpro crashes the machine by memory corruption.

Tested by mlarkin@ on octeon & i386, by myself on amd64 & arm64 and by
sthen@ on i386 port bulk.

ok beck@ some time ago.

Previous commit message:

Simplify the aiodone daemon which is only used for async writes.

- Remove unused support for asynchronous read, including error conditions

- Grab the proper lock for each page that has been written to swap. This
allows to enable an assertion in uvm_page_unbusy().

- Move the uvm_anon_release() call outside of uvm_page_unbusy() and
assert for the different anon cases.

ok beck@, kettenis@

Grab the mutex before iterating on the global list.

Prevent race reported some months ago.

ok mlarkin@, jsg@, kn@

remove unused uvm_aiobuf_pool

Use PMAP_PREFER_ALIGN() == 0 rather than !defined(PMAP_PREFER) to enable the
fast path in the pager code; this benefits most mips64 platforms.

ok kettenis@ mpi@

Bring back the direct map optimization when PMAP_PREFER isn't defined.

PMAP_PREFER is checked to prevent cache aliasing issues when the physical
page is mapped at different virtual addresses.

This speeds up file-based mmap faults.

ok kettenis@

Revert simplification of the aiodone daemon it breaks swap on arm64.

Found the hard way by mlarkin@ and deraadt@.

Revert the changes made in rev 1.82. It is important to use pmap_enter(9)
and pmap_remove(9) here since we're dealing with managed pages here. Found
out the hard way by deraadt@ on landisk where we're running into issues
with virtual cache aliases because multiple mappings exist for the
pages we're dealing with here. The pmap_enter(9) and pmap_remove(9)
functions handle conflicting cache aliases, whereas pmap_map_direct(9) and
pmap_kenter_pa(9) assume that the pages is exclusively mapped in the kernel
pmap.

ok deraadt@

Simplify the aiodone daemon which is only used for async writes.

- Remove unused support for asynchronous read, including error conditions

- Grab the proper lock for each page that has been written to swap. This
allows to enable an assertion in uvm_page_unbusy().

- Move the uvm_anon_release() call outside of uvm_page_unbusy() and
assert for the different anon cases.

ok beck@, kettenis@

Reduce allocations and possible failures in uvm_pagermapin/out().

. If a direct map exists use it to map single-page allocations
. Use pmap_kenter_pa() instead of pmap_enter() in all other cases.

This speeds up file-based mmap up to 75% when I/O are performed and it
also reduces possible allocations failtures in the page daemon making
it more stable in OOM situations.

ok kettenis@, beck@

Reserve a second segment for the page daemon.

This ensures uvm_swap_io() can succeeds even in OOM situations because two
uvm_pagermapin() allocations, requiring a segment each, are needed to bounce
or encrypt a swap cluster.

ok beck@, kettenis@

Move the guts of uvm_aio_aiodone() into a separate function.

This will help dealing with error cases.

No functionnal changes.

Do not clean PG_BUSY before calling uvm_anon_release().

Fix an assertion reported by gkoehler@.

ok kettenis@

Convert KVA allocation to km_alloc(9).

ok mpi@

Use a per-UVM object lock to serialize the lower part of the fault handler.

Like the per-amap lock the `vmobjlock' is principally used to serialized
access to objects in the fault handler to allow faults occurring on
different CPUs and different objects to be processed in parallel.

The fault handler now acquires the `vmobjlock' of a given UVM object as
soon as it finds one. For now a write-lock is always acquired even if
some operations could use a read-lock.

Every pager, corresponding to a different kind of UVM object, now expect
the UVM object to be locked and some operations, like *_get() return it
unlocked. This is enforced by assertions checking for rw_write_held().

The KERNEL_LOCK() is now pushed to the VFS boundary in the vnode pager.

To ensure the correct amap or object lock is held when modifying a page
many uvm_page* operations are now asserting for the "owner" lock.
However, fields of the "struct vm_page" are still being protected by the
global `pageqlock'. To prevent lock ordering issues with the new
`vmobjlock' and to reduce differences with NetBSD this lock is now taken
and released for each page instead of around the whole loop.

This commit does not remove the KERNEL_LOCK/UNLOCK() dance. Unlocking
will follow if there is no fallout.

Ported from NetBSD, tested by many, thanks!

ok kettenis@, kn@

Remove parenthesis around return value to reduce the diff with NetBSD.

No functional change.

ok mlarkin@

spelling

ok mpi@

(re)Introduce locking for amaps & anons.

A rwlock is attached to every amap and is shared with all its anon. The
same lock will be used by multiple amaps if they have anons in common.

This should be enough to get the upper part of the fault handler out of the
KERNEL_LOCK() which seems to bring up to 20% improvements in builds.

This is based/copied/adapted from the most recent work done in NetBSD which
is an evolution of the precendent simple_lock scheme.

Tested by many, thanks!

ok kettenis@, mvs@

Constify and use C99 initializer for "struct uvm_pagerops".

While here put some KERNEL_ASSERT_LOCKED() in the functions called from
the page fault handler. The removal of locking of `uobj' will need to be
revisited and these are good indicator that something is missing and that
many comments are lying.

ok kettenis

Convert infinite sleeps to {m,t}sleep_nsec(9).

ok visa@, jca@

remove lock.h from uvm_extern.h. another holdover from the simpletonlock
era. fix uvm including c files to include lock.h or atomic.h as necessary.
ok deraadt

Replace a plethora of historical protection options with just
PROT_NONE, PROT_READ, PROT_WRITE, and PROT_EXEC from mman.h.
PROT_MASK is introduced as the one true way of extracting those bits.
Remove UVM_ADV_* wrapper, using the standard names.
ok doug guenther kettenis

remove uneeded proc.h includes
ok mpi@ kspillner@

Chuck Cranor rescinded clauses in his license
on the 2nd of February 2011 in NetBSD.

http://marc.info/?l=netbsd-source-changes&m=129658899212732&w=2
http://marc.info/?l=netbsd-source-changes&m=129659095515558&w=2
http://marc.info/?l=netbsd-source-changes&m=129659157916514&w=2
http://marc.info/?l=netbsd-source-changes&m=129665962324372&w=2
http://marc.info/?l=netbsd-source-changes&m=129666033625342&w=2
http://marc.info/?l=netbsd-source-changes&m=129666052825545&w=2
http://marc.info/?l=netbsd-source-changes&m=129666922906480&w=2
http://marc.info/?l=netbsd-source-changes&m=129667725518082&w=2

stop using B_AGE, it was effectively retired some time ago.

Fix some potential integer overflows caused by converting a page number into
an offset/size/address by shifting by PAGE_SHIFT. Make uvm_objwrire/unwire
use voff_t instead of off_t. The former is the right type here even if it is
equivalent to the latter.

Inspired by a somewhat similar changes in Bitrig.

ok deraadt@, guenther@

compress code by turning four line comments into one line comments.
emphatic ok usual suspects, grudging ok miod

No need to cast constants or simple variables to (daddr_t). Use
(u_int64_t) instead of (daddr_t) when casting a variable in an
expression passed to DL_SETDSIZE().

Change a variable counting open files from daddr_t to int64_t.

ok deraadt@ with the tweak to fix that pesky expression.

final removal of daddr64_t. daddr_t has been 64 bit for a long enough
test period; i think 3 years ago the last bugs fell out.
ok otto beck others

remove lots of comments about locking per beck's request

remove simple_locks from uvm code. ok beck deraadt

Rip out and burn support for UVM_HIST.

The vm hackers don't use it, don't maintain it and have to look at it all the
time. About time this 800 lines of code hit /dev/null.

``never liked it'' tedu@. ariane@ was very happy when i told her i wrote
this diff.

for uvm_pager_dropcluster in the PG_RELEASED case we specifically unbusy the
page so that um_anfree will free it for us.

uvm_anfree does a pmap_page_protect(, VM_PROT_NONE) just before it frees the
page, so we don't need to do it here ourselves.

ok ariane@

move uvm_pageratop from uvm_pager.c local to a general uvm function
(uvm_atopg) and use it in uvm_km_doputpage to replace some handrolled
code. Shrinks the kernel a trivial amount.

ok beck@ and miod@ (who suggested i name it uvm_atopg not uvm_atop)

Don't sleep while holding the uvm_psel_lck mutex. Should fix "locking against
myself" panics that some people have seen over the last year-and-a-half.

Cherry picked from a more complex (and therefore scarier) diff from oga@.

ok tedu@, oga@

Kill another #ifdef UBC chunk that was annoying me while doing something
else.

ok thib@

introduce a uvm_km_valloc_try function that won't get a lower level lock
for use by the uvm pseg code. this is the path of least resistance until
we sort out how many of these functions we really need. problem found by mikeb
ok kettenis oga

Put the PG_RELEASED changes diff back in.

This has has been tested very very thoroughly on all archs we have
excepting 88k and 68k. Please see cvs log for the individual commit
messages.

ok beck@, thib@

date based reversion of uvm to the 4th May.

More backouts in line with previous ones, this appears to bring us back to a
stable condition.

A machine forced to 64mb of ram cycled 10GB through swap with this diff
and is still running as I type this. Other tests by ariane@ and thib@
also seem to show that it's alright.

ok deraadt@, thib@, ariane@

Backout all the PG_RELEASED changes.

This is for the same reason as the earlier backouts, to avoid the bug
either added or exposed sometime around c2k9. This *should* be the last
one.

prompted by deraadt@

ok ariane@

More PG_RELEASED cleaning.

similar to the aobj.c changes, this one does vnodes. Vnodes are more
complex because they actaully have to sync to backing store. So firstly,
convert sync to sleep instead of setting released.

Now, for backing store, in the PGO_FREE case, if we set PG_RELEASED
*before* an async io, (nothing else will see it, the page is busy), then
we can ignore the page after the io is done. We could do something
similar for PGO_DEACTIVATE too, but that is another change. On error we
just clear the released flag, nothing else sets it for uobj pages other
than aiodoned.

ok thib@, beck@, ariane@

Instead of keeping two ints in the uvm structure specifically just to
sleep on them (and otherwise ignore them) sleep on the pointer to the
{aiodoned,pagedaemon}_proc members, and nuke the two extra words.

"no objections" art@, ok beck@.

Instead of doing splbio(); simple_lock(&uvm.aiodoned_lock); just replace
the simple lock with a real lock - a IPL_BIO mutex. While i'm here, make
the sleeping condition one hell of a lot simpler in the aio daemon.

some ideas from and ok art@.

Move all of the pseudo-inline functions in uvm into C files.

By pseudo-inline, I mean that if a certain macro was defined, they would
be inlined. However, no architecture defines that, and none has for a
very very long time. Therefore mainly this just makes the code a damned
sight easier to read. Some k&r -> ansi declarations while I'm in there.

"just commit it" art@. ok weingart@.

While working on some stuff in uvm I've gotten REALLY sick of reading
K&R function declarations, so switch them all over to ansi-style, in
accordance with the prophesy.

"go for it" art@

Simplify page-out/page-in map management; fix rare pager deadlock.

Ok: miod, tedu

garbage collect uvm_errno2vmerror();

ok miod@, art@

In uvm_aio_aiodone(), kill the variable-sized array on stack, and use a
fixed size array which size should match any buf; if a bogus buf is passed
to this function, the kernel will KASSERT instead of potentially running out
of stack and having an undefined behaviour.

ok deraadt@

now that all partition size/offsets are potentially 64-bit, change the
type of all variables to daddr64_t. this includes the APIs for XXsize()
and XXdump(), all range checks inside bio drivers, internal variables
for disklabel handling, and even uvm's swap offsets. re-read numerous
times by otto, miod, krw, thib to look for errors

While splitting flags and pqflags might have been a good idea in theory
to separate locking, on most modern machines this is not enough
since operations on short types touch other short types that share the
same word in memory.

Merge pg_flags and pqflags again and now use atomic operations to change
the flags. Also bump wire_count to an int and pg_version might go
int as well, just for alignment.

tested by many, many. ok miod@

Mechanically rename the "flags" and "version" fields in struct vm_page
to "pg_flags" and "pg_version", so that they are a bit easier to work with.
Whoever uses generic names like this for a popular struct obviously doesn't
read much code.

Most architectures compile and there are no functionality changes.

deraadt@ ok ("if something fails to compile, we fix that by hand")

remove KERN_SUCCESS and use 0 instead.
eyeballed by miod@ and pedro@

fix uvmhist #2: args are always u_long so fix missing %d and %x and no %ll; no change for normal code

fix fmts for UVMHIST_LOG() entries making it more useful on 64bit archs; miod@ ok

- Make a UVM_OBJ_IS_DEVICE macro.
- Use it to skip device mappings while dumping core.
- Ignore EFAULT errors while dumping core since they can happen
even for valid mappings. Just skip that part of the core file and
let it get automagically zero-filled.

This fixes the broken X core dumps that people have been seeing and also
fixes some other potential problems that could prevent core dumps (mmaps
beyond EOF, etc.).

tedu@ ok

add a new field to vm_space and use it to track the number of anon
pages a process uses. this is now the userland "data size" value.
ok art deraadt tdeval. thanks testers.

sync of pmap_update() calls with NetBSD. pmap_update is defined away on
all architectures but arm, where it is needed.

ubchist is not a fully cooked kadaver and though use the other well formed pdhist one until ubc gaets back. art@ ok

Since memory deallocation can't fail, remove the error return from
uvm_unmap, uvm_deallocate and a few other functions.
Simplifies some code and reduces diff to the UBC branch.

Make sure that b_iodone handlers are called at splbio (and splassert(IPL_BIO) in all known callers, just to make sure).

splassert(IPL_BIO) in the b_iodone handlers.

Back out a few more uvm changes, especially wrt swap usage.
This unbreaks m68k m88k sparc and perhaps others, which eventually froze
when hitting swap.
Tested by various people on various platforms.
ok art@

UBC was a disaster. It worked very good when it worked, but on some
machines or some configurations or in some phase of the moon (we actually
don't know when or why) files disappeared. Since we've not been able to
track down the problem in two weeks intense debugging and we need -current
to be stable, back out everything to a state it had before UBC.

We apologise for the inconvenience.

branches: 1.28.2;
Yet another sync to NetBSD uvm.
Today we add a pmap argument to pmap_update() and allocate map entries for
kernel_map from kmem_map instead of using the static entries. This should
get rid of MAX_KMAPENT panics. Also some uvm_loan problems are fixed.

Call buf_cleanout, which handles wakeups

Correctly handle b_vp with bgetvp and brelvp in {get,put}pages.
Prevents panics caused by vnodes being recycled under our feet.

Sync in more uvm from NetBSD. Mostly just cosmetic stuff.
Contains also support for page coloring.

Sync in more uvm changes from NetBSD.
This time we're getting rid of KERN_* and VM_PAGER_* error codes and
use errnos instead.

Merge in the unified buffer cache code as found in NetBSD 2001/03/10. The
code is written mostly by Chuck Silvers <chuq@chuq.com>/<chs@netbsd.org>.

Tested for the past few weeks by many developers, should be in a pretty stable
state, but will require optimizations and additional cleanups.

Bring in more changes from NetBSD. Mostly pagedaemon improvements.

Merge in some parts of the ubc work that has been done in NetBSD that are not
UBC, but prerequsites for it.

- Create a daemon that processes async I/O (swap and paging in the future)
requests that need processing in process context and that were processed
in the pagedaemon before.
- Convert some ugly ifdef DIAGNOSTIC code to less intrusive KASSERTs.
- misc other cleanups.

Another sync of uvm to NetBSD. Just minor fiddling, no major changes.

Add an alignment argument to uvm_map that specifies an alignment hint
for the virtual address.

Move the last content from vm/ to uvm/
The only thing left in vm/ are just dumb wrappers.
vm/vm.h includes uvm/uvm_extern.h
vm/pmap.h includes uvm/uvm_pmap.h
vm/vm_page.h includes uvm/uvm_page.h

Minor sync to NetBSD.

occured->occurred

idea from deraadt@ via NetBSD
millert@ ok

merge vm/vm_kern.h into uvm/uvm_extern.h; art@ ok

Various random fixes from NetBSD.
Including support for zeroing pages in the idle loop (not enabled yet).

Add a new type voff_t (right now it's typedefed as off_t) used for offsets
into objects.

Gives the possibilty to mmap beyond the size of vaddr_t.

From NetBSD.

Add support for disabling swap devices (swapctl -d).
Improve error handling on I/O errors to swap.
From NetBSD

Change the pmap_enter interface to merge access_type and the wired boolean
and arbitrary flags into one argument.

One new flag is PMAP_CANFAIL that tells pmap_enter that it can fail if there
are not enough resources to satisfy the request. If this flag is not passed,
pmap_enter should panic as it should have done before this change (XXX - many
pmaps are still not doing that).

Only i386 and alpha implement CANFAIL for now.

Includes uvm updates from NetBSD.

Get rid of the PMAP_NEW option by making it mandatory for all archs.
The archs that didn't have a proper PMAP_NEW now have a dummy implementation
with wrappers around the old functions.

Change the paddr_t pmap_extract(struct pmap *, vaddr_t) interface to
boolean_t pmap_extract(struct pmap *, vaddr_t, paddr_t *).
Matches NetBSD. Tested by various people on various platforms.

Sync style, typo, and comments a little closer to NetBSD. art@ ok

Replace thread_wakeup() with wakeup(). It is defined in vm_extern.h as a
wrapper, so this removes a dependence on the old VM system. From NetBSD.
art@ ok

$OpenBSD$

Bring in some new UVM code from NetBSD (not current).

- Introduce a new type of map that are interrupt safe and never allow faults
in them. mb_map and kmem_map are made intrsafe.
- Add "access protection" to uvm_vslock (to be passed down to uvm_fault and
later to pmap_enter).
- madvise(2) now works.
- various cleanups.

branches: 1.4.4;
Change the pmap_enter api to pass down an argument that indicates
the access type that caused this mapping. This is to simplify pmaps
with mod/ref emulation (none for the moment) and in some cases speed
up pmap_is_{referenced,modified}.
At the same time, clean up some mappings that had too high protection.

XXX - the access type is incorrect in old vm, it's only used by uvm and MD code.
The actual use of this in pmap_enter implementations is not in this commit.

sync with NetBSD from 1999.05.24 (there is a reason for this date)
Mostly cleanups, but also a few improvements to pagedaemon for better
handling of low memory and/or low swap conditions.

add OpenBSD tags

Import of uvm from NetBSD. Some local changes, some code disabled

Put back in the simplification of the aiodone daemon.

Previous "breakage" of the swap on arm64 has been found to be an issue
on one machine the rockpro/arm64 related to a deadlock built into the
sdmmc(4) stack interacting with swapping code both running under
KERNEL_LOCK().

This issue is easily reproducible on -current and entering swap when
building LLVM on a rockpro crashes the machine by memory corruption.

Tested by mlarkin@ on octeon & i386, by myself on amd64 & arm64 and by
sthen@ on i386 port bulk.

ok beck@ some time ago.

Previous commit message:

Simplify the aiodone daemon which is only used for async writes.

- Remove unused support for asynchronous read, including error conditions

- Grab the proper lock for each page that has been written to swap. This
allows to enable an assertion in uvm_page_unbusy().

- Move the uvm_anon_release() call outside of uvm_page_unbusy() and
assert for the different anon cases.

ok beck@, kettenis@

Grab the mutex before iterating on the global list.

Prevent race reported some months ago.

ok mlarkin@, jsg@, kn@

remove unused uvm_aiobuf_pool

Use PMAP_PREFER_ALIGN() == 0 rather than !defined(PMAP_PREFER) to enable the
fast path in the pager code; this benefits most mips64 platforms.

ok kettenis@ mpi@

Bring back the direct map optimization when PMAP_PREFER isn't defined.

PMAP_PREFER is checked to prevent cache aliasing issues when the physical
page is mapped at different virtual addresses.

This speeds up file-based mmap faults.

ok kettenis@

Revert simplification of the aiodone daemon it breaks swap on arm64.

Found the hard way by mlarkin@ and deraadt@.

Revert the changes made in rev 1.82. It is important to use pmap_enter(9)
and pmap_remove(9) here since we're dealing with managed pages here. Found
out the hard way by deraadt@ on landisk where we're running into issues
with virtual cache aliases because multiple mappings exist for the
pages we're dealing with here. The pmap_enter(9) and pmap_remove(9)
functions handle conflicting cache aliases, whereas pmap_map_direct(9) and
pmap_kenter_pa(9) assume that the pages is exclusively mapped in the kernel
pmap.

ok deraadt@

Simplify the aiodone daemon which is only used for async writes.

- Remove unused support for asynchronous read, including error conditions

- Grab the proper lock for each page that has been written to swap. This
allows to enable an assertion in uvm_page_unbusy().

- Move the uvm_anon_release() call outside of uvm_page_unbusy() and
assert for the different anon cases.

ok beck@, kettenis@

Reduce allocations and possible failures in uvm_pagermapin/out().

. If a direct map exists use it to map single-page allocations
. Use pmap_kenter_pa() instead of pmap_enter() in all other cases.

This speeds up file-based mmap up to 75% when I/O are performed and it
also reduces possible allocations failtures in the page daemon making
it more stable in OOM situations.

ok kettenis@, beck@

Reserve a second segment for the page daemon.

This ensures uvm_swap_io() can succeeds even in OOM situations because two
uvm_pagermapin() allocations, requiring a segment each, are needed to bounce
or encrypt a swap cluster.

ok beck@, kettenis@

Move the guts of uvm_aio_aiodone() into a separate function.

This will help dealing with error cases.

No functionnal changes.

Do not clean PG_BUSY before calling uvm_anon_release().

Fix an assertion reported by gkoehler@.

ok kettenis@

Convert KVA allocation to km_alloc(9).

ok mpi@

Use a per-UVM object lock to serialize the lower part of the fault handler.

Like the per-amap lock the `vmobjlock' is principally used to serialized
access to objects in the fault handler to allow faults occurring on
different CPUs and different objects to be processed in parallel.

The fault handler now acquires the `vmobjlock' of a given UVM object as
soon as it finds one. For now a write-lock is always acquired even if
some operations could use a read-lock.

Every pager, corresponding to a different kind of UVM object, now expect
the UVM object to be locked and some operations, like *_get() return it
unlocked. This is enforced by assertions checking for rw_write_held().

The KERNEL_LOCK() is now pushed to the VFS boundary in the vnode pager.

To ensure the correct amap or object lock is held when modifying a page
many uvm_page* operations are now asserting for the "owner" lock.
However, fields of the "struct vm_page" are still being protected by the
global `pageqlock'. To prevent lock ordering issues with the new
`vmobjlock' and to reduce differences with NetBSD this lock is now taken
and released for each page instead of around the whole loop.

This commit does not remove the KERNEL_LOCK/UNLOCK() dance. Unlocking
will follow if there is no fallout.

Ported from NetBSD, tested by many, thanks!

ok kettenis@, kn@

Remove parenthesis around return value to reduce the diff with NetBSD.

No functional change.

ok mlarkin@

spelling

ok mpi@

(re)Introduce locking for amaps & anons.

A rwlock is attached to every amap and is shared with all its anon. The
same lock will be used by multiple amaps if they have anons in common.

This should be enough to get the upper part of the fault handler out of the
KERNEL_LOCK() which seems to bring up to 20% improvements in builds.

This is based/copied/adapted from the most recent work done in NetBSD which
is an evolution of the precendent simple_lock scheme.

Tested by many, thanks!

ok kettenis@, mvs@

Constify and use C99 initializer for "struct uvm_pagerops".

While here put some KERNEL_ASSERT_LOCKED() in the functions called from
the page fault handler. The removal of locking of `uobj' will need to be
revisited and these are good indicator that something is missing and that
many comments are lying.

ok kettenis

Convert infinite sleeps to {m,t}sleep_nsec(9).

ok visa@, jca@

remove lock.h from uvm_extern.h. another holdover from the simpletonlock
era. fix uvm including c files to include lock.h or atomic.h as necessary.
ok deraadt

Replace a plethora of historical protection options with just
PROT_NONE, PROT_READ, PROT_WRITE, and PROT_EXEC from mman.h.
PROT_MASK is introduced as the one true way of extracting those bits.
Remove UVM_ADV_* wrapper, using the standard names.
ok doug guenther kettenis

remove uneeded proc.h includes
ok mpi@ kspillner@

Chuck Cranor rescinded clauses in his license
on the 2nd of February 2011 in NetBSD.

http://marc.info/?l=netbsd-source-changes&m=129658899212732&w=2
http://marc.info/?l=netbsd-source-changes&m=129659095515558&w=2
http://marc.info/?l=netbsd-source-changes&m=129659157916514&w=2
http://marc.info/?l=netbsd-source-changes&m=129665962324372&w=2
http://marc.info/?l=netbsd-source-changes&m=129666033625342&w=2
http://marc.info/?l=netbsd-source-changes&m=129666052825545&w=2
http://marc.info/?l=netbsd-source-changes&m=129666922906480&w=2
http://marc.info/?l=netbsd-source-changes&m=129667725518082&w=2

stop using B_AGE, it was effectively retired some time ago.

Fix some potential integer overflows caused by converting a page number into
an offset/size/address by shifting by PAGE_SHIFT. Make uvm_objwrire/unwire
use voff_t instead of off_t. The former is the right type here even if it is
equivalent to the latter.

Inspired by a somewhat similar changes in Bitrig.

ok deraadt@, guenther@

compress code by turning four line comments into one line comments.
emphatic ok usual suspects, grudging ok miod

No need to cast constants or simple variables to (daddr_t). Use
(u_int64_t) instead of (daddr_t) when casting a variable in an
expression passed to DL_SETDSIZE().

Change a variable counting open files from daddr_t to int64_t.

ok deraadt@ with the tweak to fix that pesky expression.

final removal of daddr64_t. daddr_t has been 64 bit for a long enough
test period; i think 3 years ago the last bugs fell out.
ok otto beck others

remove lots of comments about locking per beck's request

remove simple_locks from uvm code. ok beck deraadt

Rip out and burn support for UVM_HIST.

The vm hackers don't use it, don't maintain it and have to look at it all the
time. About time this 800 lines of code hit /dev/null.

``never liked it'' tedu@. ariane@ was very happy when i told her i wrote
this diff.

for uvm_pager_dropcluster in the PG_RELEASED case we specifically unbusy the
page so that um_anfree will free it for us.

uvm_anfree does a pmap_page_protect(, VM_PROT_NONE) just before it frees the
page, so we don't need to do it here ourselves.

ok ariane@

move uvm_pageratop from uvm_pager.c local to a general uvm function
(uvm_atopg) and use it in uvm_km_doputpage to replace some handrolled
code. Shrinks the kernel a trivial amount.

ok beck@ and miod@ (who suggested i name it uvm_atopg not uvm_atop)

Don't sleep while holding the uvm_psel_lck mutex. Should fix "locking against
myself" panics that some people have seen over the last year-and-a-half.

Cherry picked from a more complex (and therefore scarier) diff from oga@.

ok tedu@, oga@

Kill another #ifdef UBC chunk that was annoying me while doing something
else.

ok thib@

introduce a uvm_km_valloc_try function that won't get a lower level lock
for use by the uvm pseg code. this is the path of least resistance until
we sort out how many of these functions we really need. problem found by mikeb
ok kettenis oga

Put the PG_RELEASED changes diff back in.

This has has been tested very very thoroughly on all archs we have
excepting 88k and 68k. Please see cvs log for the individual commit
messages.

ok beck@, thib@

date based reversion of uvm to the 4th May.

More backouts in line with previous ones, this appears to bring us back to a
stable condition.

A machine forced to 64mb of ram cycled 10GB through swap with this diff
and is still running as I type this. Other tests by ariane@ and thib@
also seem to show that it's alright.

ok deraadt@, thib@, ariane@

Backout all the PG_RELEASED changes.

This is for the same reason as the earlier backouts, to avoid the bug
either added or exposed sometime around c2k9. This *should* be the last
one.

prompted by deraadt@

ok ariane@

More PG_RELEASED cleaning.

similar to the aobj.c changes, this one does vnodes. Vnodes are more
complex because they actaully have to sync to backing store. So firstly,
convert sync to sleep instead of setting released.

Now, for backing store, in the PGO_FREE case, if we set PG_RELEASED
*before* an async io, (nothing else will see it, the page is busy), then
we can ignore the page after the io is done. We could do something
similar for PGO_DEACTIVATE too, but that is another change. On error we
just clear the released flag, nothing else sets it for uobj pages other
than aiodoned.

ok thib@, beck@, ariane@

Instead of keeping two ints in the uvm structure specifically just to
sleep on them (and otherwise ignore them) sleep on the pointer to the
{aiodoned,pagedaemon}_proc members, and nuke the two extra words.

"no objections" art@, ok beck@.

Instead of doing splbio(); simple_lock(&uvm.aiodoned_lock); just replace
the simple lock with a real lock - a IPL_BIO mutex. While i'm here, make
the sleeping condition one hell of a lot simpler in the aio daemon.

some ideas from and ok art@.

Move all of the pseudo-inline functions in uvm into C files.

By pseudo-inline, I mean that if a certain macro was defined, they would
be inlined. However, no architecture defines that, and none has for a
very very long time. Therefore mainly this just makes the code a damned
sight easier to read. Some k&r -> ansi declarations while I'm in there.

"just commit it" art@. ok weingart@.

While working on some stuff in uvm I've gotten REALLY sick of reading
K&R function declarations, so switch them all over to ansi-style, in
accordance with the prophesy.

"go for it" art@

Simplify page-out/page-in map management; fix rare pager deadlock.

Ok: miod, tedu

garbage collect uvm_errno2vmerror();

ok miod@, art@

In uvm_aio_aiodone(), kill the variable-sized array on stack, and use a
fixed size array which size should match any buf; if a bogus buf is passed
to this function, the kernel will KASSERT instead of potentially running out
of stack and having an undefined behaviour.

ok deraadt@

now that all partition size/offsets are potentially 64-bit, change the
type of all variables to daddr64_t. this includes the APIs for XXsize()
and XXdump(), all range checks inside bio drivers, internal variables
for disklabel handling, and even uvm's swap offsets. re-read numerous
times by otto, miod, krw, thib to look for errors

While splitting flags and pqflags might have been a good idea in theory
to separate locking, on most modern machines this is not enough
since operations on short types touch other short types that share the
same word in memory.

Merge pg_flags and pqflags again and now use atomic operations to change
the flags. Also bump wire_count to an int and pg_version might go
int as well, just for alignment.

tested by many, many. ok miod@

Mechanically rename the "flags" and "version" fields in struct vm_page
to "pg_flags" and "pg_version", so that they are a bit easier to work with.
Whoever uses generic names like this for a popular struct obviously doesn't
read much code.

Most architectures compile and there are no functionality changes.

deraadt@ ok ("if something fails to compile, we fix that by hand")

remove KERN_SUCCESS and use 0 instead.
eyeballed by miod@ and pedro@

fix uvmhist #2: args are always u_long so fix missing %d and %x and no %ll; no change for normal code

fix fmts for UVMHIST_LOG() entries making it more useful on 64bit archs; miod@ ok

- Make a UVM_OBJ_IS_DEVICE macro.
- Use it to skip device mappings while dumping core.
- Ignore EFAULT errors while dumping core since they can happen
even for valid mappings. Just skip that part of the core file and
let it get automagically zero-filled.

This fixes the broken X core dumps that people have been seeing and also
fixes some other potential problems that could prevent core dumps (mmaps
beyond EOF, etc.).

tedu@ ok

add a new field to vm_space and use it to track the number of anon
pages a process uses. this is now the userland "data size" value.
ok art deraadt tdeval. thanks testers.

sync of pmap_update() calls with NetBSD. pmap_update is defined away on
all architectures but arm, where it is needed.

ubchist is not a fully cooked kadaver and though use the other well formed pdhist one until ubc gaets back. art@ ok

Since memory deallocation can't fail, remove the error return from
uvm_unmap, uvm_deallocate and a few other functions.
Simplifies some code and reduces diff to the UBC branch.

Make sure that b_iodone handlers are called at splbio (and splassert(IPL_BIO) in all known callers, just to make sure).

splassert(IPL_BIO) in the b_iodone handlers.

Back out a few more uvm changes, especially wrt swap usage.
This unbreaks m68k m88k sparc and perhaps others, which eventually froze
when hitting swap.
Tested by various people on various platforms.
ok art@

UBC was a disaster. It worked very good when it worked, but on some
machines or some configurations or in some phase of the moon (we actually
don't know when or why) files disappeared. Since we've not been able to
track down the problem in two weeks intense debugging and we need -current
to be stable, back out everything to a state it had before UBC.

We apologise for the inconvenience.

branches: 1.28.2;
Yet another sync to NetBSD uvm.
Today we add a pmap argument to pmap_update() and allocate map entries for
kernel_map from kmem_map instead of using the static entries. This should
get rid of MAX_KMAPENT panics. Also some uvm_loan problems are fixed.

Call buf_cleanout, which handles wakeups

Correctly handle b_vp with bgetvp and brelvp in {get,put}pages.
Prevents panics caused by vnodes being recycled under our feet.

Sync in more uvm from NetBSD. Mostly just cosmetic stuff.
Contains also support for page coloring.

Sync in more uvm changes from NetBSD.
This time we're getting rid of KERN_* and VM_PAGER_* error codes and
use errnos instead.

Merge in the unified buffer cache code as found in NetBSD 2001/03/10. The
code is written mostly by Chuck Silvers <chuq@chuq.com>/<chs@netbsd.org>.

Tested for the past few weeks by many developers, should be in a pretty stable
state, but will require optimizations and additional cleanups.

Bring in more changes from NetBSD. Mostly pagedaemon improvements.

Merge in some parts of the ubc work that has been done in NetBSD that are not
UBC, but prerequsites for it.

- Create a daemon that processes async I/O (swap and paging in the future)
requests that need processing in process context and that were processed
in the pagedaemon before.
- Convert some ugly ifdef DIAGNOSTIC code to less intrusive KASSERTs.
- misc other cleanups.

Another sync of uvm to NetBSD. Just minor fiddling, no major changes.

Add an alignment argument to uvm_map that specifies an alignment hint
for the virtual address.

Move the last content from vm/ to uvm/
The only thing left in vm/ are just dumb wrappers.
vm/vm.h includes uvm/uvm_extern.h
vm/pmap.h includes uvm/uvm_pmap.h
vm/vm_page.h includes uvm/uvm_page.h

Minor sync to NetBSD.

occured->occurred

idea from deraadt@ via NetBSD
millert@ ok

merge vm/vm_kern.h into uvm/uvm_extern.h; art@ ok

Various random fixes from NetBSD.
Including support for zeroing pages in the idle loop (not enabled yet).

Add a new type voff_t (right now it's typedefed as off_t) used for offsets
into objects.

Gives the possibilty to mmap beyond the size of vaddr_t.

From NetBSD.

Add support for disabling swap devices (swapctl -d).
Improve error handling on I/O errors to swap.
From NetBSD

Change the pmap_enter interface to merge access_type and the wired boolean
and arbitrary flags into one argument.

One new flag is PMAP_CANFAIL that tells pmap_enter that it can fail if there
are not enough resources to satisfy the request. If this flag is not passed,
pmap_enter should panic as it should have done before this change (XXX - many
pmaps are still not doing that).

Only i386 and alpha implement CANFAIL for now.

Includes uvm updates from NetBSD.

Get rid of the PMAP_NEW option by making it mandatory for all archs.
The archs that didn't have a proper PMAP_NEW now have a dummy implementation
with wrappers around the old functions.

Change the paddr_t pmap_extract(struct pmap *, vaddr_t) interface to
boolean_t pmap_extract(struct pmap *, vaddr_t, paddr_t *).
Matches NetBSD. Tested by various people on various platforms.

Sync style, typo, and comments a little closer to NetBSD. art@ ok

Replace thread_wakeup() with wakeup(). It is defined in vm_extern.h as a
wrapper, so this removes a dependence on the old VM system. From NetBSD.
art@ ok

$OpenBSD$

Bring in some new UVM code from NetBSD (not current).

- Introduce a new type of map that are interrupt safe and never allow faults
in them. mb_map and kmem_map are made intrsafe.
- Add "access protection" to uvm_vslock (to be passed down to uvm_fault and
later to pmap_enter).
- madvise(2) now works.
- various cleanups.

branches: 1.4.4;
Change the pmap_enter api to pass down an argument that indicates
the access type that caused this mapping. This is to simplify pmaps
with mod/ref emulation (none for the moment) and in some cases speed
up pmap_is_{referenced,modified}.
At the same time, clean up some mappings that had too high protection.

XXX - the access type is incorrect in old vm, it's only used by uvm and MD code.
The actual use of this in pmap_enter implementations is not in this commit.

sync with NetBSD from 1999.05.24 (there is a reason for this date)
Mostly cleanups, but also a few improvements to pagedaemon for better
handling of low memory and/or low swap conditions.

add OpenBSD tags

Import of uvm from NetBSD. Some local changes, some code disabled

Grab the mutex before iterating on the global list.

Prevent race reported some months ago.

ok mlarkin@, jsg@, kn@

remove unused uvm_aiobuf_pool

Use PMAP_PREFER_ALIGN() == 0 rather than !defined(PMAP_PREFER) to enable the
fast path in the pager code; this benefits most mips64 platforms.

ok kettenis@ mpi@

Bring back the direct map optimization when PMAP_PREFER isn't defined.

PMAP_PREFER is checked to prevent cache aliasing issues when the physical
page is mapped at different virtual addresses.

This speeds up file-based mmap faults.

ok kettenis@

Revert simplification of the aiodone daemon it breaks swap on arm64.

Found the hard way by mlarkin@ and deraadt@.

Revert the changes made in rev 1.82. It is important to use pmap_enter(9)
and pmap_remove(9) here since we're dealing with managed pages here. Found
out the hard way by deraadt@ on landisk where we're running into issues
with virtual cache aliases because multiple mappings exist for the
pages we're dealing with here. The pmap_enter(9) and pmap_remove(9)
functions handle conflicting cache aliases, whereas pmap_map_direct(9) and
pmap_kenter_pa(9) assume that the pages is exclusively mapped in the kernel
pmap.

ok deraadt@

Simplify the aiodone daemon which is only used for async writes.

- Remove unused support for asynchronous read, including error conditions

- Grab the proper lock for each page that has been written to swap. This
allows to enable an assertion in uvm_page_unbusy().

- Move the uvm_anon_release() call outside of uvm_page_unbusy() and
assert for the different anon cases.

ok beck@, kettenis@

Reduce allocations and possible failures in uvm_pagermapin/out().

. If a direct map exists use it to map single-page allocations
. Use pmap_kenter_pa() instead of pmap_enter() in all other cases.

This speeds up file-based mmap up to 75% when I/O are performed and it
also reduces possible allocations failtures in the page daemon making
it more stable in OOM situations.

ok kettenis@, beck@

Reserve a second segment for the page daemon.

This ensures uvm_swap_io() can succeeds even in OOM situations because two
uvm_pagermapin() allocations, requiring a segment each, are needed to bounce
or encrypt a swap cluster.

ok beck@, kettenis@

Move the guts of uvm_aio_aiodone() into a separate function.

This will help dealing with error cases.

No functionnal changes.

Do not clean PG_BUSY before calling uvm_anon_release().

Fix an assertion reported by gkoehler@.

ok kettenis@

Convert KVA allocation to km_alloc(9).

ok mpi@

Use a per-UVM object lock to serialize the lower part of the fault handler.

Like the per-amap lock the `vmobjlock' is principally used to serialized
access to objects in the fault handler to allow faults occurring on
different CPUs and different objects to be processed in parallel.

The fault handler now acquires the `vmobjlock' of a given UVM object as
soon as it finds one. For now a write-lock is always acquired even if
some operations could use a read-lock.

Every pager, corresponding to a different kind of UVM object, now expect
the UVM object to be locked and some operations, like *_get() return it
unlocked. This is enforced by assertions checking for rw_write_held().

The KERNEL_LOCK() is now pushed to the VFS boundary in the vnode pager.

To ensure the correct amap or object lock is held when modifying a page
many uvm_page* operations are now asserting for the "owner" lock.
However, fields of the "struct vm_page" are still being protected by the
global `pageqlock'. To prevent lock ordering issues with the new
`vmobjlock' and to reduce differences with NetBSD this lock is now taken
and released for each page instead of around the whole loop.

This commit does not remove the KERNEL_LOCK/UNLOCK() dance. Unlocking
will follow if there is no fallout.

Ported from NetBSD, tested by many, thanks!

ok kettenis@, kn@

Remove parenthesis around return value to reduce the diff with NetBSD.

No functional change.

ok mlarkin@

spelling

ok mpi@

(re)Introduce locking for amaps & anons.

A rwlock is attached to every amap and is shared with all its anon. The
same lock will be used by multiple amaps if they have anons in common.

This should be enough to get the upper part of the fault handler out of the
KERNEL_LOCK() which seems to bring up to 20% improvements in builds.

This is based/copied/adapted from the most recent work done in NetBSD which
is an evolution of the precendent simple_lock scheme.

Tested by many, thanks!

ok kettenis@, mvs@

Constify and use C99 initializer for "struct uvm_pagerops".

While here put some KERNEL_ASSERT_LOCKED() in the functions called from
the page fault handler. The removal of locking of `uobj' will need to be
revisited and these are good indicator that something is missing and that
many comments are lying.

ok kettenis

Convert infinite sleeps to {m,t}sleep_nsec(9).

ok visa@, jca@

remove lock.h from uvm_extern.h. another holdover from the simpletonlock
era. fix uvm including c files to include lock.h or atomic.h as necessary.
ok deraadt

Replace a plethora of historical protection options with just
PROT_NONE, PROT_READ, PROT_WRITE, and PROT_EXEC from mman.h.
PROT_MASK is introduced as the one true way of extracting those bits.
Remove UVM_ADV_* wrapper, using the standard names.
ok doug guenther kettenis

remove uneeded proc.h includes
ok mpi@ kspillner@

Chuck Cranor rescinded clauses in his license
on the 2nd of February 2011 in NetBSD.

http://marc.info/?l=netbsd-source-changes&m=129658899212732&w=2
http://marc.info/?l=netbsd-source-changes&m=129659095515558&w=2
http://marc.info/?l=netbsd-source-changes&m=129659157916514&w=2
http://marc.info/?l=netbsd-source-changes&m=129665962324372&w=2
http://marc.info/?l=netbsd-source-changes&m=129666033625342&w=2
http://marc.info/?l=netbsd-source-changes&m=129666052825545&w=2
http://marc.info/?l=netbsd-source-changes&m=129666922906480&w=2
http://marc.info/?l=netbsd-source-changes&m=129667725518082&w=2

stop using B_AGE, it was effectively retired some time ago.

Fix some potential integer overflows caused by converting a page number into
an offset/size/address by shifting by PAGE_SHIFT. Make uvm_objwrire/unwire
use voff_t instead of off_t. The former is the right type here even if it is
equivalent to the latter.

Inspired by a somewhat similar changes in Bitrig.

ok deraadt@, guenther@

compress code by turning four line comments into one line comments.
emphatic ok usual suspects, grudging ok miod

No need to cast constants or simple variables to (daddr_t). Use
(u_int64_t) instead of (daddr_t) when casting a variable in an
expression passed to DL_SETDSIZE().

Change a variable counting open files from daddr_t to int64_t.

ok deraadt@ with the tweak to fix that pesky expression.

final removal of daddr64_t. daddr_t has been 64 bit for a long enough
test period; i think 3 years ago the last bugs fell out.
ok otto beck others

remove lots of comments about locking per beck's request

remove simple_locks from uvm code. ok beck deraadt

Rip out and burn support for UVM_HIST.

The vm hackers don't use it, don't maintain it and have to look at it all the
time. About time this 800 lines of code hit /dev/null.

``never liked it'' tedu@. ariane@ was very happy when i told her i wrote
this diff.

for uvm_pager_dropcluster in the PG_RELEASED case we specifically unbusy the
page so that um_anfree will free it for us.

uvm_anfree does a pmap_page_protect(, VM_PROT_NONE) just before it frees the
page, so we don't need to do it here ourselves.

ok ariane@

move uvm_pageratop from uvm_pager.c local to a general uvm function
(uvm_atopg) and use it in uvm_km_doputpage to replace some handrolled
code. Shrinks the kernel a trivial amount.

ok beck@ and miod@ (who suggested i name it uvm_atopg not uvm_atop)

Don't sleep while holding the uvm_psel_lck mutex. Should fix "locking against
myself" panics that some people have seen over the last year-and-a-half.

Cherry picked from a more complex (and therefore scarier) diff from oga@.

ok tedu@, oga@

Kill another #ifdef UBC chunk that was annoying me while doing something
else.

ok thib@

introduce a uvm_km_valloc_try function that won't get a lower level lock
for use by the uvm pseg code. this is the path of least resistance until
we sort out how many of these functions we really need. problem found by mikeb
ok kettenis oga

Put the PG_RELEASED changes diff back in.

This has has been tested very very thoroughly on all archs we have
excepting 88k and 68k. Please see cvs log for the individual commit
messages.

ok beck@, thib@

date based reversion of uvm to the 4th May.

More backouts in line with previous ones, this appears to bring us back to a
stable condition.

A machine forced to 64mb of ram cycled 10GB through swap with this diff
and is still running as I type this. Other tests by ariane@ and thib@
also seem to show that it's alright.

ok deraadt@, thib@, ariane@

Backout all the PG_RELEASED changes.

This is for the same reason as the earlier backouts, to avoid the bug
either added or exposed sometime around c2k9. This *should* be the last
one.

prompted by deraadt@

ok ariane@

More PG_RELEASED cleaning.

similar to the aobj.c changes, this one does vnodes. Vnodes are more
complex because they actaully have to sync to backing store. So firstly,
convert sync to sleep instead of setting released.

Now, for backing store, in the PGO_FREE case, if we set PG_RELEASED
*before* an async io, (nothing else will see it, the page is busy), then
we can ignore the page after the io is done. We could do something
similar for PGO_DEACTIVATE too, but that is another change. On error we
just clear the released flag, nothing else sets it for uobj pages other
than aiodoned.

ok thib@, beck@, ariane@

Instead of keeping two ints in the uvm structure specifically just to
sleep on them (and otherwise ignore them) sleep on the pointer to the
{aiodoned,pagedaemon}_proc members, and nuke the two extra words.

"no objections" art@, ok beck@.

Instead of doing splbio(); simple_lock(&uvm.aiodoned_lock); just replace
the simple lock with a real lock - a IPL_BIO mutex. While i'm here, make
the sleeping condition one hell of a lot simpler in the aio daemon.

some ideas from and ok art@.

Move all of the pseudo-inline functions in uvm into C files.

By pseudo-inline, I mean that if a certain macro was defined, they would
be inlined. However, no architecture defines that, and none has for a
very very long time. Therefore mainly this just makes the code a damned
sight easier to read. Some k&r -> ansi declarations while I'm in there.

"just commit it" art@. ok weingart@.

While working on some stuff in uvm I've gotten REALLY sick of reading
K&R function declarations, so switch them all over to ansi-style, in
accordance with the prophesy.

"go for it" art@

Simplify page-out/page-in map management; fix rare pager deadlock.

Ok: miod, tedu

garbage collect uvm_errno2vmerror();

ok miod@, art@

In uvm_aio_aiodone(), kill the variable-sized array on stack, and use a
fixed size array which size should match any buf; if a bogus buf is passed
to this function, the kernel will KASSERT instead of potentially running out
of stack and having an undefined behaviour.

ok deraadt@

now that all partition size/offsets are potentially 64-bit, change the
type of all variables to daddr64_t. this includes the APIs for XXsize()
and XXdump(), all range checks inside bio drivers, internal variables
for disklabel handling, and even uvm's swap offsets. re-read numerous
times by otto, miod, krw, thib to look for errors

While splitting flags and pqflags might have been a good idea in theory
to separate locking, on most modern machines this is not enough
since operations on short types touch other short types that share the
same word in memory.

Merge pg_flags and pqflags again and now use atomic operations to change
the flags. Also bump wire_count to an int and pg_version might go
int as well, just for alignment.

tested by many, many. ok miod@

Mechanically rename the "flags" and "version" fields in struct vm_page
to "pg_flags" and "pg_version", so that they are a bit easier to work with.
Whoever uses generic names like this for a popular struct obviously doesn't
read much code.

Most architectures compile and there are no functionality changes.

deraadt@ ok ("if something fails to compile, we fix that by hand")

remove KERN_SUCCESS and use 0 instead.
eyeballed by miod@ and pedro@

fix uvmhist #2: args are always u_long so fix missing %d and %x and no %ll; no change for normal code

fix fmts for UVMHIST_LOG() entries making it more useful on 64bit archs; miod@ ok

- Make a UVM_OBJ_IS_DEVICE macro.
- Use it to skip device mappings while dumping core.
- Ignore EFAULT errors while dumping core since they can happen
even for valid mappings. Just skip that part of the core file and
let it get automagically zero-filled.

This fixes the broken X core dumps that people have been seeing and also
fixes some other potential problems that could prevent core dumps (mmaps
beyond EOF, etc.).

tedu@ ok

add a new field to vm_space and use it to track the number of anon
pages a process uses. this is now the userland "data size" value.
ok art deraadt tdeval. thanks testers.

sync of pmap_update() calls with NetBSD. pmap_update is defined away on
all architectures but arm, where it is needed.

ubchist is not a fully cooked kadaver and though use the other well formed pdhist one until ubc gaets back. art@ ok

Since memory deallocation can't fail, remove the error return from
uvm_unmap, uvm_deallocate and a few other functions.
Simplifies some code and reduces diff to the UBC branch.

Make sure that b_iodone handlers are called at splbio (and splassert(IPL_BIO) in all known callers, just to make sure).

splassert(IPL_BIO) in the b_iodone handlers.

Back out a few more uvm changes, especially wrt swap usage.
This unbreaks m68k m88k sparc and perhaps others, which eventually froze
when hitting swap.
Tested by various people on various platforms.
ok art@

UBC was a disaster. It worked very good when it worked, but on some
machines or some configurations or in some phase of the moon (we actually
don't know when or why) files disappeared. Since we've not been able to
track down the problem in two weeks intense debugging and we need -current
to be stable, back out everything to a state it had before UBC.

We apologise for the inconvenience.

branches: 1.28.2;
Yet another sync to NetBSD uvm.
Today we add a pmap argument to pmap_update() and allocate map entries for
kernel_map from kmem_map instead of using the static entries. This should
get rid of MAX_KMAPENT panics. Also some uvm_loan problems are fixed.

Call buf_cleanout, which handles wakeups

Correctly handle b_vp with bgetvp and brelvp in {get,put}pages.
Prevents panics caused by vnodes being recycled under our feet.

Sync in more uvm from NetBSD. Mostly just cosmetic stuff.
Contains also support for page coloring.

Sync in more uvm changes from NetBSD.
This time we're getting rid of KERN_* and VM_PAGER_* error codes and
use errnos instead.

Merge in the unified buffer cache code as found in NetBSD 2001/03/10. The
code is written mostly by Chuck Silvers <chuq@chuq.com>/<chs@netbsd.org>.

Tested for the past few weeks by many developers, should be in a pretty stable
state, but will require optimizations and additional cleanups.

Bring in more changes from NetBSD. Mostly pagedaemon improvements.

Merge in some parts of the ubc work that has been done in NetBSD that are not
UBC, but prerequsites for it.

- Create a daemon that processes async I/O (swap and paging in the future)
requests that need processing in process context and that were processed
in the pagedaemon before.
- Convert some ugly ifdef DIAGNOSTIC code to less intrusive KASSERTs.
- misc other cleanups.

Another sync of uvm to NetBSD. Just minor fiddling, no major changes.

Add an alignment argument to uvm_map that specifies an alignment hint
for the virtual address.

Move the last content from vm/ to uvm/
The only thing left in vm/ are just dumb wrappers.
vm/vm.h includes uvm/uvm_extern.h
vm/pmap.h includes uvm/uvm_pmap.h
vm/vm_page.h includes uvm/uvm_page.h

Minor sync to NetBSD.

occured->occurred

idea from deraadt@ via NetBSD
millert@ ok

merge vm/vm_kern.h into uvm/uvm_extern.h; art@ ok

Various random fixes from NetBSD.
Including support for zeroing pages in the idle loop (not enabled yet).

Add a new type voff_t (right now it's typedefed as off_t) used for offsets
into objects.

Gives the possibilty to mmap beyond the size of vaddr_t.

From NetBSD.

Add support for disabling swap devices (swapctl -d).
Improve error handling on I/O errors to swap.
From NetBSD

Change the pmap_enter interface to merge access_type and the wired boolean
and arbitrary flags into one argument.

One new flag is PMAP_CANFAIL that tells pmap_enter that it can fail if there
are not enough resources to satisfy the request. If this flag is not passed,
pmap_enter should panic as it should have done before this change (XXX - many
pmaps are still not doing that).

Only i386 and alpha implement CANFAIL for now.

Includes uvm updates from NetBSD.

Get rid of the PMAP_NEW option by making it mandatory for all archs.
The archs that didn't have a proper PMAP_NEW now have a dummy implementation
with wrappers around the old functions.

Change the paddr_t pmap_extract(struct pmap *, vaddr_t) interface to
boolean_t pmap_extract(struct pmap *, vaddr_t, paddr_t *).
Matches NetBSD. Tested by various people on various platforms.

Sync style, typo, and comments a little closer to NetBSD. art@ ok

Replace thread_wakeup() with wakeup(). It is defined in vm_extern.h as a
wrapper, so this removes a dependence on the old VM system. From NetBSD.
art@ ok

$OpenBSD$

Bring in some new UVM code from NetBSD (not current).

- Introduce a new type of map that are interrupt safe and never allow faults
in them. mb_map and kmem_map are made intrsafe.
- Add "access protection" to uvm_vslock (to be passed down to uvm_fault and
later to pmap_enter).
- madvise(2) now works.
- various cleanups.

branches: 1.4.4;
Change the pmap_enter api to pass down an argument that indicates
the access type that caused this mapping. This is to simplify pmaps
with mod/ref emulation (none for the moment) and in some cases speed
up pmap_is_{referenced,modified}.
At the same time, clean up some mappings that had too high protection.

XXX - the access type is incorrect in old vm, it's only used by uvm and MD code.
The actual use of this in pmap_enter implementations is not in this commit.

sync with NetBSD from 1999.05.24 (there is a reason for this date)
Mostly cleanups, but also a few improvements to pagedaemon for better
handling of low memory and/or low swap conditions.

add OpenBSD tags

Import of uvm from NetBSD. Some local changes, some code disabled

remove unused uvm_aiobuf_pool

Use PMAP_PREFER_ALIGN() == 0 rather than !defined(PMAP_PREFER) to enable the
fast path in the pager code; this benefits most mips64 platforms.

ok kettenis@ mpi@

Bring back the direct map optimization when PMAP_PREFER isn't defined.

PMAP_PREFER is checked to prevent cache aliasing issues when the physical
page is mapped at different virtual addresses.

This speeds up file-based mmap faults.

ok kettenis@

Revert simplification of the aiodone daemon it breaks swap on arm64.

Found the hard way by mlarkin@ and deraadt@.

Revert the changes made in rev 1.82. It is important to use pmap_enter(9)
and pmap_remove(9) here since we're dealing with managed pages here. Found
out the hard way by deraadt@ on landisk where we're running into issues
with virtual cache aliases because multiple mappings exist for the
pages we're dealing with here. The pmap_enter(9) and pmap_remove(9)
functions handle conflicting cache aliases, whereas pmap_map_direct(9) and
pmap_kenter_pa(9) assume that the pages is exclusively mapped in the kernel
pmap.

ok deraadt@

Simplify the aiodone daemon which is only used for async writes.

- Remove unused support for asynchronous read, including error conditions

- Grab the proper lock for each page that has been written to swap. This
allows to enable an assertion in uvm_page_unbusy().

- Move the uvm_anon_release() call outside of uvm_page_unbusy() and
assert for the different anon cases.

ok beck@, kettenis@

Reduce allocations and possible failures in uvm_pagermapin/out().

. If a direct map exists use it to map single-page allocations
. Use pmap_kenter_pa() instead of pmap_enter() in all other cases.

This speeds up file-based mmap up to 75% when I/O are performed and it
also reduces possible allocations failtures in the page daemon making
it more stable in OOM situations.

ok kettenis@, beck@

Reserve a second segment for the page daemon.

This ensures uvm_swap_io() can succeeds even in OOM situations because two
uvm_pagermapin() allocations, requiring a segment each, are needed to bounce
or encrypt a swap cluster.

ok beck@, kettenis@

Move the guts of uvm_aio_aiodone() into a separate function.

This will help dealing with error cases.

No functionnal changes.

Do not clean PG_BUSY before calling uvm_anon_release().

Fix an assertion reported by gkoehler@.

ok kettenis@

Convert KVA allocation to km_alloc(9).

ok mpi@

Use a per-UVM object lock to serialize the lower part of the fault handler.

Like the per-amap lock the `vmobjlock' is principally used to serialized
access to objects in the fault handler to allow faults occurring on
different CPUs and different objects to be processed in parallel.

The fault handler now acquires the `vmobjlock' of a given UVM object as
soon as it finds one. For now a write-lock is always acquired even if
some operations could use a read-lock.

Every pager, corresponding to a different kind of UVM object, now expect
the UVM object to be locked and some operations, like *_get() return it
unlocked. This is enforced by assertions checking for rw_write_held().

The KERNEL_LOCK() is now pushed to the VFS boundary in the vnode pager.

To ensure the correct amap or object lock is held when modifying a page
many uvm_page* operations are now asserting for the "owner" lock.
However, fields of the "struct vm_page" are still being protected by the
global `pageqlock'. To prevent lock ordering issues with the new
`vmobjlock' and to reduce differences with NetBSD this lock is now taken
and released for each page instead of around the whole loop.

This commit does not remove the KERNEL_LOCK/UNLOCK() dance. Unlocking
will follow if there is no fallout.

Ported from NetBSD, tested by many, thanks!

ok kettenis@, kn@

Remove parenthesis around return value to reduce the diff with NetBSD.

No functional change.

ok mlarkin@

spelling

ok mpi@

(re)Introduce locking for amaps & anons.

A rwlock is attached to every amap and is shared with all its anon. The
same lock will be used by multiple amaps if they have anons in common.

This should be enough to get the upper part of the fault handler out of the
KERNEL_LOCK() which seems to bring up to 20% improvements in builds.

This is based/copied/adapted from the most recent work done in NetBSD which
is an evolution of the precendent simple_lock scheme.

Tested by many, thanks!

ok kettenis@, mvs@

Constify and use C99 initializer for "struct uvm_pagerops".

While here put some KERNEL_ASSERT_LOCKED() in the functions called from
the page fault handler. The removal of locking of `uobj' will need to be
revisited and these are good indicator that something is missing and that
many comments are lying.

ok kettenis

Convert infinite sleeps to {m,t}sleep_nsec(9).

ok visa@, jca@

remove lock.h from uvm_extern.h. another holdover from the simpletonlock
era. fix uvm including c files to include lock.h or atomic.h as necessary.
ok deraadt

Replace a plethora of historical protection options with just
PROT_NONE, PROT_READ, PROT_WRITE, and PROT_EXEC from mman.h.
PROT_MASK is introduced as the one true way of extracting those bits.
Remove UVM_ADV_* wrapper, using the standard names.
ok doug guenther kettenis

remove uneeded proc.h includes
ok mpi@ kspillner@

Chuck Cranor rescinded clauses in his license
on the 2nd of February 2011 in NetBSD.

http://marc.info/?l=netbsd-source-changes&m=129658899212732&w=2
http://marc.info/?l=netbsd-source-changes&m=129659095515558&w=2
http://marc.info/?l=netbsd-source-changes&m=129659157916514&w=2
http://marc.info/?l=netbsd-source-changes&m=129665962324372&w=2
http://marc.info/?l=netbsd-source-changes&m=129666033625342&w=2
http://marc.info/?l=netbsd-source-changes&m=129666052825545&w=2
http://marc.info/?l=netbsd-source-changes&m=129666922906480&w=2
http://marc.info/?l=netbsd-source-changes&m=129667725518082&w=2

stop using B_AGE, it was effectively retired some time ago.

Fix some potential integer overflows caused by converting a page number into
an offset/size/address by shifting by PAGE_SHIFT. Make uvm_objwrire/unwire
use voff_t instead of off_t. The former is the right type here even if it is
equivalent to the latter.

Inspired by a somewhat similar changes in Bitrig.

ok deraadt@, guenther@

compress code by turning four line comments into one line comments.
emphatic ok usual suspects, grudging ok miod

No need to cast constants or simple variables to (daddr_t). Use
(u_int64_t) instead of (daddr_t) when casting a variable in an
expression passed to DL_SETDSIZE().

Change a variable counting open files from daddr_t to int64_t.

ok deraadt@ with the tweak to fix that pesky expression.

final removal of daddr64_t. daddr_t has been 64 bit for a long enough
test period; i think 3 years ago the last bugs fell out.
ok otto beck others

remove lots of comments about locking per beck's request

remove simple_locks from uvm code. ok beck deraadt

Rip out and burn support for UVM_HIST.

The vm hackers don't use it, don't maintain it and have to look at it all the
time. About time this 800 lines of code hit /dev/null.

``never liked it'' tedu@. ariane@ was very happy when i told her i wrote
this diff.

for uvm_pager_dropcluster in the PG_RELEASED case we specifically unbusy the
page so that um_anfree will free it for us.

uvm_anfree does a pmap_page_protect(, VM_PROT_NONE) just before it frees the
page, so we don't need to do it here ourselves.

ok ariane@

move uvm_pageratop from uvm_pager.c local to a general uvm function
(uvm_atopg) and use it in uvm_km_doputpage to replace some handrolled
code. Shrinks the kernel a trivial amount.

ok beck@ and miod@ (who suggested i name it uvm_atopg not uvm_atop)

Don't sleep while holding the uvm_psel_lck mutex. Should fix "locking against
myself" panics that some people have seen over the last year-and-a-half.

Cherry picked from a more complex (and therefore scarier) diff from oga@.

ok tedu@, oga@

Kill another #ifdef UBC chunk that was annoying me while doing something
else.

ok thib@

introduce a uvm_km_valloc_try function that won't get a lower level lock
for use by the uvm pseg code. this is the path of least resistance until
we sort out how many of these functions we really need. problem found by mikeb
ok kettenis oga

Put the PG_RELEASED changes diff back in.

This has has been tested very very thoroughly on all archs we have
excepting 88k and 68k. Please see cvs log for the individual commit
messages.

ok beck@, thib@

date based reversion of uvm to the 4th May.

More backouts in line with previous ones, this appears to bring us back to a
stable condition.

A machine forced to 64mb of ram cycled 10GB through swap with this diff
and is still running as I type this. Other tests by ariane@ and thib@
also seem to show that it's alright.

ok deraadt@, thib@, ariane@

Backout all the PG_RELEASED changes.

This is for the same reason as the earlier backouts, to avoid the bug
either added or exposed sometime around c2k9. This *should* be the last
one.

prompted by deraadt@

ok ariane@

More PG_RELEASED cleaning.

similar to the aobj.c changes, this one does vnodes. Vnodes are more
complex because they actaully have to sync to backing store. So firstly,
convert sync to sleep instead of setting released.

Now, for backing store, in the PGO_FREE case, if we set PG_RELEASED
*before* an async io, (nothing else will see it, the page is busy), then
we can ignore the page after the io is done. We could do something
similar for PGO_DEACTIVATE too, but that is another change. On error we
just clear the released flag, nothing else sets it for uobj pages other
than aiodoned.

ok thib@, beck@, ariane@

Instead of keeping two ints in the uvm structure specifically just to
sleep on them (and otherwise ignore them) sleep on the pointer to the
{aiodoned,pagedaemon}_proc members, and nuke the two extra words.

"no objections" art@, ok beck@.

Instead of doing splbio(); simple_lock(&uvm.aiodoned_lock); just replace
the simple lock with a real lock - a IPL_BIO mutex. While i'm here, make
the sleeping condition one hell of a lot simpler in the aio daemon.

some ideas from and ok art@.

Move all of the pseudo-inline functions in uvm into C files.

By pseudo-inline, I mean that if a certain macro was defined, they would
be inlined. However, no architecture defines that, and none has for a
very very long time. Therefore mainly this just makes the code a damned
sight easier to read. Some k&r -> ansi declarations while I'm in there.

"just commit it" art@. ok weingart@.

While working on some stuff in uvm I've gotten REALLY sick of reading
K&R function declarations, so switch them all over to ansi-style, in
accordance with the prophesy.

"go for it" art@

Simplify page-out/page-in map management; fix rare pager deadlock.

Ok: miod, tedu

garbage collect uvm_errno2vmerror();

ok miod@, art@

In uvm_aio_aiodone(), kill the variable-sized array on stack, and use a
fixed size array which size should match any buf; if a bogus buf is passed
to this function, the kernel will KASSERT instead of potentially running out
of stack and having an undefined behaviour.

ok deraadt@

now that all partition size/offsets are potentially 64-bit, change the
type of all variables to daddr64_t. this includes the APIs for XXsize()
and XXdump(), all range checks inside bio drivers, internal variables
for disklabel handling, and even uvm's swap offsets. re-read numerous
times by otto, miod, krw, thib to look for errors

While splitting flags and pqflags might have been a good idea in theory
to separate locking, on most modern machines this is not enough
since operations on short types touch other short types that share the
same word in memory.

Merge pg_flags and pqflags again and now use atomic operations to change
the flags. Also bump wire_count to an int and pg_version might go
int as well, just for alignment.

tested by many, many. ok miod@

Mechanically rename the "flags" and "version" fields in struct vm_page
to "pg_flags" and "pg_version", so that they are a bit easier to work with.
Whoever uses generic names like this for a popular struct obviously doesn't
read much code.

Most architectures compile and there are no functionality changes.

deraadt@ ok ("if something fails to compile, we fix that by hand")

remove KERN_SUCCESS and use 0 instead.
eyeballed by miod@ and pedro@

fix uvmhist #2: args are always u_long so fix missing %d and %x and no %ll; no change for normal code

fix fmts for UVMHIST_LOG() entries making it more useful on 64bit archs; miod@ ok

- Make a UVM_OBJ_IS_DEVICE macro.
- Use it to skip device mappings while dumping core.
- Ignore EFAULT errors while dumping core since they can happen
even for valid mappings. Just skip that part of the core file and
let it get automagically zero-filled.

This fixes the broken X core dumps that people have been seeing and also
fixes some other potential problems that could prevent core dumps (mmaps
beyond EOF, etc.).

tedu@ ok

add a new field to vm_space and use it to track the number of anon
pages a process uses. this is now the userland "data size" value.
ok art deraadt tdeval. thanks testers.

sync of pmap_update() calls with NetBSD. pmap_update is defined away on
all architectures but arm, where it is needed.

ubchist is not a fully cooked kadaver and though use the other well formed pdhist one until ubc gaets back. art@ ok

Since memory deallocation can't fail, remove the error return from
uvm_unmap, uvm_deallocate and a few other functions.
Simplifies some code and reduces diff to the UBC branch.

Make sure that b_iodone handlers are called at splbio (and splassert(IPL_BIO) in all known callers, just to make sure).

splassert(IPL_BIO) in the b_iodone handlers.

Back out a few more uvm changes, especially wrt swap usage.
This unbreaks m68k m88k sparc and perhaps others, which eventually froze
when hitting swap.
Tested by various people on various platforms.
ok art@

UBC was a disaster. It worked very good when it worked, but on some
machines or some configurations or in some phase of the moon (we actually
don't know when or why) files disappeared. Since we've not been able to
track down the problem in two weeks intense debugging and we need -current
to be stable, back out everything to a state it had before UBC.

We apologise for the inconvenience.

branches: 1.28.2;
Yet another sync to NetBSD uvm.
Today we add a pmap argument to pmap_update() and allocate map entries for
kernel_map from kmem_map instead of using the static entries. This should
get rid of MAX_KMAPENT panics. Also some uvm_loan problems are fixed.

Call buf_cleanout, which handles wakeups

Correctly handle b_vp with bgetvp and brelvp in {get,put}pages.
Prevents panics caused by vnodes being recycled under our feet.

Sync in more uvm from NetBSD. Mostly just cosmetic stuff.
Contains also support for page coloring.

Sync in more uvm changes from NetBSD.
This time we're getting rid of KERN_* and VM_PAGER_* error codes and
use errnos instead.

Merge in the unified buffer cache code as found in NetBSD 2001/03/10. The
code is written mostly by Chuck Silvers <chuq@chuq.com>/<chs@netbsd.org>.

Tested for the past few weeks by many developers, should be in a pretty stable
state, but will require optimizations and additional cleanups.

Bring in more changes from NetBSD. Mostly pagedaemon improvements.

Merge in some parts of the ubc work that has been done in NetBSD that are not
UBC, but prerequsites for it.

- Create a daemon that processes async I/O (swap and paging in the future)
requests that need processing in process context and that were processed
in the pagedaemon before.
- Convert some ugly ifdef DIAGNOSTIC code to less intrusive KASSERTs.
- misc other cleanups.

Another sync of uvm to NetBSD. Just minor fiddling, no major changes.

Add an alignment argument to uvm_map that specifies an alignment hint
for the virtual address.

Move the last content from vm/ to uvm/
The only thing left in vm/ are just dumb wrappers.
vm/vm.h includes uvm/uvm_extern.h
vm/pmap.h includes uvm/uvm_pmap.h
vm/vm_page.h includes uvm/uvm_page.h

Minor sync to NetBSD.

occured->occurred

idea from deraadt@ via NetBSD
millert@ ok

merge vm/vm_kern.h into uvm/uvm_extern.h; art@ ok

Various random fixes from NetBSD.
Including support for zeroing pages in the idle loop (not enabled yet).

Add a new type voff_t (right now it's typedefed as off_t) used for offsets
into objects.

Gives the possibilty to mmap beyond the size of vaddr_t.

From NetBSD.

Add support for disabling swap devices (swapctl -d).
Improve error handling on I/O errors to swap.
From NetBSD

Change the pmap_enter interface to merge access_type and the wired boolean
and arbitrary flags into one argument.

One new flag is PMAP_CANFAIL that tells pmap_enter that it can fail if there
are not enough resources to satisfy the request. If this flag is not passed,
pmap_enter should panic as it should have done before this change (XXX - many
pmaps are still not doing that).

Only i386 and alpha implement CANFAIL for now.

Includes uvm updates from NetBSD.

Get rid of the PMAP_NEW option by making it mandatory for all archs.
The archs that didn't have a proper PMAP_NEW now have a dummy implementation
with wrappers around the old functions.

Change the paddr_t pmap_extract(struct pmap *, vaddr_t) interface to
boolean_t pmap_extract(struct pmap *, vaddr_t, paddr_t *).
Matches NetBSD. Tested by various people on various platforms.

Sync style, typo, and comments a little closer to NetBSD. art@ ok

Replace thread_wakeup() with wakeup(). It is defined in vm_extern.h as a
wrapper, so this removes a dependence on the old VM system. From NetBSD.
art@ ok

$OpenBSD$

Bring in some new UVM code from NetBSD (not current).

- Introduce a new type of map that are interrupt safe and never allow faults
in them. mb_map and kmem_map are made intrsafe.
- Add "access protection" to uvm_vslock (to be passed down to uvm_fault and
later to pmap_enter).
- madvise(2) now works.
- various cleanups.

branches: 1.4.4;
Change the pmap_enter api to pass down an argument that indicates
the access type that caused this mapping. This is to simplify pmaps
with mod/ref emulation (none for the moment) and in some cases speed
up pmap_is_{referenced,modified}.
At the same time, clean up some mappings that had too high protection.

XXX - the access type is incorrect in old vm, it's only used by uvm and MD code.
The actual use of this in pmap_enter implementations is not in this commit.

sync with NetBSD from 1999.05.24 (there is a reason for this date)
Mostly cleanups, but also a few improvements to pagedaemon for better
handling of low memory and/or low swap conditions.

add OpenBSD tags

Import of uvm from NetBSD. Some local changes, some code disabled

Use PMAP_PREFER_ALIGN() == 0 rather than !defined(PMAP_PREFER) to enable the
fast path in the pager code; this benefits most mips64 platforms.

ok kettenis@ mpi@

Bring back the direct map optimization when PMAP_PREFER isn't defined.

PMAP_PREFER is checked to prevent cache aliasing issues when the physical
page is mapped at different virtual addresses.

This speeds up file-based mmap faults.

ok kettenis@

Revert simplification of the aiodone daemon it breaks swap on arm64.

Found the hard way by mlarkin@ and deraadt@.

Revert the changes made in rev 1.82. It is important to use pmap_enter(9)
and pmap_remove(9) here since we're dealing with managed pages here. Found
out the hard way by deraadt@ on landisk where we're running into issues
with virtual cache aliases because multiple mappings exist for the
pages we're dealing with here. The pmap_enter(9) and pmap_remove(9)
functions handle conflicting cache aliases, whereas pmap_map_direct(9) and
pmap_kenter_pa(9) assume that the pages is exclusively mapped in the kernel
pmap.

ok deraadt@

Simplify the aiodone daemon which is only used for async writes.

- Remove unused support for asynchronous read, including error conditions

- Grab the proper lock for each page that has been written to swap. This
allows to enable an assertion in uvm_page_unbusy().

- Move the uvm_anon_release() call outside of uvm_page_unbusy() and
assert for the different anon cases.

ok beck@, kettenis@

Reduce allocations and possible failures in uvm_pagermapin/out().

. If a direct map exists use it to map single-page allocations
. Use pmap_kenter_pa() instead of pmap_enter() in all other cases.

This speeds up file-based mmap up to 75% when I/O are performed and it
also reduces possible allocations failtures in the page daemon making
it more stable in OOM situations.

ok kettenis@, beck@

Reserve a second segment for the page daemon.

This ensures uvm_swap_io() can succeeds even in OOM situations because two
uvm_pagermapin() allocations, requiring a segment each, are needed to bounce
or encrypt a swap cluster.

ok beck@, kettenis@

Move the guts of uvm_aio_aiodone() into a separate function.

This will help dealing with error cases.

No functionnal changes.

Do not clean PG_BUSY before calling uvm_anon_release().

Fix an assertion reported by gkoehler@.

ok kettenis@

Convert KVA allocation to km_alloc(9).

ok mpi@

Use a per-UVM object lock to serialize the lower part of the fault handler.

Like the per-amap lock the `vmobjlock' is principally used to serialized
access to objects in the fault handler to allow faults occurring on
different CPUs and different objects to be processed in parallel.

The fault handler now acquires the `vmobjlock' of a given UVM object as
soon as it finds one. For now a write-lock is always acquired even if
some operations could use a read-lock.

Every pager, corresponding to a different kind of UVM object, now expect
the UVM object to be locked and some operations, like *_get() return it
unlocked. This is enforced by assertions checking for rw_write_held().

The KERNEL_LOCK() is now pushed to the VFS boundary in the vnode pager.

To ensure the correct amap or object lock is held when modifying a page
many uvm_page* operations are now asserting for the "owner" lock.
However, fields of the "struct vm_page" are still being protected by the
global `pageqlock'. To prevent lock ordering issues with the new
`vmobjlock' and to reduce differences with NetBSD this lock is now taken
and released for each page instead of around the whole loop.

This commit does not remove the KERNEL_LOCK/UNLOCK() dance. Unlocking
will follow if there is no fallout.

Ported from NetBSD, tested by many, thanks!

ok kettenis@, kn@

Remove parenthesis around return value to reduce the diff with NetBSD.

No functional change.

ok mlarkin@

spelling

ok mpi@

(re)Introduce locking for amaps & anons.

A rwlock is attached to every amap and is shared with all its anon. The
same lock will be used by multiple amaps if they have anons in common.

This should be enough to get the upper part of the fault handler out of the
KERNEL_LOCK() which seems to bring up to 20% improvements in builds.

This is based/copied/adapted from the most recent work done in NetBSD which
is an evolution of the precendent simple_lock scheme.

Tested by many, thanks!

ok kettenis@, mvs@

Constify and use C99 initializer for "struct uvm_pagerops".

While here put some KERNEL_ASSERT_LOCKED() in the functions called from
the page fault handler. The removal of locking of `uobj' will need to be
revisited and these are good indicator that something is missing and that
many comments are lying.

ok kettenis

Convert infinite sleeps to {m,t}sleep_nsec(9).

ok visa@, jca@

remove lock.h from uvm_extern.h. another holdover from the simpletonlock
era. fix uvm including c files to include lock.h or atomic.h as necessary.
ok deraadt

Replace a plethora of historical protection options with just
PROT_NONE, PROT_READ, PROT_WRITE, and PROT_EXEC from mman.h.
PROT_MASK is introduced as the one true way of extracting those bits.
Remove UVM_ADV_* wrapper, using the standard names.
ok doug guenther kettenis

remove uneeded proc.h includes
ok mpi@ kspillner@

Chuck Cranor rescinded clauses in his license
on the 2nd of February 2011 in NetBSD.

http://marc.info/?l=netbsd-source-changes&m=129658899212732&w=2
http://marc.info/?l=netbsd-source-changes&m=129659095515558&w=2
http://marc.info/?l=netbsd-source-changes&m=129659157916514&w=2
http://marc.info/?l=netbsd-source-changes&m=129665962324372&w=2
http://marc.info/?l=netbsd-source-changes&m=129666033625342&w=2
http://marc.info/?l=netbsd-source-changes&m=129666052825545&w=2
http://marc.info/?l=netbsd-source-changes&m=129666922906480&w=2
http://marc.info/?l=netbsd-source-changes&m=129667725518082&w=2

stop using B_AGE, it was effectively retired some time ago.

Fix some potential integer overflows caused by converting a page number into
an offset/size/address by shifting by PAGE_SHIFT. Make uvm_objwrire/unwire
use voff_t instead of off_t. The former is the right type here even if it is
equivalent to the latter.

Inspired by a somewhat similar changes in Bitrig.

ok deraadt@, guenther@

compress code by turning four line comments into one line comments.
emphatic ok usual suspects, grudging ok miod

No need to cast constants or simple variables to (daddr_t). Use
(u_int64_t) instead of (daddr_t) when casting a variable in an
expression passed to DL_SETDSIZE().

Change a variable counting open files from daddr_t to int64_t.

ok deraadt@ with the tweak to fix that pesky expression.

final removal of daddr64_t. daddr_t has been 64 bit for a long enough
test period; i think 3 years ago the last bugs fell out.
ok otto beck others

remove lots of comments about locking per beck's request

remove simple_locks from uvm code. ok beck deraadt

Rip out and burn support for UVM_HIST.

The vm hackers don't use it, don't maintain it and have to look at it all the
time. About time this 800 lines of code hit /dev/null.

``never liked it'' tedu@. ariane@ was very happy when i told her i wrote
this diff.

for uvm_pager_dropcluster in the PG_RELEASED case we specifically unbusy the
page so that um_anfree will free it for us.

uvm_anfree does a pmap_page_protect(, VM_PROT_NONE) just before it frees the
page, so we don't need to do it here ourselves.

ok ariane@

move uvm_pageratop from uvm_pager.c local to a general uvm function
(uvm_atopg) and use it in uvm_km_doputpage to replace some handrolled
code. Shrinks the kernel a trivial amount.

ok beck@ and miod@ (who suggested i name it uvm_atopg not uvm_atop)

Don't sleep while holding the uvm_psel_lck mutex. Should fix "locking against
myself" panics that some people have seen over the last year-and-a-half.

Cherry picked from a more complex (and therefore scarier) diff from oga@.

ok tedu@, oga@