Copy stable/10@r272459 to releng/10.1 as part of
the 10.1-RELEASE process.

Approved by: re (implicit)
Sponsored by: The FreeBSD Foundation

Fix a leak of the wired pages when unwiring of the PROT_NONE-mapped
wired region. Rework the handling of unwire to do the it in batch,
both at pmap and object level.

All commits below are by alc.

MFC r268327:
Introduce pmap_unwire().

MFC r268591:
Implement pmap_unwire() for powerpc.

MFC r268776:
Implement pmap_unwire() for arm.

MFC r268806:
pmap_unwire(9) man page.

MFC r269134:
When unwiring a region of an address space, do not assume that the
underlying physical pages are mapped by the pmap. This fixes a leak
of the wired pages on the unwiring of the region mapped with no access
allowed.

MFC r269339:
In the implementation of the new function pmap_unwire(), the call to
MOEA64_PVO_TO_PTE() must be performed before any changes are made to the
PVO. Otherwise, MOEA64_PVO_TO_PTE() will panic.

MFC r269365:
Correct a long-standing problem in moea{,64}_pvo_enter() that was revealed
by the combination of r268591 and r269134: When we attempt to add the
wired attribute to an existing mapping, moea{,64}_pvo_enter() do nothing.
(They only set the wired attribute on newly created mappings.)

MFC r269433:
Handle wiring failures in vm_map_wire() with the new functions
pmap_unwire() and vm_object_unwire().
Retire vm_fault_{un,}wire(), since they are no longer used.

MFC r269438:
Rewrite a loop in vm_map_wire() so that gcc doesn't think that the variable
"rv" is uninitialized.

MFC r269485:
Retire pmap_change_wiring().

Reviewed by: alc

Merge the changes to pmap_enter(9) for sleep-less operation (requested
by flag). The ia64 pmap.c changes are direct commit, since ia64 is
removed on head.

MFC r269368 (by alc):
Retire PVO_EXECUTABLE.

MFC r269728:
Change pmap_enter(9) interface to take flags parameter and superpage
mapping size (currently unused).

MFC r269759 (by alc):
Update the text of a KASSERT() to reflect the changes in r269728.

MFC r269822 (by alc):
Change {_,}pmap_allocpte() so that they look for the flag
PMAP_ENTER_NOSLEEP instead of M_NOWAIT/M_WAITOK when deciding whether
to sleep on page table page allocation.

MFC r270151 (by alc):
Replace KASSERT that no PV list locks are held with a conditional
unlock.

Reviewed by: alc
Approved by: re (gjb)
Sponsored by: The FreeBSD Foundation

MFC r266826, r266827
Move some duplicated hook definitions from machine-dependent files to
kern_dtrace.c.

MFC 263036, 263059: delete advertising clause in licenses, renumber.

MFC r261423, r261424, r261516, r261513, r261562, r261563, r261564, r261565,
r261596, r261606

Add the imx sdhci controller.

Move Open Firmware device root on PowerPC, ARM, and MIPS systems to
a sub-node of nexus (ofwbus) rather than direct attach under nexus. This
fixes FDT on x86 and will make coexistence with ACPI on ARM systems easier.
SPARC is unchanged.

Add the missing ')' at end of sentence. Reword it to use a more common idiom.

Pass the kernel physical address to initarm through the boot param struct.

Make functions only used in vfp.c static, and remove vfp_enable.

Fix __syscall on armeb EABI. As it returns a 64-bit value it needs to
place 32-bit data in r1, not r0. 64-bit data is already packed correctly.

Use abp_physaddr for the physical address over KERNPHYSADDR. This helps us
remove the need to load the kernel at a fixed address.

Remove references to PHYSADDR where it's used only in debugging output.

Dynamically generate the page table. This will allow us to detect the
physical address we are loaded at to change the mapping.

MFC r257702, r257745, r257746, r257747, r257751, r257791, r257793,
r257794, r257795, r257992

Teach nexus(4) about Open Firmware (e.g. FDT) on ARM and MIPS, retiring
fdtbus in most cases.

Make OF_nextprop() work correctly for FDT by using the libfdt
fdt_next_property_offset() API.

Do not panic if pmap_mincore() is called.

An addendum: it is possible, though of questionable utility, for a node
to have no properties at all.
Add definition for the Atheros 8021 gigabit PHY.

Consolidate Apple firmware hacks and improve them by switching on the
presence of mac-io devices in the tree, which uniquely identifies Apple
hardware.

Allow OF_decode_addr() to also be able to map resources on big-endian
devices.

Make tsec work with the device tree present on the RB800.

Be more flexible about which compatible strings to accept. This brings up
the PCI Express bus on the RB800 using the firmware device tree.

Rename the "bare" platform "mpc85xx", which is what it actually is, and
add actual platform probing based on PVR.

MFC r257334, r257336, r257337, r257338, r257341, r257342, r257343, r257370,
r257368, r257416

Hints-only devices should return BUS_PROBE_NOWILDCARD from their probe
methods.

Merge r264984

Retire smp_active. It was racey and caused demonstrated problems with
the cpufreq code. Replace its use with smp_started. There's at least
one userland tool that still looks at the kern.smp.active sysctl, so
preserve it but point it to smp_started as well.

Obtained from: Netflix, Inc.

MFC r263289: Update NetBSD Foundation copyrights to 2-clause BSD

The NetBSD Foundation states "Third parties are encouraged to change the
license on any files which have a 4-clause license contributed to the
NetBSD Foundation to a 2-clause license."

This change removes clauses 3 and 4 from copyright / license blocks that
list The NetBSD Foundation as the only copyright holder.

Sponsored by: The FreeBSD Foundation

MFR r261294:

Remove an unneeded space in the BERI merge.

Sponsored by: DARPA, AFRL

MFC r257417: Remove references to an unused fasttrap probe hook

MFC r256939

MFP4:
Change 221767 by rwatson@rwatson_zenith_cl_cam_ac_uk on 2013/02/05 14:18:53

When printing out information on a TLB MOD exception for a user
process (e.g., an attempt to write to a read-only page), report
it as a "write" in the console message, rather than "unknown".

Change 221768 by rwatson@rwatson_zenith_cl_cam_ac_uk on 2013/02/05 14:28:00

Fix post-compile but pre-commit typo in last changeset.

Sponsored by: DARPA/AFRL

MFC r256937:

MFP4:
Change 231031 by brooks@brooks_zenith on 2013/07/11 16:22:08

Turn the unused and uncompilable MIPS_DISABLE_L1_CACHE define in
cache.c into an option and when set force I- and D-cache line
sizes to 0 (the latter part might be better as a tunable).

Fix some casts in an #if 0'd bit of code which attempts to
disable L1 cache ops when the cache is coherent.

Sponsored by: DARPA/AFRL

MFC r257228:
Add bus_dmamap_load_ma() function to load map with the array of
vm_pages.

MFC r256935

MFP4:
Change 228019 by bz@bz_zenith on 2013/04/23 13:55:30

Add kernel side support for large TLB on BERI/CHERI.
Modelled similar to NLM

Sponsored by: DAPRA/AFRL
Approved by: re (delphij)

Copy head (r256279) to stable/10 as part of the 10.0-RELEASE cycle.

Approved by: re (implicit)
Sponsored by: The FreeBSD Foundation

Add read/write bus barriers for MIPS.

Tested:

* AR9344 - mips74k SoC

Approved by: re@ (delphij)

Shuffle the includes around so they occur after opt_global.h is included;
this way the CPU ABI / core #define items are there.

Reviewed by: imp@
Approved by: re@ (gjb)

Add pagemask debugging output in "show tlb" in the debugger.

Approved by: re (marius)
Obtained from: bsdimp

- Create kern.ipc.sendfile namespace, and put the new "readhead" OID
there as "kern.ipc.sendfile.readahead".
- Push all nsfbuf related tunables into MD code. Don't move them
to new namespace in favor of POLA.

Reviewed by: scottl
Approved by: re (gjb)

The pmap function pmap_clear_reference() is no longer used. Remove it.

pmap_clear_reference() has had exactly one caller in the kernel for
several years, more precisely, since FreeBSD 8. Now, that call no
longer exists.

Approved by: re (kib)
Sponsored by: EMC / Isilon Storage Division

Use a better version of memcpy/bcopy for mips kernel.

Use a variant of mips libc memcpy for kernel. This implementation uses
64-bit operations when compiled for 64-bit, and is significantly faster
in that case.

Submitted by: Tanmay Jagdale <tanmayj@broadcom.com>

On those machines, where sf_bufs do not represent any real object, make
sf_buf_alloc()/sf_buf_free() inlines, to save two calls to an absolutely
empty functions.

Reviewed by: alc, kib, scottl
Sponsored by: Nginx, Inc.
Sponsored by: Netflix

pmap_protect() on MIPS does not need to acquire the pvh global lock.

Implement pmap_advise().

Significantly reduce the cost, i.e., run time, of calls to madvise(...,
MADV_DONTNEED) and madvise(..., MADV_FREE). Specifically, introduce a new
pmap function, pmap_advise(), that operates on a range of virtual addresses
within the specified pmap, allowing for a more efficient implementation of
MADV_DONTNEED and MADV_FREE. Previously, the implementation of
MADV_DONTNEED and MADV_FREE relied on per-page pmap operations, such as
pmap_clear_reference(). Intuitively, the problem with this implementation
is that the pmap-level locks are acquired and released and the page table
traversed repeatedly, once for each resident page in the range
that was specified to madvise(2). A more subtle flaw with the previous
implementation is that pmap_clear_reference() would clear the reference bit
on all mappings to the specified page, not just the mapping in the range
specified to madvise(2).

Since our malloc(3) makes heavy use of madvise(2), this change can have a
measureable impact. For example, the system time for completing a parallel
"buildworld" on a 6-core amd64 machine was reduced by about 1.5% to 2.0%.

Note: This change only contains pmap_advise() implementations for a subset
of our supported architectures. I will commit implementations for the
remaining architectures after further testing. For now, a stub function is
sufficient because of the advisory nature of pmap_advise().

Discussed with: jeff, jhb, kib
Tested by: pho (i386), marcel (ia64)
Sponsored by: EMC / Isilon Storage Division

Revert r254501. Instead, reuse the type stability of the struct pmap
which is the part of struct vmspace, allocated from UMA_ZONE_NOFREE
zone. Initialize the pmap lock in the vmspace zone init function, and
remove pmap lock initialization and destruction from pmap_pinit() and
pmap_release().

Suggested and reviewed by: alc (previous version)
Tested by: pho
Sponsored by: The FreeBSD Foundation

The soft and hard busy mechanism rely on the vm object lock to work.
Unify the 2 concept into a real, minimal, sxlock where the shared
acquisition represent the soft busy and the exclusive acquisition
represent the hard busy.
The old VPO_WANTED mechanism becames the hard-path for this new lock
and it becomes per-page rather than per-object.
The vm_object lock becames an interlock for this functionality:
it can be held in both read or write mode.
However, if the vm_object lock is held in read mode while acquiring
or releasing the busy state, the thread owner cannot make any
assumption on the busy state unless it is also busying it.

Also:
- Add a new flag to directly shared busy pages while vm_page_alloc
and vm_page_grab are being executed. This will be very helpful
once these functions happen under a read object lock.
- Move the swapping sleep into its own per-object flag

The KPI is heavilly changed this is why the version is bumped.
It is very likely that some VM ports users will need to change
their own code.

Sponsored by: EMC / Isilon storage division
Discussed with: alc
Reviewed by: jeff, kib
Tested by: gavin, bapt (older version)
Tested by: pho, scottl

Replace kernel virtual address space allocation with vmem. This provides
transparent layering and better fragmentation.

- Normalize functions that allocate memory to use kmem_*
- Those that allocate address space are named kva_*
- Those that operate on maps are named kmap_*
- Implement recursive allocation handling for kmem_arena in vmem.

Reviewed by: alc
Tested by: pho
Sponsored by: EMC / Isilon Storage Division

Include sys/systm.h after sys/param.h.

Suggested by: pluknet

Introduce new structure sfstat for collecting sendfile's statistics
and remove corresponding fields from struct mbstat. Use PCPU counters
and SFSTAT_INC() macro for update these statistics.

Discussed with: glebius

Make support for atomics on ARM complete.

Provide both __sync_*-style and __atomic_*-style functions that perform
the atomic operations on ARMv5 by using Restartable Atomic Sequences.

While there, clean up some pieces of code where it's sufficient to use
regular uint32_t to store register contents and don't need full reg_t's.
Also sync this back to the MIPS code.

- Add a BIT_FFS() macro and use it to replace cpusetffs_obj()

Discussed with: attilio
Sponsored by: EMC / Isilon Storage Division

Bring the stdatomic for MIPS code slightly more in sync with the ARM version.

Merge the 1 and 2 byte versions of the atomic functions into one.

After pushing in my fix for the 2 byte functions, I realized that the
functions for 1 and 2 byte operations had become identical. Reduce the
code size by merging the functions for 1 and 2 byte operations together.

While there, slightly improve variable naming and comments.

Actually make the 2-byte atomics work.

Even though I tested the 1-byte operations on arbitrarily aligned bytes,
it seems I did not do this for the 2-byte operations.

Create easy to read functions that are used to get/put bytes and
halfwords in words. To keep the compiler happy, explicitly read two
bytes into a union to obtain a 16-bit value.

Add proper __sync_*() intrinsics for MIPS.

To make <stdatomic.h> work on MIPS (and ARM) using GCC, we need to
provide implementations of the __sync_*() functions. I already added
these functions for 4 and 8 byte types to libcompiler-rt some time ago,
based on top of <machine/atomic.h>.

Unfortunately, <machine/atomic.h> only provides a subset of the features
needed to implement <stdatomic.h>. This means that in some cases we had
to do compare-and-exchange calls in loops, where a simple ll/sc would
suffice.

Also implement these functions for 1 and 2 byte types. MIPS only
provides ll/sc instructions for 4 and 8 byte types, but this is of
course no limitation. We can simply load 4 bytes and use some bitmask
tricks to modify only the bytes affected.

Discussed on: mips, arch
Tested with: QEMU

Don't assign the copyright to the FreeBSD foundation for the years
this file is in FreeBSD. There's formality to this that hasn't
happened and Juniper is perfectly fine with being the holder.

Discussed with: eadler, imp, jhb

Change the copyright notice to a standard BSD 2-clause license and assign
ownership to the FreeBSD foundation for the years this file has been in
the FreeBSD repository.

This file was originally created by Juniper as part of upgrading to FreeBSD
4.10 (which had no MIPS support) and held functions found on other machines
It grew actual functionality over time. The functionaliy was copied from
other architectures and ported to MIPS on a as-needed basis.

Approved by: Mark Baushke (Juniper IP)
Approved by: Megan Sugiyama (Juniper legal)
Pointed out by: jmallett@
Requested by: core (jhb@)

o Relax locking assertions for vm_page_find_least()
o Relax locking assertions for pmap_enter_object() and add them also
to architectures that currently don't have any
o Introduce VM_OBJECT_LOCK_DOWNGRADE() which is basically a downgrade
operation on the per-object rwlock
o Use all the mechanisms above to make vm_map_pmap_enter() to work
mostl of the times only with readlocks.

Sponsored by: EMC / Isilon storage division
Reviewed by: alc

Fix several typos

PR: kern/176054
Submitted by: Christoph Mallon <christoph.mallon@gmx.de>
MFC after: 3 days

MFp4 CH=227440:

Convert the structures to C99 style initialisation, which makes it
a lot easier to check that all of them are set and to generate a
derived template from them.

Sponsored by: DARPA, AFRL
MFC after: 2 weeks

Don't include asm.h in non-asm files.
Remove #define to get kludges that asm.h used to define
Move clever macros to access assembler instructions to trap.c
Remove __ASSEMBLER__ ifdefs in regdef.h: they aren't needed anymore.

Remove asm.h dependency in a hackish way.
Fixed comment.

Minor whitespace nit

Use the offsets from pcb.h rather than regnum.h to store the registers
in the pcb. setjmp/longjmp in the kernel also used these values, so
continue to use them although their use isn't technically the pcb
register array (matching is all that's important for setjmp/longjmp in
the kernel). Finally, eliminate the old register names from regnum.h.

This is a lexical change only. The non-debug .o files have the same md5.

Use the defines from pcb.h over the ones from regnum.h for this 'C'
code. In theory, the ones from regnum.h should be used only for
assembler code.

Fix N32/N64 ABIs to use proper registers after recent changes.

Pointy Hat to: imp

Update trapframe to be consistent with the changes made to regnum.h. This
should fix the booting problems people have been seeing.

MFP4 223084, 227821:

Partially implement generic_bs_*_8() for MIPS platforms.

This is known to work with TARGET_ARCH=mips64 with FreeBSD/BERI.
Assuming that other definitions in cpufunc.h are correct it will
work on non-o64 ABI systems except sibyte. On sibyte and o32 systems
generic_bs_*_8() will remain panic() implementations.

Sponsored by: DARPA, AFRL
Reviewed by: imp, jmallett (older versions)

Move MIPS_MAX_TLB_ENTRIES definition from cpuregs.h to tlb.c

Having MIPS_MAX_TLB_ENTRIES defined to 128 is misleading, since it used
to be 64 in older releases of MIPS architecture (where it could be read
from Config1) and can be much more than 128 for the newer processors.

For now, move the definition to the only file using it (mips/mips/tlb.c)
and define MIPS_MAX_TLB_ENTRIES depending on the MIPS cpu defined. Also
add few checks so that we do not write beyond the end of the tlb_state
array.

This fixes a kernel data corruption seen in Netlogic XLP, which was casued
by tlb_save() writing beyond the end of tlb_state array when breaking into
debugger.

Implement the concept of the unmapped VMIO buffers, i.e. buffers which
do not map the b_pages pages into buffer_map KVA. The use of the
unmapped buffers eliminate the need to perform TLB shootdown for
mapping on the buffer creation and reuse, greatly reducing the amount
of IPIs for shootdown on big-SMP machines and eliminating up to 25-30%
of the system time on i/o intensive workloads.

The unmapped buffer should be explicitely requested by the GB_UNMAPPED
flag by the consumer. For unmapped buffer, no KVA reservation is
performed at all. The consumer might request unmapped buffer which
does have a KVA reserve, to manually map it without recursing into
buffer cache and blocking, with the GB_KVAALLOC flag.

When the mapped buffer is requested and unmapped buffer already
exists, the cache performs an upgrade, possibly reusing the KVA
reservation.

Unmapped buffer is translated into unmapped bio in g_vfs_strategy().
Unmapped bio carry a pointer to the vm_page_t array, offset and length
instead of the data pointer. The provider which processes the bio
should explicitely specify a readiness to accept unmapped bio,
otherwise g_down geom thread performs the transient upgrade of the bio
request by mapping the pages into the new bio_transient_map KVA
submap.

The bio_transient_map submap claims up to 10% of the buffer map, and
the total buffer_map + bio_transient_map KVA usage stays the
same. Still, it could be manually tuned by kern.bio_transient_maxcnt
tunable, in the units of the transient mappings. Eventually, the
bio_transient_map could be removed after all geom classes and drivers
can accept unmapped i/o requests.

Unmapped support can be turned off by the vfs.unmapped_buf_allowed
tunable, disabling which makes the buffer (or cluster) creation
requests to ignore GB_UNMAPPED and GB_KVAALLOC flags. Unmapped
buffers are only enabled by default on the architectures where
pmap_copy_page() was implemented and tested.

In the rework, filesystem metadata is not the subject to maxbufspace
limit anymore. Since the metadata buffers are always mapped, the
buffers still have to fit into the buffer map, which provides a
reasonable (but practically unreachable) upper bound on it. The
non-metadata buffer allocations, both mapped and unmapped, is
accounted against maxbufspace, as before. Effectively, this means that
the maxbufspace is forced on mapped and unmapped buffers separately.
The pre-patch bufspace limiting code did not worked, because
buffer_map fragmentation does not allow the limit to be reached.

By Jeff Roberson request, the getnewbuf() function was split into
smaller single-purpose functions.

Sponsored by: The FreeBSD Foundation
Discussed with: jeff (previous version)
Tested by: pho, scottl (previous version), jhb, bf
MFC after: 2 weeks

Add pmap function pmap_copy_pages(), which copies the content of the
pages around, taking array of vm_page_t both for source and
destination. Starting offsets and total transfer size are specified.

The function implements optimal algorithm for copying using the
platform-specific optimizations. For instance, on the architectures
were the direct map is available, no transient mappings are created,
for i386 the per-cpu ephemeral page frame is used. The code was
typically borrowed from the pmap_copy_page() for the same
architecture.

Only i386/amd64, powerpc aim and arm/arm-v6 implementations were
tested at the time of commit. High-level code, not committed yet to
the tree, ensures that the use of the function is only allowed after
explicit enablement.

For sparc64, the existing code has known issues and a stab is added
instead, to allow the kernel linking.

Sponsored by: The FreeBSD Foundation
Tested by: pho (i386, amd64), scottl (amd64), ian (arm and arm-v6)
MFC after: 2 weeks

Switch the vm_object mutex to be a rwlock. This will enable in the
future further optimizations where the vm_object lock will be held
in read mode most of the time the page cache resident pool of pages
are accessed for reading purposes.

The change is mostly mechanical but few notes are reported:
* The KPI changes as follow:
- VM_OBJECT_LOCK() -> VM_OBJECT_WLOCK()
- VM_OBJECT_TRYLOCK() -> VM_OBJECT_TRYWLOCK()
- VM_OBJECT_UNLOCK() -> VM_OBJECT_WUNLOCK()
- VM_OBJECT_LOCK_ASSERT(MA_OWNED) -> VM_OBJECT_ASSERT_WLOCKED()
(in order to avoid visibility of implementation details)
- The read-mode operations are added:
VM_OBJECT_RLOCK(), VM_OBJECT_TRYRLOCK(), VM_OBJECT_RUNLOCK(),
VM_OBJECT_ASSERT_RLOCKED(), VM_OBJECT_ASSERT_LOCKED()
* The vm/vm_pager.h namespace pollution avoidance (forcing requiring
sys/mutex.h in consumers directly to cater its inlining functions
using VM_OBJECT_LOCK()) imposes that all the vm/vm_pager.h
consumers now must include also sys/rwlock.h.
* zfs requires a quite convoluted fix to include FreeBSD rwlocks into
the compat layer because the name clash between FreeBSD and solaris
versions must be avoided.
At this purpose zfs redefines the vm_object locking functions
directly, isolating the FreeBSD components in specific compat stubs.

The KPI results heavilly broken by this commit. Thirdy part ports must
be updated accordingly (I can think off-hand of VirtualBox, for example).

Sponsored by: EMC / Isilon storage division
Reviewed by: jeff
Reviewed by: pjd (ZFS specific review)
Discussed with: alc
Tested by: pho

MFcalloutng:
Switch eventtimers(9) from using struct bintime to sbintime_t.
Even before this not a single driver really supported full dynamic range of
struct bintime even in theory, not speaking about practical inexpediency.
This change legitimates the status quo and cleans up the code.

Reform the busdma API so that new types may be added without modifying
every architecture's busdma_machdep.c. It is done by unifying the
bus_dmamap_load_buffer() routines so that they may be called from MI
code. The MD busdma is then given a chance to do any final processing
in the complete() callback.

The cam changes unify the bus_dmamap_load* handling in cam drivers.

The arm and mips implementations are updated to track virtual
addresses for sync(). Previously this was done in a type specific
way. Now it is done in a generic way by recording the list of
virtuals in the map.

Submitted by: jeff (sponsored by EMC/Isilon)
Reviewed by: kan (previous version), scottl,
mjacob (isp(4), no objections for target mode changes)
Discussed with: ian (arm changes)
Tested by: marius (sparc64), mips (jmallet), isci(4) on x86 (jharris),
amd64 (Fabian Keil <freebsd-listen@fabiankeil.de>)

Merge Perforce changeset 219925 to head:

Provided a bus_space implementation for FDT, modelled on
bus_space_generic, but with a local version of the map address
routine that does a P->V translation, as is the case with NLM's
similar routine for XLP. It's not clear to me that this is the
right solution -- possibly this belongs in simplebus -- however,
it is sufficient to get the DE4 LED driver working.

Sponsored by: DARPA, AFRL

Merge Perforce chance 219924 to head:

In a sign of weakness, replicate the MIPS bus_space_generic.c to
produce a new FDT version, which will perform necessary address
space translation for bus_space -- the solution used in NLM's MIPS
FDT support, but possibly not quite the right thing. This is
inconsistent with regular I/O via the nexus and the generic
bus_space, which instead perform translation via pmap_mapdev()
when a resource is activated. However, it will work while I
attempt to identify what the right way to reconcile possible
approaches.

(Another approach might be to make simplebus use Nexus's activate
routine instead of a generic one?)

Sponsored by: DARPA, AFRL

Remove one wholly-unused and buggy routine and some nearby alternative symbols.

While here, also correct a comment that seems to imply that this file is
NetBSD's all-singing, all-dancing locore.S, rather than our conservative set of
assembly support routines.

Flip the semantic of M_NOWAIT to only require the allocation to not
sleep, and perform the page allocations with VM_ALLOC_SYSTEM
class. Previously, the allocation was also allowed to completely drain
the reserve of the free pages, being translated to VM_ALLOC_INTERRUPT
request class for vm_page_alloc() and similar functions.

Allow the caller of malloc* to request the 'deep drain' semantic by
providing M_USE_RESERVE flag, now translated to VM_ALLOC_INTERRUPT
class. Previously, it resulted in less aggressive VM_ALLOC_SYSTEM
allocation class.

Centralize the translation of the M_* malloc(9) flags in the single
inline function malloc2vm_flags().

Discussion started by: "Sears, Steven" <Steven.Sears@netapp.com>
Reviewed by: alc, mdf (previous version)
Tested by: pho (previous version)
MFC after: 2 weeks

The function pmap_alloc_direct_page() unconditionally zeroes the returned
page. Therefore, it is really inappropriate for use by the function
uma_small_alloc(). The effect of using it was that every page was zeroed
at least once and possibly twice if M_ZERO was passed as a "wait" flag.

Rework the known rwlock to benefit about staying on their own
cache line in order to avoid manual frobbing but using
struct rwlock_padalign.

Reviewed by: alc, jimharris

Drop this from 500 to 128, to save a little space on memory constrained
platforms.

Free the dma map -after- it's checked, not before. Or you'll be
potentially referencing already-freed memory.

Eliminate a redundant TLB invalidation from pmap_pv_reclaim().

Correct an error in pmap_pv_reclaim(). It can legitimately encounter
wired mappings. If it does, it should just skip them.

Optimize the TLB invalidations performed by pmap_protect(). Use the new
range-based invalidation function instead of performing per-page
invalidations. Don't bother invalidating mappings that don't have PTE_D
set, since they don't allow write access.

In general pmap implementations do not set the wired attribute on
the temporary mappings that are used to implement operations like
pmap_zero_page(). There is no reason for the MIPS pmap to deviate
from that practice.

Correct two pessimizations in pmap_extract_and_hold(). Test the PTE for
having PTE_RO set instead of PTE_D. This avoids some unnecessary failures
by pmap_extract_and_hold() that will have to be handled by a call to
vm_fault_hold(). Testing the PTE for both being non-zero and having PTE_V
set is redundant. The latter suffices.

Eliminate a stale and a duplicated comment.

Reimplement pmap_qremove() using the new TLB invalidation function for
efficiently invalidating address ranges.

Introduce a new TLB invalidation function for efficiently invalidating
address ranges, and use this function in pmap_remove().

Tested by: jchandra

Eliminate a stale comment. It describes another use case for the pmap in
Mach that doesn't exist in FreeBSD.

Prefer __containerof() above member2struct().

The first does proper checking of the argument types, while the latter
does not.

Simplify pmap_unmapdev(). Since kmem_free() eventually calls pmap_remove(),
pmap_unmapdev()'s own direct efforts to destroy the page table entries are
redundant, so eliminate them.

Don't set PTE_W on the page table entry in pmap_kenter{,_attr}() on MIPS.
Setting PTE_W on MIPS is inconsistent with the implementation of this
function on other architectures. Moreover, PTE_W should not be set, unless
the pmap's wired mapping count is incremented, which pmap_kenter{,_attr}()
doesn't do.

MFC after: 10 days

userret() already checks for td_locks when INVARIANTS is enabled, so
there is no need to check if Giant is acquired after it.

Reviewed by: kib
MFC after: 1 week

pmap_remove:

Avoid re-walking the page table from the root for every PTE examined.

Tidy up some of pmap_remove()'s helper functions.

pmap_enter:

Set PV_TABLE_REF whenever the physical page being mapped is managed, not
just when it is writeable.

Only call pmap_update_page() when the old mapping was valid. If there was
no prior valid mapping, then pmap_update_page() serves no useful purpose.
However, it will interrupt every processor that has the pmap active.

pmap_enter_quick_locked:

Always set PTE_RO.

pmap_emulate_modified:

Don't set PV_TABLE_REF.

Eliminate a nonsensical comment.

Eliminate unnecessary NULL checks.

Dynamically allocate the S/G lists passed to callback routines rather than
allocating them on the stack of various bus_dmamap_load*() functions. The
S/G lists are stored in the DMA tags. This matches the implementation on
all other platforms.

Discussed with: scottl, gibbs
Tested by: stas (arm@)

Rename {_,}pmap_unwire_pte_hold() to {_,}pmap_unwire_ptp() and update the
comment describing them. Both the function names and the comment had grown
stale. Quite some time has passed since these pmap implementations last
used the page's hold count to track the number of valid mapping within a
page table page. Also, returning TRUE from pmap_unwire_ptp() rather than
_pmap_unwire_ptp() eliminates a few instructions from callers like
pmap_enter_quick_locked() where pmap_unwire_ptp()'s return value is used
directly by a conditional statement.

Calculate the new PTE value in pmap_enter() before acquiring any locks.

Move an assertion to the beginning of pmap_enter().

Introduce a new software PTE flag that indicates whether the mapping is
for a managed page.

Tested by: jchandra

Retire PV_TABLE_MOD. When we destroy or write protect a dirty mapping,
we call vm_page_dirty(). Maintaining the PV_TABLE_MOD flag, in addition,
serves no useful purpose.

Add preliminary support for the SRI International / University of Cambridge
Bluespec Extensible RISC Implementation (BERI) processor. BERI is a 64-bit
MIPS ISA soft CPU core that can be synthesised to Altera and Xilinx FPGAs,
and is being used for CPU and OS research at several institutions.

Sponsored by: DARPA, AFRL

Provide basic glue to allow syscons to be used on MIPS, modelled
on PowerPC support. This was clearly not something syscons was
designed to do (very specific assumptions about the nature of VGA
consoles on PCs), but fortunately others have long since blazed
the way on making it work regardless of that.

Sponsored by: DARPA, AFRL

On MIPS, when printing page fault information for an unexpected exception
type, explicitly print out "unknown" rather than the empty string, and
include the exception type number for ease of debugging.

Sponsored by: DARPA, AFRL

Eliminate another vestige of page coloring.

Fix two problems with pmap_clear_modify().

First, pmap_clear_modify() is write protecting all mappings to the specified
page, not just clearing the modified bit. Specifically, it sets PTE_RO on
the PTE, which is wrong. Moreover, it is calling vm_page_dirty(), which is
not the expected behavior for pmap_clear_modify(). Generally speaking, the
machine-independent VM layer masks these mistakes. For example, setting
PTE_RO will result in additional soft faults, but not a catastrophe.

Second, pmap_clear_modify() may not clear the modified bits because it only
iterates over the PV list when the page has the PV_TABLE_MOD flag set and
elsewhere the pmap clears the PV_TABLE_MOD flag anytime a modified mapping
is write protected or destroyed. However, the page may still have other
mappings with the modified bit set.

Eliminate a stale comment.

Eliminate an unused parameter from init_pte_prot().

Eliminate stray whitespace within init_pte_prot().

Eliminate a gratuitous variable initialization from pmap_enter().

Replace all uses of the vm page queues lock by a r/w lock that is private
to this pmap.

Tidy up the #include's.

Remove the (now) unused #define PMAP_SHPGPERPROC. (This should have
been removed in r239236.)

Tested by: jchandra

Port the new PV entry allocator from amd64/i386. This allocator has two
advantages. First, PV entries are roughly half the size. Second, this
allocator doesn't access the paging queues, and thus it will allow for the
removal of the page queues lock from this pmap.

Fix a rather serious bug in pmap_remove_write(). After removing write
access from the specified page's first mapping, pmap_remove_write() then
used the wrong "next" pointer. Consequently, the page's second, third,
etc. mappings were not write protected.

Tested by: jchandra

Merge r134393 from amd64/i386:
The machine-independent parts of the virtual memory system always pass a
valid pmap to the pmap functions that require one. Remove the checks for
NULL. (These checks have their origins in the Mach pmap.c that was
integrated into BSD. None of the new code written specifically for
FreeBSD included them.)

Merge r132141 and r111272 from amd64/i386:
Reduce the size of a PV entry by eliminating pv_ptem. There is no need
to store a pointer to the page table page in the PV entry because it is
easily computed during the walk down the page table.

Eliminate the ptphint from the pmap. Long, long ago, page table pages
belonged to a vm object, and we would look up page table pages based
upon their offset within this vm object. In those days, this hint may
have had tangible benefits.

Tested by: jchandra

Fix obvious problem with emulate_fp sysctl.

Submitted by: Paul Ambrose <ambrosehua@gmail.com>

Merge FreeBSD/beri Perforce change @211945 to head:

Modify MIPS page table entry (PTE) initialisation so that cachability
bits are set only once, using is_cacheable_mem() to determine what
caching properties are required, rather than also unconditionally
setting PTE_C_CACHE in init_pte_prot(). As PTE_C_CACHE |
PTE_C_UNCACHED == PTE_C_CACHE, this meant that all userspace memory
mappings of device memory (incorrectly) used caching TLB entries.

This is arguably not quite what we want, even though it is (more)
consistent with the MIPS pmap design: PTE caching properties should
be derived from machine-independent page table attributes, but this
is a substantially more complex change as the MIPS pmap doesn't yet
know about page attributes, causing it to ignore requests by device
drivers that want uncached userspace memory mappings as they
describe memory-mapped FIFOs or shared memory with a device not
participating in the cache coherence scheme.

This fixes cacheability issues (specifically, undesired and
unrequested caching) seen in userspace memory mappings of Avalon SoC
bus device memory on BERI MIPS.

Discussed with: jmallett, alc
Sponsored by: DARPA, AFRL
MFC after: 3 days

Move what remains of vm/vm_contig.c into vm/vm_pageout.c, where similar
code resides. Rename vm_contig_grow_cache() to vm_pageout_grow_cache().

Reviewed by: kib

Handle case when result of pmap_pte is NULL. This issue was uncovered
by r237367

Clean up various aspects of the MIPS generic busspace implementation:

1) Always implement missing bus space methods using a panic() stub rather
than a NULL pointer. This appeared not to trip up any existing device
drivers, but due to the nature of the devices I'm supporting locally,
I'm making use of some of the more obscure busspace methods, and
panic() is a preferred failure mode. For example, do this for the
setregion methods.

2) Hook up several existing busspace method implementations that were
provided in the file, but not actually present in the methods
structure. Especially, single-byte bus I/O routines. This should
allow bugs to be fixed in the Atheros 802.11 driver.

There are still some remaining unimplemented methods that would be
desirable to implement -- especially, 64-bit I/O calls that would
observably accelerate device performance on FPGA-based soft CPU cores
that are typically clocked an order of magnitude slower than
conventional hard core CPUs, but that remains for another day.

MFC after: 3 weeks
Discussed with: jmallett, scottl
Sponsored by: DARPA, AFRL

Allow for MIPS devices to have a specific probe/attach order.

The default priority is now '1000' rather than '0'. This may cause some
unforseen regressions.

Submitted by: Stefan Bethke <stb@lassitu.de>
Reviewed by: imp

Reinstate the XTLB handler for CPU_NLM and CPU_RMI

These platforms set the KX bit even when booted in 32 bit mode. So
the XLTB handler is needed even when __mips_n64 is not defined.

Disable FP instruction emulation by default on !o32 because of ABI concerns.

Note that in practice this isn't needed because we get a coprocessor unusable
exception first, but that's actually something like a bug.

Add DTrace-related part to machine-dependent code:

- DTrace trap handler
- invop-related variables (unused on MIPS but still referenced from dtrace)

Move PMC hook invocation to cpu_intr. The idea is the same as with ast()
call but there is no reason to implement it in assembler.

Fix return type of mips pmap_kextract()

The return type should be vm_paddr_t, not vm_offset_t.

Clean-up fake preload data generator:
- Use macros to push scalar values
- Fix type mismatch for module size

Fill out fake preload structure to let userland tools like pmc(3) know
about kernel module base address and actual size

o) Use ABI, not ISA_* options, to determine whether to compile bits if libkern
required for the ABI the kernel is being built for.
XXX This is implemented in a kind-of nasty way that involves including source
files, but it's still an improvement.
o) Retire ISA_* options since they're unused and were always wrong.

Remove more unused stuff, primarily a set of (unused, thankfully) PIO
functions.

Adjust nearby style of one assembly function END().

Remove more unused code and declarations, and add dire warnings to the 64-bit
atomic ops used by 32-bit kernels.

o) Remove some CPU_CNMIPS-related magical thinking about the status register's
contents for user programs.
o) Conditionalize the installation of an XTLB handler on ABI, not CPU family.

Get rid of mainbus.c. The version in nexus.c is being used and is
perfectly-sufficient and equally-crufty.

Don't truncate physical addresses to 32-bits.

Reduce diffs with freebsd32_sysarch.

o) Bump INTRCOUNT_COUNT to 256, since Octeon already has >128.
XXX It would be good to use a better way to size intrcnt.
o) Fix literal 4s that are supposed to be sizeof (u_long).
XXX Why the * 2 here? Is this an artifact of a different system that this
code came from? We seem to allocate twice as much space for intrcnt
as we admit to in sintrcnt.

Get rid of duplicated versions of the KSU bits.

Remove unused file.

At the risk of reducing source compatibility with old NetBSD and Sprite:
o) Get rid of some unused macros related to features we don't intend to
provide.
o) Get rid of macro definitions for MIPS-I CPUs. We are not likely to
support anything that predartes MIPS-III.
o) Respell MIPS3_* macros as MIPS_*, which is how most of them were being
used already.
o) Eliminate a duplicate and mostly-unused set of exception vector macros.

There's still considerable duplication and lots more obsolete in our headers,
but this reduces one of the larger files to a size where one could reckon
about the correctness of its contents with a mere few hours of contemplation.

There is, of course, a question of whether we need definitions for fields,
registers and configurations that we are unlikely to ever use or implement,
even if they're not obsolete since 1991. FreeBSD is not a processor
reference manual, and things that aren't used may be wrong, or may be
duplicated because nobody could possibly actually know whether they're
already defined.

Garbage collect some unused symbols.

Make the native sigreturn just wrap set_mcontext, much as freebsd32_sigreturn
does.

Store TLS base in the sigframe just as is done in freebsd32_sendsig. Because
the native sigreturn doesn't use set_mcontext like the COMPAT_FREEBSD32 version
does, this wouldn't actually result in overwriting the TLS base. Probably it
makes sense to restructure the native sigreturn to use set_mcontext for
consistency, and to allow sigreturn to change the TLS base.

Fix two and a half oversights in COMPAT_FREEBSD32 related to contexts and
TLS:
o) The mc_tls field used to store the TLS base when doing context gets and
restores was left a pointer and not converted to a 32-bit integer. This
had the bug of not correctly capturing the TLS value desired by the user,
and the extra nastiness of making the structure the wrong size.
o) The mc_tls field was not being saved by sendsig. As a result, the TLS base
would always be set to NULL when restoring from a signal handler.

Thanks to gonzo for helping track down a bunch of other TLS bugs that came out
of tracking these down.

When emulating rdhwr for TLS, use the 32-bit offset under COMPAT_FREEBSD32.

Prepare for large TLS redo. Save pointer to the beginning of TLS area,
and offset it only if requested by RDHWR handler. Otherwise things
get overly complicated - we need to track whether address passsed in
request for setting td_md.md_tls is already offseted or not.

In the trap messages that aid the primitive debugging environment of MIPS,
include the tid as well, so it's easier to tell which thread of a process
with multiple is responsible for a crash.

Fix tls base computation with COMPAT_FREEBSD32 on n64 kernels. The previous
version was missing an else and would always use the n64 TP_OFFSET. Eliminate
some duplication of logic here.

It may be worth getting rid of some of the ifdefs and introducing gratuitous
SV_ILP32 runtime checks on n64 kernels without COMPAT_FREEBSD32 and on o32
kernels, similarly to how PowerPC works.

Unbreak n64 build without COMPAT_FREEBSD32 by fixing mismatched preprocessor
conditionals.

o) Add COMPAT_FREEBSD32 support for MIPS kernels using the n64 ABI with userlands
using the o32 ABI. This mostly follows nwhitehorn's lead in implementing
COMPAT_FREEBSD32 on powerpc64.
o) Add a new type to the freebsd32 compat layer, time32_t, which is time_t in the
32-bit ABI being used. Since the MIPS port is relatively-new, even the 32-bit
ABIs use a 64-bit time_t.
o) Because time{spec,val}32 has the same size and layout as time{spec,val} on MIPS
with 32-bit compatibility, then, disable some code which assumes otherwise
wrongly when built for MIPS. A more general macro to check in this case would
seem like a good idea eventually. If someone adds support for using n32
userland with n64 kernels on MIPS, then they will have to add a variety of
flags related to each piece of the ABI that can vary. That's probably the
right time to generalize further.
o) Add MIPS to the list of architectures which use PAD64_REQUIRED in the
freebsd32 compat code. Probably this should be generalized at some point.

Reviewed by: gonzo

When creating a handle for a subregion, be sure to actually math out the new
handle address, where we're using handles as raw addresses.

This fixes devices with subregions on Octeon PCI specifically, and likely also on
MIPS more generally, where there isn't another bus_space in use that was doing the
math already.

- Change contigmalloc() to use the vm_paddr_t type instead of an unsigned
long for specifying a boundary constraint.
- Change bus_dma tags to use bus_addr_t instead of bus_size_t for boundary
constraints.

These allow boundary constraints to be fully expressed for cases where
sizeof(bus_addr_t) != sizeof(bus_size_t). Specifically, it allows a
driver to properly specify a 4GB boundary in a PAE kernel.

Note that this cannot be safely MFC'd without a lot of compat shims due
to KBI changes, so I do not intend to merge it.

Reviewed by: scottl

When initialising the CP0 status register during boot on 64-bit MIPS,
set all three of the kernel, supervisor, and user-mode 64-bit mode
flags. While FreeBSD does not currently use the supervisor ring (and
hence this is effectively a NOP on most systems), doing this avoids
triggering an exception on 64-bit MIPS CPUs that don't support 32-bit
compatibility mode, and therefore don't allow clearing the SX bit.

Reviewed by: gonzo
MFC after: 3 days
Sponsored by: DARPA, SRI International

- Reverse logic so base tls is fixed up with correct number

Fix-up value passed by thr_new syscall to make it compatible
with MIPS_TLS_GET/MIPS_TLS_SET sysarch API.

Fix n32 build breakage

- Emulate RDHWR instruction for TLS support

Reading register $29 with RDHWR is becoming the de-facto standard to
implement TLS. According to linux-mips wiki, MIPS Technologies has
reserved hardware register $29 for ABI use. Furthermore current GCC
makes the following assumptions:
- RDHWR is natively available or otherwise emulated by the kernel
- Register $29 holds the TLS pointer

Submitted by: Robert Millan <rmh@debian.org>

Fix backtrace for MIPS64:
- Properly print 64-bit addresses
- Get whole 64 bits of address using kdbpeekd
- Make check for kernel address compatible with MIPS64

- Fix .rela case of R_MIPS_26 relocation. Addednds save diferently for
.rel and .rela sections. It's shifted right two bits for former
but saved as-is for latter.

Fix relocations for MIPS64:
- Use Elf32_Addr as default, the only field that is
64 bitw wide is R_MIPS_64
- Add R_MIPS_HIGHER and R_MIPS_HGHEST handlers
- Handle R_MIPS_HI16 and R_MIPS_LO16 for both .rel and
.rela sections

Fix DDB x/i addr[,count] command for count > 1 case

- Add better COP2 (crypto coprocessor) context handler for Octeon. Keep
COP2 disabled and lazily allocate COP2 context structure in exception
handler. Keep kernel and userland contexts separated.

Use curthread rather than PCPU_GET(curthread). 'curthread' uses
special-case optimizations on several platforms and is preferred.

Reported by: dim (indirectly)
MFC after: 2 weeks

- Fix backtrace for MIPS64 platform

Fixup cache flush definitions for XLP

mco_icache_sync_range was earlier set to mipsNN_icache_sync_range_index_32
which is not necessary, revert this.
Also, the data cache is coherent so write back is not really needed. This
change is experimental.

Do pmap update only on active CPUs.

The pmap update_page/invalidate_page/invalidate_all operations has to be
done only on active cpus. In the simplest case, if the process is not
active on any other CPUs, we can just do the operation on the current CPU.

This change replaces the call to smp_rendezvous() for these operations with
smp_rendezvous_cpus() in case there more than one active CPU, or with a direct
function call if there is just one active CPU.

This change give significant performance increase in fork/exec benchmarks
on XLR/XLS/XLP with 32 cpus.

Reviewed by: alc

Mark all SYSCTL_NODEs static that have no corresponding SYSCTL_DECLs.

The SYSCTL_NODE macro defines a list that stores all child-elements of
that node. If there's no SYSCTL_DECL macro anywhere else, there's no
reason why it shouldn't be static.

Add support for VM_ALLOC_WIRED and VM_ALLOC_ZERO to vm_page_alloc_freelist()
and use these new options in the mips pmap.

Wake up the page daemon in vm_page_alloc_freelist() if the number of free
and cached pages becomes too low.

Tidy up vm_page_alloc_init(). In particular, add a comment about an
important restriction on its use.

Tested by: jchandra@

Fix wakeup latency when sleeping with 'wait'

If we handle an interrupt just before the 'wait' and the interrupt
schedules some work, we need to skip the 'wait' call. The simple solution
of calling sched_runnable() with interrupts disabled immediately before
wait still leaves a window after the call and before 'wait' in which
the same issue can occur.

The solution implemented is to check the EPC in the interrupt handler, and
if it is in a region before the 'wait' call, to fix up the EPC to skip the
wait call.

Reported/analysed by: adrian
Fix suggested by: kib

Reviewed by: jmallett, imp

Fix for crash on MIPS /dev/mem read.

Reading /dev/mem in 64 bit kernel crashes. This is because the page
used to call uiomove_fromphys() from memrw() does not have md.pv_list
initialized correctly.

The fix is to call pmap_page_init() on the page to initialize it.

Convert MIPS to the syscallenter/syscallret system call sequence handlers.
This was the last architecture used custom syscall entry sequence.

Reviewed, debugged, tested and approved by: jchandra
MFC after: 1 month

Disable using wait in cpu_idle() until a better solution to timer and
interrupt handling can be implemented.

In order to maximize the re-usability of kernel code in user space this
patch modifies makesyscalls.sh to prefix all of the non-compatibility
calls (e.g. not linux_, freebsd32_) with sys_ and updates the kernel
entry points and all places in the code that use them. It also
fixes an additional name space collision between the kernel function
psignal and the libc function of the same name by renaming the kernel
psignal kern_psignal(). By introducing this change now we will ease future
MFCs that change syscalls.

Reviewed by: rwatson
Approved by: re (bz)

Split the vm_page flags PG_WRITEABLE and PG_REFERENCED into atomic
flags field. Updates to the atomic flags are performed using the atomic
ops on the containing word, do not require any vm lock to be held, and
are non-blocking. The vm_page_aflag_set(9) and vm_page_aflag_clear(9)
functions are provided to modify afalgs.

Document the changes to flags field to only require the page lock.

Introduce vm_page_reference(9) function to provide a stable KPI and
KBI for filesystems like tmpfs and zfs which need to mark a page as
referenced.

Reviewed by: alc, attilio
Tested by: marius, flo (sparc64); andreast (powerpc, powerpc64)
Approved by: re (bz)

Fix a typo in cb_dumpdata() about wrongly calling wdog_kern_pat().

Submitted by: Andrew Boyer <aboyer at averesystems dot com>
Approved by: re (kib)

- Move the PG_UNMANAGED flag from m->flags to m->oflags, renaming the flag
to VPO_UNMANAGED (and also making the flag protected by the vm object
lock, instead of vm page queue lock).
- Mark the fake pages with both PG_FICTITIOUS (as it is now) and
VPO_UNMANAGED. As a consequence, pmap code now can use use just
VPO_UNMANAGED to decide whether the page is unmanaged.

Reviewed by: alc
Tested by: pho (x86, previous version), marius (sparc64),
marcel (arm, ia64, powerpc), ray (mips)
Sponsored by: The FreeBSD Foundation
Approved by: re (bz)

Fix for arm and mips case the size of storage for sintrcnt/sintrnames.
It seems that "info as" is not much precise on what expect by pseudo-op
.word, by the way.

No MFC is previewed for this patch.

Tested by: andreast, pluknet
Approved by: re (kib)

On 64 bit architectures size_t is 8 bytes, thus it should use an 8 bytes
storage.
Fix the sintrcnt/sintrnames specification.

No MFC is previewed for this patch.

Reported, reviewed and tested by: marcel
Approved by: re (kib)

- Remove the eintrcnt/eintrnames usage and introduce the concept of
sintrcnt/sintrnames which are symbols containing the size of the 2
tables.
- For amd64/i386 remove the storage of intr* stuff from assembly files.
This area can be widely improved by applying the same to other
architectures and likely finding an unified approach among them and
move the whole code to be MI. More work in this area is expected to
happen fairly soon.

No MFC is previewed for this patch.

Tested by: pluknet
Reviewed by: jhb
Approved by: re (kib)

Fix compilation of mips kernels adding apropriate #ifdefs after r224115.

Pointed out by: Luiz Otavio O Souza (lists.br gmail.com)

MIPS changes for Netlogic XLP support.

This patch adds support for the Netlogic XLP mips64 processors in
the common MIPS code. The changes are :

- Add CPU_NLM processor type
- Add cases for CPU_NLM, mostly were CPU_RMI is used.
- Update cache flush changes for CPU_NLM
- Add kernel build configuration files for xLP.

In collaboration with: Prabhath Raman <prabhathpr at netlogicmicro com>

Approved by: bz(re), jmallett, imp(mips)

Support compiling MIPS elf trampoline with a different ABI.

Allow changing the trampoline ABI with makeoptions, this will allow
us to have a trampoline with a different ABI from the kernel.
Useful in cases where we have to boot a 64 bit kernel from a
bootloader which supports only 32 bit or vice versa.

Approved by: bz(re), jmallett, imp

With retirement of cpumask_t and usage of cpuset_t for representing a
mask of CPUs, pc_other_cpus and pc_cpumask become highly inefficient.

Remove them and replace their usage with custom pc_cpuid magic (as,
atm, pc_cpumask can be easilly represented by (1 << pc_cpuid) and
pc_other_cpus by (all_cpus & ~(1 << pc_cpuid))).

This change is not targeted for MFC because of struct pcpu members
removal and dependency by cpumask_t retirement.

MD review by: marcel, marius, alc
Tested by: pluknet
MD testing by: marcel, marius, gonzo, andreast

When iterating over a paging queue, explicitly check for PG_MARKER, instead
of relying on zeroed memory being interpreted as an empty PV list.

Reviewed by: kib

Add a new option, OBJPR_NOTMAPPED, to vm_object_page_remove(). Passing this
option to vm_object_page_remove() asserts that the specified range of pages
is not mapped, or more precisely that none of these pages have any managed
mappings. Thus, vm_object_page_remove() need not call pmap_remove_all() on
the pages.

This change not only saves time by eliminating pointless calls to
pmap_remove_all(), but it also eliminates an inconsistency in the use of
pmap_remove_all() versus related functions, like pmap_remove_write(). It
eliminates harmless but pointless calls to pmap_remove_all() that were being
performed on PG_UNMANAGED pages.

Update all of the existing assertions on pmap_remove_all() to reflect this
change.

Reviewed by: kib

Remove duplicate header includes

etire the cpumask_t type and replace it with cpuset_t usage.

This is intended to fix the bug where cpu mask objects are
capped to 32. MAXCPU, then, can now arbitrarely bumped to whatever
value. Anyway, as long as several structures in the kernel are
statically allocated and sized as MAXCPU, it is suggested to keep it
as low as possible for the time being.

Technical notes on this commit itself:
- More functions to handle with cpuset_t objects are introduced.
The most notable are cpusetobj_ffs() (which calculates a ffs(3)
for a cpuset_t object), cpusetobj_strprint() (which prepares a string
representing a cpuset_t object) and cpusetobj_strscan() (which
creates a valid cpuset_t starting from a string representation).
- pc_cpumask and pc_other_cpus are target to be removed soon.
With the moving from cpumask_t to cpuset_t they are now inefficient
and not really useful. Anyway, for the time being, please note that
access to pcpu datas is protected by sched_pin() in order to avoid
migrating the CPU while reading more than one (possible) word
- Please note that size of cpuset_t objects may differ between kernel
and userland. While this is not directly related to the patch itself,
it is good to understand that concept and possibly use the patch
as a reference on how to deal with cpuset_t objects in userland, when
accessing kernland members.
- KTR_CPUMASK is changed and now is represented through a string, to be
set as the example reported in NOTES.

Please additively note that no MAXCPU is bumped in this patch, but
private testing has been done until to MAXCPU=128 on a real 8x8x2(htt)
machine (amd64).

Please note that the FreeBSD version is not yet bumped because of
the upcoming pcpu changes. However, note that this patch is not
targeted for MFC.

People to thank for the time spent on this patch:
- sbruno, pluknet and Nicholas Esborn (nick AT desert DOT net) tested
several revision of the patches and really helped in improving
stability of this work.
- marius fixed several bugs in the sparc64 implementation and reviewed
patches related to ktr.
- jeff and jhb discussed the basic approach followed.
- kib and marcel made targeted review on some specific part of the
patch.
- marius, art, nwhitehorn and andreast reviewed MD specific part of
the patch.
- marius, andreast, gonzo, nwhitehorn and jceel tested MD specific
implementations of the patch.
- Other people have made contributions on other patches that have been
already committed and have been listed separately.

Companies that should be mentioned for having participated at several
degrees:
- Yahoo! for having offered the machines used for testing on big
count of CPUs.
- The FreeBSD Foundation for having sponsored my devsummit attendance,
which has been instrumental.
- Sandvine for having offered offices and infrastructure during
development.

(I really hope I didn't forget anyone, if it happened I apologize in
advance).

On multi-core, multi-threaded PPC systems, it is important that the threads
be brought up in the order they are enumerated in the device tree (in
particular, that thread 0 on each core be brought up first). The SLIST
through which we loop to start the CPUs has all of its entries added with
SLIST_INSERT_HEAD(), which means it is in reverse order of enumeration
and so AP startup would always fail in such situations (causing a machine
check or RTAS failure). Fix this by changing the SLIST into an STAILQ,
and inserting new CPUs at the end.

Reviewed by: jhb

Change rman_manage_region() to actually honor the rm_start and rm_end
constraints on the rman and reject attempts to manage a region that is out
of range.
- Fix various places that set rm_end incorrectly (to ~0 or ~0u instead of
~0ul).
- To preserve existing behavior, change rman_init() to set rm_start and
rm_end to allow managing the full range (0 to ~0ul) if they are not set by
the caller when rman_init() is called.

Add the watchdogs patting during the (shutdown time) disk syncing and
disk dumping.
With the option SW_WATCHDOG on, these operations are doomed to let
watchdog fire, fi they take too long.

I implemented the stubs this way because I really want wdog_kern_*
KPI to not be dependant by SW_WATCHDOG being on (and really, the option
only enables watchdog activation in hardclock) and also avoid to
call them when not necessary (avoiding not-volountary watchdog
activations).

Sponsored by: Sandvine Incorporated
Discussed with: emaste, des
MFC after: 2 weeks

o) Properly size caches and TLB on Octeon.
o) Make COP0_SYNC do nothing on Octeon, which is fully interlocked.

Submitted by: Bhanu Prakash (with modifications)

Extend struct sysvec with new method sv_schedtail, which is used for an
explicit process at fork trampoline path instead of eventhadler(schedtail)
invocation for each child process.

Remove eventhandler(schedtail) code and change linux ABI to use newly added
sysvec method.

While here replace explicit comparing of module sysentvec structure with the
newly created process sysentvec to detect the linux ABI.

Discussed with: kib

MFC after: 2 Week

Use new thread's stack to invoke pmap_activate in cpu_switch()

Restore the SP from the new thread's PCB before calling pmap_activate.
Remove some old FIXME comments.

Use correct types and fromats for physical address

- Use vm_paddr_t for pa in pmap_steal_memory()
- Use uintmax_t and %jx to ensure that physical address are printed
correctly in cpu_startup() and pmap_bootstrap()

Use vm_offset_t for virtual addresses.

The unsigned int type used now is incorrect in n64 compilation.

Fix typos - remove duplicate "the".

PR: bin/154928
Submitted by: Eitan Adler <lists at eitanadler.com>
MFC after: 3 days

Allow the platform code to return a bitmask of running cores rather than just
a number of cores, this allows for a sparse set of CPUs. Implement support
for sparse core masks on Octeon.

XXX jeff@ suggests that all_cpus should include cores that are offline or
running other applications/OSes, so the platform API should be further
extended to allow us to set all_cpus to include all cores that are
physically-present as opposed to only those that are running FreeBSD.

Submitted by: Bhanu Prakash (with modifications)
Reviewed by: jchandra
Glanced at by: kib, jeff, jhb

Put the general logic for being a CPU hog into a new function
should_yield(). Use this in various places. Encapsulate the common
case of check-and-yield into a new function maybe_yield().

Change several checks for a magic number of iterations to use
should_yield() instead.

MFC after: 1 week

Implement sf_buf using direct map (XKPHYS) in MIPS n64.

- Provide trivial implementation of sf_buf_alloc(), sf_buf_free(),
sf_buf_kva() and sf_buf_page() using direct map for n64.
- uio_machdep.c - use macros so that the direct map will be used in
case of n64.

Reviewed by: imp (earlier version)
Obtained from: jmallett (user/jmallett/octeon)

Make MSGBUF_SIZE kernel option a loader tunable kern.msgbufsize.

Submitted by: perryh pluto.rain.com (previous version)
Reviewed by: jhb
Approved by: kib (mentor)
Tested by: universe

Introduce signed and unsigned version of CTLTYPE_QUAD, renaming
existing uses. Rename sysctl_handle_quad() to sysctl_handle_64().

For architectures not using direct map , and requiring real KVA page for
sf buf allocation, use wakeup() instead of wakeup_one() to notify sf
buffer waiters about free buffer.

sf_buf_alloc() calls msleep(PCATCH) when SFB_CATCH flag was given,
and for simultaneous wakeup and signal delivery, msleep() returns
EINTR/ERESTART despite the thread was selected for wakeup_one(). As
result, we loose a wakeup, and some other waiter will not be woken up.

Reported and tested by: az
Reviewed by: alc, jhb
MFC after: 1 week

Remove empty dev_mem_md_init() stubs.

Make cpu_model public (otherwise there's no way to set it) and bump it
to 80 characters.
Add hw.board to export board information, if known, from the mips kernel.

Support for 64 bit PTEs on n32 and n64 compilation.

In n32 and n64, add support for physical address above 4GB by having
64 bit page table entries and physical addresses. Major changes are:
- param.h: update PTE sizes, masks and shift values to support 64 bit PTEs.
- param.h: remove DELAY(), mips_btop(same as atop), mips_ptob (same as
ptoa), and reformat.
- param.h: remove casting to unsigned long in trunc_page and round_page
since this will be used on physical addresses.
- _types.h: have 64 bit __vm_paddr_t for n32.
- pte.h: update TLB LO0/1 access macros to support 64 bit PTE
- pte.h: assembly macros for PTE operations.
- proc.h: md_upte is now 64 bit for n32 and n64.
- exception.S and swtch.S: use the new PTE macros for PTE operations.
- cpufunc.h: TLB_LO0/1 registers are 64bit for n32 and n64.
- xlr_machdep.c: Add memory segments above 4GB to phys_avail[] as they are
supported now.

Reviewed by: jmallett (earlier version)

Cleanup physical address and PTE types on MIPS.

1. Use vm_paddr_t for physical addresses.

There are a few places in the MIPS platform code where vm_offset_t is
used for physical addresses, change these to use vm_paddr_t:
- phys_avail[], physmem_desc[] arrays
- pmap_mapdev(), page_is_managed(), is_cacheable_mem() pmap_map() args
- local variables of various pmap functions

2. Change init_pte_prot() return from int to pt_entry_t, as this can be
64 bit when using 64 bit TLB entries.

3. Update printing of pt_entry_t and of vm_paddr_t to use 'j' format with
uintmax_t. This will be useful later if we plan to use 64bit phsical addr
on 32 bit n32 compilation.

Reviewed by: imp

Remove bogus usage of INTR_FAST. "Fast" interrupts are now indicated by
registering a filter handler rather than a threaded handler. Also remove
a bogus use of INTR_MPSAFE for a filter.

o) When trying to determine whether the pcpu pointer is a managed address, check
not just that it is greater than the minimal kernel virtual address, but also
that it is less than the maximal kernel virtual address. On n64 kernels, the
pcpup comes out of a direct-mapped address that, with an unsigned compare, is
rather greater than the minimal kernel virtual address.
o) Turn the panic if interrupts are disabled in cpu_idle into a KASSERT since on
other architectures it's behind INVARIANTS anyway.
o) Add a check that not all interrupts are masked, too.
o) Add cpu_idleclock() and cpu_activeclock() use to cpu_idle as is done on other
architectures.

Make the ELF trampoline binary ELF executable (and do some cleanup).

- Remove the -shared flag for the trampoline binary, generate an
ELF executable instead of a shared object.
- No need to generate tmphack.S, move the code to sys/mips/mips/inckern.S
- No need generate opt_kernname.h, KERNNAME can be passed with -D

Reviewed by: gonzo, imp

Fix compilation when DEBUG option is enabled.

- remove unused code in mips/rmi/xlr_pci.c
- remove unused variable in mips/rmi/dev/nlge/if_nlge.c
- fix reference to old function in mips/mips/pmap.c

Reported by: Prabhath Raman (prabhath at netlogicmicro com)

swi_vm() for mips.

UMA_MD_SMALL_ALLOC for mips.

Implement uma_small_alloc() and uma_small_free() for mips that allocates
pages from direct mapped memory. Uses the same mechanism as the page table
page allocator, so that we allocate from KSEG0 in 32 bit, and from XKPHYS
on 64 bit.

Reviewed by: alc, jmallett

Fix double ;;

1. Fix off by one errors in calls to MIPS_DIRECT_MAPPABLE, reported by alc@
2. Remove unnecessary #defines from vmparam.h

Submitted by: alc (2)
Reviewed by: alc (1)

Fixup for r216141, dump_add_page needs to be non-static now.
Add it to sys/mips/include/md_var.h, make dump_drop_page non-static too
for completeness.

Fix issue noted by alc while reviewing r215938:
The current implementation of vm_page_alloc_freelist() does not handle
order > 0 correctly. Remove order parameter to the function and use it
only for order 0 pages.

Submitted by: alc

After some off-list discussion, revert a number of changes to the
DPCPU_DEFINE and VNET_DEFINE macros, as these cause problems for various
people working on the affected files. A better long-term solution is
still being considered. This reversal may give some modules empty
set_pcpu or set_vnet sections, but these are harmless.

Changes reverted:

------------------------------------------------------------------------
r215318 | dim | 2010-11-14 21:40:55 +0100 (Sun, 14 Nov 2010) | 4 lines

Instead of unconditionally emitting .globl's for the __start_set_xxx and
__stop_set_xxx symbols, only emit them when the set_vnet or set_pcpu
sections are actually defined.

------------------------------------------------------------------------
r215317 | dim | 2010-11-14 21:38:11 +0100 (Sun, 14 Nov 2010) | 3 lines

Apply the STATIC_VNET_DEFINE and STATIC_DPCPU_DEFINE macros throughout
the tree.

------------------------------------------------------------------------
r215316 | dim | 2010-11-14 21:23:02 +0100 (Sun, 14 Nov 2010) | 2 lines

Add macros to define static instances of VNET_DEFINE and DPCPU_DEFINE.

Apply the STATIC_VNET_DEFINE and STATIC_DPCPU_DEFINE macros throughout
the tree.

Fix typos.

PR: bin/148894
Submitted by: olgeni

- Provide more registers for GDB

- Add minidump support for FreeBSD/mips

Adjust the order of operations in spinlock_enter() and spinlock_exit() to
work properly with single-stepping in a kernel debugger. Specifically,
these routines have always disabled interrupts before increasing the nesting
count and restored the prior state of interrupts after decreasing the nesting
count to avoid problems with a nested interrupt not disabling interrupts
when acquiring a spin lock. However, trap interrupts for single-stepping
can still occur even when interrupts are disabled. Now the saved state of
interrupts is not saved in the thread until after interrupts have been
disabled and the nesting count has been increased. Similarly, the saved
state from the thread cannot be read once the nesting count has been
decreased to zero. To fix this, use temporary variables to store interrupt
state and shuffle it between the thread's MD area and the appropriate
registers.

In cooperation with: bde
MFC after: 1 month

Use ABI-aware macros for setting up a fake frame.

Enforce that 'pmap_kenter()' is only used to establish cacheable mappings.

Mappings with other cacheability attributes can be established, if needed,
by using 'pmap_kenter_attr()'.

Suggested by: jchandra, imp

Get rid of the unnecessary redirection of 'is_cacheable_mem()' to
'is_physical_memory()' through a macro.

Implement 'is_cacheable_mem()' directly instead.

Make the meaning of the 'mask' argument to 'set_intr_mask(mask)' consistent
with the meaning of IM bits in the status register.

Reviewed by: jmallett, jchandra

Port r212559 to mips.

Do not explicitly enable interrupts in smp_init_secondary() because it
renders any spinlock protected code after that point to run with
interrupts enabled. This is because the processor is executing in the
context of idlethread whose 'md_spinlock_count' is already set to 1.

Instead just let sched_throw() re-enable interrupts when it releases
the spinlock.

The original powerpc commit log for r212559 is available here:
http://svn.freebsd.org/viewvc/base?view=revision&revision=212559

Enforce that pmap_mapdev() always returns uncacheable mappings.

Reviewed by: imp, jchandra, jmallett

Refactor timer management code with priority to one-shot operation mode.
The main goal of this is to generate timer interrupts only when there is
some work to do. When CPU is busy interrupts are generating at full rate
of hz + stathz to fullfill scheduler and timekeeping requirements. But
when CPU is idle, only minimum set of interrupts (down to 8 interrupts per
second per CPU now), needed to handle scheduled callouts is executed.
This allows significantly increase idle CPU sleep time, increasing effect
of static power-saving technologies. Also it should reduce host CPU load
on virtualized systems, when guest system is idle.

There is set of tunables, also available as writable sysctls, allowing to
control wanted event timer subsystem behavior:
kern.eventtimer.timer - allows to choose event timer hardware to use.
On x86 there is up to 4 different kinds of timers. Depending on whether
chosen timer is per-CPU, behavior of other options slightly differs.
kern.eventtimer.periodic - allows to choose periodic and one-shot
operation mode. In periodic mode, current timer hardware taken as the only
source of time for time events. This mode is quite alike to previous kernel
behavior. One-shot mode instead uses currently selected time counter
hardware to schedule all needed events one by one and program timer to
generate interrupt exactly in specified time. Default value depends of
chosen timer capabilities, but one-shot mode is preferred, until other is
forced by user or hardware.
kern.eventtimer.singlemul - in periodic mode specifies how much times
higher timer frequency should be, to not strictly alias hardclock() and
statclock() events. Default values are 2 and 4, but could be reduced to 1
if extra interrupts are unwanted.
kern.eventtimer.idletick - makes each CPU to receive every timer interrupt
independently of whether they busy or not. By default this options is
disabled. If chosen timer is per-CPU and runs in periodic mode, this option
has no effect - all interrupts are generating.

As soon as this patch modifies cpu_idle() on some platforms, I have also
refactored one on x86. Now it makes use of MONITOR/MWAIT instrunctions
(if supported) under high sleep/wakeup rate, as fast alternative to other
methods. It allows SMP scheduler to wake up sleeping CPUs much faster
without using IPI, significantly increasing performance on some highly
task-switching loads.

Tested by: many (on i386, amd64, sparc64 and powerc)
H/W donated by: Gheorghe Ardelean
Sponsored by: iXsystems, Inc.

The functions in sys/mips/mips/psraccess.S can be implemented with
mips_rd_status/mips_wr_status. Implement them in mips/include/cpufunc.h,
and remove psraccess.S.

Reviewed by: neel, imp

bus_add_child: change type of order parameter to u_int

This reflects actual type used to store and compare child device orders.
Change is mostly done via a Coccinelle (soon to be devel/coccinelle)
semantic patch.
Verified by LINT+modules kernel builds.

Followup to: r212213
MFC after: 10 days

Whitespace fixes - indent with tabs instead of spaces.
Few other style(9) changes.

On boards with >512MB memory, the result of vtophys cannot be converted
to KSEG1 address - use pmap_mapdev/pmap_unmapdev instead.

Remove unused variable maxphys.

Apply MIPS pmap clean up patch from alc@ (with minor change to KASSERT):

PMAP_DIAGNOSTIC was eliminated from amd64/i386, and, in fact, the
non-MIPS parts of the kernel, several years ago. Any of the interesting
checks were turned into KASSERT()s. Basically, the motivation was that
lots of people run with INVARIANTS but no one runs with DIAGNOSTIC.

panic strings needn't and shouldn't have a terminating newline.

Finally, there is one functional change. The sched_pin() in
pmap_remove_pages() is an artifact of the way we temporarily map page
table pages on i386. (The mappings are processor private. We don't do
a system-wide shootdown.) It isn't needed by MIPS.

Tested by: jchandra

Submitted by: alc

MIPS n64 support - continued...

1. On n64, use XKPHYS to map page table pages instead of KSEG0. Maintain
just one freepages list on n64.

The changes are mainly to introduce MIPS_PHYS_TO_DIRECT(pa),
MIPS_DIRECT_TO_PHYS(), which will use KSEG0 in 32 bit compilation
and XKPHYS in 64 bit compilation.

2. Change macro based PMAP_LMEM_MAP1(), PMAP_LMEM_MAP2(), PMAP_LMEM_UNMAP()
to inline functions.

3. Introduce MIPS_DIRECT_MAPPABLE(pa), which will further reduce the cases
in which we will need to have a special case for 64 bit compilation.

4. Update CP0 hazard definitions for CPU_RMI - the cpu does not need any
nops

Reviewed by: neel

Make return statements style(9) compliant in MIPS pmap.c

Reported by: neel

Reviewed by: neel (earlier version)

Rename TARGET_XLR_XLS to CPU_RMI to match other CPU_xxx definitions.
use CPU_RMI all XLR configurations. Update ident string for N32 and
N64 kernels.

SMP support in n64.

- Enable KX and UX bits on CPU startup for non-boot CPUs
- Keep the KX bit when in userspace - XTLB handler needs it to access
PCPU data
- revert r210638 partly - we don't need to enable KX on kernel entry
now

Reviewed by: jmallett, imp

Implement pmap changes suggested by alc@:

1. Move dirty bit emulation code that is duplicted for kernel and user
in trap.c to a function pmap_emulate_modified() in pmap.c.

2. While doing dirty bit emulation, it is not necessary to update the
TLB entry on all CPUs using smp_rendezvous(), we can just update the
TLB entry on the current CPU, and let the other CPUs update their TLB
entry lazily if they get an exception.

Reviewed by: alc, neel

Optimization for pmap_kenter(), call pmap_update_page() only when
necessary. On SMP, pmap_update_page() can be costly as it needs a
a smp_rendezvous().

Reviewed by: alc, neel
Obtained from: jmallett (http://svn.freebsd.org/base/user/jmallett/octeon)

Re-arrange the pmap calls that use smp_rendezvous() on SMP, so that their
per-cpu variants are also available to be called. The per-cpu variants
are needed for some later optimizations.

Also remove unnecessary casts, do some style fixes.

Reviewed by: alc, neel

Fix compilation when DDB disabled. Adds 'ifdef DDB' around
DB_SHOW_COMMAND definitions.

loadandclear() for PTEs are not needed on MIPS. The PTEs are software
managed and we already take pmap lock for PTE operations(see r210922)

Reviewed by: alc

- Consolidate the the cache coherence attribute definitions in a single place.
Adapted from Juli's changes to pte.h in the octeon branch:
http://svn.freebsd.org/viewvc/base/user/jmallett/octeon/sys/mips/include/pte.h

- Set the KX and UX bits in the status register for n64 kernels.

Reviewed by: jmallett

Add a new ipi_cpu() function to the MI IPI API that can be used to send an
IPI to a specific CPU by its cpuid. Replace calls to ipi_selected() that
constructed a mask for a single CPU with calls to ipi_cpu() instead. This
will matter more in the future when we transition from cpumask_t to
cpuset_t for CPU masks in which case building a CPU mask is more expensive.

Submitted by: peter, sbruno
Reviewed by: rookie
Obtained from: Yahoo! (x86)
MFC after: 1 month

Fix issue reported by alc :

MIPS doesn't really need to use atomic_cmpset_int() in situations like
this because the software dirty bit emulation in trap.c acquires
the pmap lock. Atomics like this appear to be a carryover from i386
where the hardware-managed TLB might concurrently set the modified bit.

Reviewed by: alc

Fix the issue reported by alc:

pmap_page_wired_mappings() counts the number of pv entries for the
specified page that have the pv entry wired flag set to TRUE.
pmap_enter() correctly initializes this flag. However,
pmap_change_wiring() doesn't update the corresponding pv entry flag,
only the PTE. So, the count returned by pmap_page_wired_mappings()
will sometimes be wrong.

In the short term, the best fix would be to eliminate the pv entry
flag and use only the PTE. That flag is wasting non-trivial memory.

Remove pv_wired flag, and use PTE flag to count the wired mappings.

Reviewed by: alc

Fix a race between clock_intr() and tick_ticker() when updating
'counter_upper' and 'counter_lower_last'. The race exists because
interrupts are enabled even though tick_ticker() executes in a
critical section.

Fix a bug in clock_intr() in how it updates the cached values of
'counter_upper' and 'counter_lower_last'. They are updated only
when the COUNT register rolls over. More interestingly it will *never*
update the cached values if 'counter_lower_last' happens to be zero.

Get rid of superfluous critical section in clock_intr(). There is no
reason to do this because clock_intr() executes in hard interrupt
context.

Switch back to using 'tick_ticker()' as the cpu ticker for Sibyte.

Reviewed by: jmallett, mav

Add 3 level page tables for MIPS in n64.

- 32 bit compilation will still use old 2 level page tables
- re-arrange pmap code so that adding another level is easier
- pmap code for 3 level page tables for n64
- update TLB handler to traverse 3 levels in n64

Reviewed by: jmallett

Use PTR_ADDU to change sp, so that it works for n64 too.

Use fuword32() to fetch instructions, this will work on both 32 and 64 bit
compilation.

n64 support - enable UX bit in STATUS for kernel and userspace.

- enable UX in kernel start, and kernel entry
- keep UX flag in cpu_fork and cpu_set_upcall
- enable UX for userspace

MIPS n64 support - support kstack in XKSEG.
- enable KX on entry from user-space, we need KX set to save to XKSEG
addresses.
- add MIPS_XKSEG_START to genassym.c
- Add n64 case for swth.S

Prepare for 3 level page tables for MIPS.

- Move page table second level shift and mask to param.h
- rename SEGOFSET to SEGMASK
- fix values for 64 bit maximum kernel and user addresses.

Add fuiword() in n64 for completeness.

MIPS 64 bit support. Define fuword64() for n64 compilation, fuword() should
be fuword64() in 64 bit.

Fix build for o32 kernels.

The emulation of 'ld' and 'sd' instructions only works for ABIs that support
64-bit registers and the instructions 'ldl' and 'ldr' that operate on those
registers.

Reviewed by: jmallett

o) Subtract 64K from the default userland stack pointer. GCC generate code
that with a 32-bit ABI on a system with 64-bit registers can attempt to
access an invalid (well, kernel) memory address rather than the intended
user address for stack-relative loads and stores. Lowering the stack
pointer works around this. [1]
o) Make TRAP_DEBUG code conditional on the trap_debug variable. Make
trap_debug default to 0 instead of 1 now but make it possible to change it
at runtime using sysctl.
o) Kill programs that attempt an unaligned access of a kernel address. Note
that with some ABIs, calling useracc() is not sufficient since the register
may be 64-bit but vm_offset_t is 32-bit so a kernel address could be
truncated to what looks like a valid user address, allowing the user to
crash the kernel.
o) Clean up unaligned access emulation to support unaligned 16-bit and 64-bit
accesses. (For 16-bit accesses it was checking for user access to too much
memory (4 bytes) and there was no 64-bit support.) This still lacks support
for unaligned load-linked and store-conditional.

Reviewed by: [1] gonzo

Add some comments.

Update MIPS timer code (except RMI) to utilize new MI event timer
infrastructure.

Reviewed by: neel

Redo the page table page allocation on MIPS, as suggested by
alc@.

The UMA zone based allocation is replaced by a scheme that creates
a new free page list for the KSEG0 region, and a new function
in sys/vm that allocates pages from a specific free page list.

This also fixes a race condition introduced by the UMA based page table
page allocation code. Dropping the page queue and pmap locks before
the call to uma_zfree, and re-acquiring them afterwards will introduce
a race condtion(noted by alc@).

The changes are :
- Revert the earlier changes in MIPS pmap.c that added UMA zone for
page table pages.
- Add a new freelist VM_FREELIST_HIGHMEM to MIPS vmparam.h for memory that
is not directly mapped (in 32bit kernel). Normal page allocations will first
try the HIGHMEM freelist and then the default(direct mapped) freelist.
- Add a new function 'vm_page_t vm_page_alloc_freelist(int flind, int
order, int req)' to vm/vm_page.c to allocate a page from a specified
freelist. The MIPS page table pages will be allocated using this function
from the freelist containing direct mapped pages.
- Move the page initialization code from vm_phys_alloc_contig() to a
new function vm_page_alloc_init(), and use this function to initialize
pages in vm_page_alloc_freelist() too.
- Split the function vm_phys_alloc_pages(int pool, int order) to create
vm_phys_alloc_freelist_pages(int flind, int pool, int order), and use
this function from both vm_page_alloc_freelist() and vm_phys_alloc_pages().

Reviewed by: alc

Remove unused file. Replaced by tlb.c.

Update the port of FreeBSD to Cavium Octeon to use the Cavium Simple Executive
library:
o) Increase inline unit / large function growth limits for MIPS to accommodate
the needs of the Simple Executive, which uses a shocking amount of inlining.
o) Remove TARGET_OCTEON and use CPU_CNMIPS to do things required by cnMIPS and
the Octeon SoC.
o) Add OCTEON_VENDOR_LANNER to use Lanner's allocation of vendor-specific
board numbers, specifically to support the MR320.
o) Add OCTEON_BOARD_CAPK_0100ND to hard-wire configuration for the CAPK-0100nd,
which improperly uses an evaluation board's board number and breaks board
detection at runtime. This board is sold by Portwell as the CAM-0100.
o) Add support for the RTC available on some Octeon boards.
o) Add support for the Octeon PCI bus. Note that rman_[sg]et_virtual for IO
ports can not work unless building for n64.
o) Clean up the CompactFlash driver to use Simple Executive macros and
structures where possible (it would be advisable to use the Simple Executive
API to set the PIO mode, too, but that is not done presently.) Also use
structures from FreeBSD's ATA layer rather than structures copied from
Linux.
o) Print available Octeon SoC features on boot.
o) Add support for the Octeon timecounter.
o) Use the Simple Executive's routines rather than local copies for doing reads
and writes to 64-bit addresses and use its macros for various device
addresses rather than using local copies.
o) Rename octeon_board_real to octeon_is_simulation to reduce differences with
Cavium-provided code originally written for Linux. Also make it use the
same simplified test that the Simple Executive and Linux both use rather
than our complex one.
o) Add support for the Octeon CIU, which is the main interrupt unit, as a bus
to use normal interrupt allocation and setup routines.
o) Use the Simple Executive's bootmem facility to allocate physical memory for
the kernel, rather than assuming we know which addresses we can steal.
NB: This may reduce the amount of RAM the kernel reports you as having if
you are leaving large temporary allocations made by U-Boot allocated
when starting FreeBSD.
o) Add a port of the Cavium-provided Ethernet driver for Linux. This changes
Ethernet interface naming from rgmxN to octeN. The new driver has vast
improvements over the old one, both in performance and functionality, but
does still have some features which have not been ported entirely and there
may be unimplemented code that can be hit in everyday use. I will make
every effort to correct those as they are reported.
o) Support loading the kernel on non-contiguous cores.
o) Add very conservative support for harvesting randomness from the Octeon
random number device.
o) Turn SMP on by default.
o) Clean up the style of the Octeon kernel configurations a little and make
them compile with -march=octeon.
o) Add support for the Lanner MR320 and the CAPK-0100nd to the Simple
Executive.
o) Modify the Simple Executive to build on FreeBSD and to build without
executive-config.h or cvmx-config.h. In the future we may want to
revert part of these changes and supply executive-config.h and
cvmx-config.h and access to the options contained in those files via
kernel configuration files.
o) Modify the Simple Executive USB routines to support getting and setting
of the USB PID.

Move common macros into asm.h. Replace MIPS_CPU_NOP_DELAY with
HAZARD_DELAY. Move HAZARD_DELAY and ITLBNOPFIX into asm.h, for
possible later optimization...

Reviewed by: jmallet, jchandra

Remove i386-ish sysctls. Also, make the bootinfo sysctl OID_AUTO.

Remove one layer of indirection. No need to call cpu_throw which then
calls mips_cpu_call via an obfuscated assembler call. Instead, delete
the current cpu_throw, and rename mips_cpu_throw to cpu_throw. This
is nicer to the cache on each context switch (since fixed jumps can be
prefected, while jumps through a register can't). Incidentally, it
also saves about 5 or 6 instructions.

Reviewed by: jmallet@

Prefer the cpuregs.h spellings of register and bit names over cpu.h.

Use the cpuregs.h spellings for these registers rather than the cpu.h
spelling.

cpu_id and fpu_id are unused, except to be set early in the boot code.
The problem with setting it there is that the last CPU to come up
wins, it seems. This also removes one more ifdef in locore.S, a noble
goal too. Since they are unused, and pollute cpu.h, remove them.

Submitted by: bde.h (cpu.h pollution)
Approved in theory by: jmallet@

Merge jmallett@'s n64 work into HEAD

Initial set of ddb changes to support n64

Reviewed by: imp
Obtained from: jmallett (http://svn.freebsd.org/base/user/jmallett/octeon)

Merge jmallett@'s n64 work into HEAD.

Merge changes for initial n64 support in pmap.c. Use direct mapped (XKPHYS)
access for a lot of operations that earlier needed temporary mapping. Add
support for using XKSEG for kernel mappings.

Reviewed by: imp
Obtained from: jmallett (http://svn.freebsd.org/base/user/jmallett/octeon)

Enable KX bit, which is needed for 64 bit access, in status register
for XLR. Update exception handlers and other functions which set/change
status registers to preserve this.

Approved by: rrs

Fix minor issue in sys/mips/mips/cache_mipsNN.c, using int type for
virtual addr will break on 64 bit, change to vm_offset_t.

Reviewed by: imp@
Approved by: rrs (mentor)

Merge jmallett@'s n64 work into HEAD - changeset 8

Updated PTE/PDE macros from http://svn.freebsd.org/base/user/jmallett/octeon
Introduce pmap_segshift() macro, use pmap_segmap() in place of pmap_pde, and
remove pmap_pde().

Approved by: rrs (mentor)
Obtained from: jmallett@

Remove save/restore of PageMask in tlb.c functions introduced in r209243.
If we save/restore the PageMask, the value set by the bootloader will
persist, and will cause problems later in TLB exception handler.
This caused a crash in AR71xx boards.

Also fixes the EntryHi mask in pte.h

Reported by: Luiz Otavio O Souza <lists.br@gmail.com>
Tested by: Luiz Otavio O Souza <lists.br@gmail.com>

Approved by: rrs (mentor)

Move prototypes for kern_sigtimedwait() and kern_sigprocmask() to
<sys/syscallsubr.h> where all other kern_<syscall> prototypes live.

Merge jmallett@'s n64 work into HEAD - changeset 7

Initial support for n32 and n64 ABIs from
http://svn.freebsd.org/base/user/jmallett/octeon

Changes are:
- syscall, exception and trap support for n32/n64 ABIs
- 64-bit address space defines
- _jmp_buf for n32/n64
- casts between registers and ptr/int updated to work on n32/n64

Approved by: rrs(mentor), jmallett

Merge jmallett@'s n64 work into HEAD - changeset 6

PTE flag cleanup from http://svn.freebsd.org/base/user/jmallett/octeon
- Rename PTE_xx flags to match their MIPS names
- Use the new pte_set/test/clear macros uniformly, instead of a mixture
of mips_pg_xxx(), pmap_pte_x() macros and direct access.
- Remove unused macros and defines from pte.h and pmap.c

Discussed on freebsd-mips@

Approved by: rrs(mentor), jmallett

Merge jmallett@'s n64 work into HEAD - changeset 5

Remove unnecessary locking and sched_pin() call while creating a temporary
mapping.

Changes from http://svn.freebsd.org/base/user/jmallett/octeon

Approved by: rrs (mentor), jmallett

Merge jmallett@'s n64 work into HEAD - changeset 4

Re-write tlb operations in C with a simpler API.
Update callers to use the new API.

Changes from http://svn.freebsd.org/base/user/jmallett/octeon

Approved by: rrs(mentor), jmallett

Relax one of the new assertions in pmap_enter() a little. Specifically,
allow pmap_enter() to be performed on an unmanaged page that doesn't have
VPO_BUSY set. Having VPO_BUSY set really only matters for managed pages.
(See, for example, pmap_remove_write().)

Reduce the scope of the page queues lock and the number of
PG_REFERENCED changes in vm_pageout_object_deactivate_pages().
Simplify this function's inner loop using TAILQ_FOREACH(), and shorten
some of its overly long lines. Update a stale comment.

Assert that PG_REFERENCED may be cleared only if the object containing
the page is locked. Add a comment documenting this.

Assert that a caller to vm_page_requeue() holds the page queues lock,
and assert that the page is on a page queue.

Push down the page queues lock into pmap_ts_referenced() and
pmap_page_exists_quick(). (As of now, there are no longer any pmap
functions that expect to be called with the page queues lock held.)

Neither pmap_ts_referenced() nor pmap_page_exists_quick() should ever
be passed an unmanaged page. Assert this rather than returning "0"
and "FALSE" respectively.

ARM:

Simplify pmap_page_exists_quick() by switching to TAILQ_FOREACH().

Push down the page queues lock inside of pmap_clearbit(), simplifying
pmap_clear_modify(), pmap_clear_reference(), and pmap_remove_write().
Additionally, this allows for avoiding the acquisition of the page
queues lock in some cases.

PowerPC/AIM:

moea*_page_exits_quick() and moea*_page_wired_mappings() will never be
called before pmap initialization is complete. Therefore, the check
for moea_initialized can be eliminated.

Push down the page queues lock inside of moea*_clear_bit(),
simplifying moea*_clear_modify() and moea*_clear_reference().

The last parameter to moea*_clear_bit() is never used. Eliminate it.

PowerPC/BookE:

Simplify mmu_booke_page_exists_quick()'s control flow.

Reviewed by: kib@

Don't set PG_WRITEABLE in init_pte_prot() (and thus pmap_enter()) unless
the page is managed.

Don't set the machine-independent layer's dirty field for the page being
mapped in init_pte_prot(). (The dirty field is only supposed to set when
a mapping is removed or write-protected and the page was managed and
modified.)

Determine whether or not to perform dirty bit emulation based on whether
or not the page is managed, i.e., pageable, not based on whether the page
is being mapped into the kernel address space. Nearly all of the kernel
address space consists of unmanaged pages, so this has neglible impact on
the overhead of dirty bit emulation for the kernel address space. However,
there can also exist unmanaged pages in the user address space. Previously,
dirty bit emulation was unnecessarily performed on these pages.

Tested by: jchandra@

Make vm_contig_grow_cache() extern, and use it when vm_phys_alloc_contig()
fails to allocate MIPS page table pages. The current usage of VM_WAIT in
case of vm_phys_alloc_contig() failure is not correct, because:

"There is no guarantee that any of the available free (or cached) pages
after the VM_WAIT will fall within the range of suitable physical
addresses. Every time this function sleeps and a single page is freed
(or cached) by someone else, this function will be reawakened. With
a little bad luck, you could spin indefinitely."

We also add low and high parameters to vm_contig_grow_cache() and
vm_contig_launder() so that we restrict vm_contig_launder() to the range
of pages we are interested in.

Reported by: alc

Reviewed by: alc
Approved by: rrs (mentor)

Fix a KASSERT() that was broken in r208665.

Reported by: jmallett

Eliminate a stale comment.

Merge portions of r208645 and supporting code from the i386 pmap:
When I pushed down the page queues lock into pmap_is_modified(), I created
an ordering dependence: A pmap operation that clears PG_WRITEABLE and calls
vm_page_dirty() must perform the call first. Otherwise, pmap_is_modified()
could return FALSE without acquiring the page queues lock because the page
is not (currently) writeable, and the caller to pmap_is_modified() might
believe that the page's dirty field is clear because it has not seen the
effect of the vm_page_dirty() call.

When I pushed down the page queues lock into pmap_is_modified(), I
overlooked one place where this ordering dependence is violated:
pmap_enter(). In a rare situation pmap_enter() can be called to replace a
dirty mapping to one page with a mapping to another page. (I say rare
because replacements generally occur as a result of a copy-on-write fault,
and so the old page is not dirty.) This change delays clearing PG_WRITEABLE
until after vm_page_dirty() has been called.

Fixing the ordering dependency also makes it easy to introduce a small
optimization: When pmap_enter() used to replace a mapping to one page with a
mapping to another page, it freed the pv entry for the first mapping and
later called the pv entry allocator for the new mapping. Now, pmap_enter()
attempts to recycle the old pv entry, saving two calls to the pv entry
allocator.

Simplify the inner loop of get_pv_entry(): While iterating over the page's
pv list, there is no point in checking whether or not the pv list is empty,
wait instead until the loop completes.

Fix lock order reversal, unlock page queue and pmap locks before
calling uma_zfree().
Also if needed, acquire page queue lock before modifying pte page
attributes.

Approved by: rrs (mentor)

Get rid of unused variable 'virtual_sys_start'.

If 'timer2hz' is zero then we don't need to call 'timer2clock()' directly. It
will be called automatically by 'timer1clock()'.

Do profiling as often as possible by running it as the same frequency as
'timer1hz'. The statistics clock is run as close to 128Hz as possible.

Pointed out by: mav@

Call VM_WAIT in pmap_ptpgzone_allocf() if M_WAITOK is set.
Removed unused variable.

Approved by: rrs (mentor)

Simplify clock interrupt handling on mips by using the new KPI - timer1clock()
and timer2clock().

Dynamically adjust the tick frequency depending on the value of 'hz'. Tested
with hz values of 100, 1000 and 2000.

- Fix kseg0 address calculation - it doesn't always start at
page boundary
- Add cache ops to ensure memory validity before/after
copy operation

Push down page queues lock acquisition in pmap_enter_object() and
pmap_is_referenced(). Eliminate the corresponding page queues lock
acquisitions from vm_map_pmap_enter() and mincore(), respectively. In
mincore(), this allows some additional cases to complete without ever
acquiring the page queues lock.

Assert that the page is managed in pmap_is_referenced().

On powerpc/aim, push down the page queues lock acquisition from
moea*_is_modified() and moea*_is_referenced() into moea*_query_bit().
Again, this will allow some additional cases to complete without ever
acquiring the page queues lock.

Reorder a few statements in vm_page_dontneed() so that a race can't lead
to an old reference persisting. This scenario is described in detail by a
comment.

Correct a spelling error in vm_page_dontneed().

Assert that the object is locked in vm_page_clear_dirty(), and restrict the
page queues lock assertion to just those cases in which the page is
currently writeable.

Add object locking to vnode_pager_generic_putpages(). This was the one
and only place where vm_page_clear_dirty() was being called without the
object being locked.

Eliminate an unnecessary vm_page_lock() around vnode_pager_setsize()'s call
to vm_page_clear_dirty().

Change vnode_pager_generic_putpages() to the modern-style of function
definition. Also, change the name of one of the parameters to follow
virtual memory system naming conventions.

Reviewed by: kib

Get rid of empty and unused KSEG0TEXT macros.

Fix mips kernel build breakage caused by revision 208504.

Roughly half of a typical pmap_mincore() implementation is machine-
independent code. Move this code into mincore(), and eliminate the
page queues lock from pmap_mincore().

Push down the page queues lock into pmap_clear_modify(),
pmap_clear_reference(), and pmap_is_modified(). Assert that these
functions are never passed an unmanaged page.

Eliminate an inaccurate comment from powerpc/powerpc/mmu_if.m:
Contrary to what the comment says, pmap_mincore() is not simply an
optimization. Without a complete pmap_mincore() implementation,
mincore() cannot return either MINCORE_MODIFIED or MINCORE_REFERENCED
because only the pmap can provide this information.

Eliminate the page queues lock from vfs_setdirty_locked_object(),
vm_pageout_clean(), vm_object_page_collect_flush(), and
vm_object_page_clean(). Generally speaking, these are all accesses
to the page's dirty field, which are synchronized by the containing
vm object's lock.

Reduce the scope of the page queues lock in vm_object_madvise() and
vm_page_dontneed().

Reviewed by: kib (an earlier version)

Reorganize syscall entry and leave handling.

Extend struct sysvec with three new elements:
sv_fetch_syscall_args - the method to fetch syscall arguments from
usermode into struct syscall_args. The structure is machine-depended
(this might be reconsidered after all architectures are converted).
sv_set_syscall_retval - the method to set a return value for usermode
from the syscall. It is a generalization of
cpu_set_syscall_retval(9) to allow ABIs to override the way to set a
return value.
sv_syscallnames - the table of syscall names.

Use sv_set_syscall_retval in kern_sigsuspend() instead of hardcoding
the call to cpu_set_syscall_retval().

The new functions syscallenter(9) and syscallret(9) are provided that
use sv_*syscall* pointers and contain the common repeated code from
the syscall() implementations for the architecture-specific syscall
trap handlers.

Syscallenter() fetches arguments, calls syscall implementation from
ABI sysent table, and set up return frame. The end of syscall
bookkeeping is done by syscallret().

Take advantage of single place for MI syscall handling code and
implement ptrace_lwpinfo pl_flags PL_FLAG_SCE, PL_FLAG_SCX and
PL_FLAG_EXEC. The SCE and SCX flags notify the debugger that the
thread is stopped at syscall entry or return point respectively. The
EXEC flag augments SCX and notifies debugger that the process address
space was changed by one of exec(2)-family syscalls.

The i386, amd64, sparc64, sun4v, powerpc and ia64 syscall()s are
changed to use syscallenter()/syscallret(). MIPS and arm are not
converted and use the mostly unchanged syscall() implementation.

Reviewed by: jhb, marcel, marius, nwhitehorn, stas
Tested by: marcel (ia64), marius (sparc64), nwhitehorn (powerpc),
stas (mips)
MFC after: 1 month

- Use ptpgzone zone to allocate page table pages irrespective of the amount of
memory on a platform. Tested on the Sibyte with 256MB and 1GB memory
configurations.

- Replace vtophys() with MIPS_KSEG0_TO_PHYS() to convert a page table
page's virtual address to physical. We can safely do this because
page table pages are allocated out of KSEG0.

- Add an assertion to verify that when a page table page is freed it
contains all zeroes. We can now use it after allocation without
zeroing it.

- Adjust the whitespace for the lines that output fields in 'show pcpu' in
DDB so that all the fields line up.
- Print out the tid of the per-CPU idlethread instead of the pid since
the idle process is now shared across all idle threads.

MFC after: 1 month

Adds JC's cleanup patches that fix it so
we call an platform dependant topo function as
well as clean up all the XLR specific ifdefs around
smp platform init.

Obtained from: JC

On entry to pmap_enter(), assert that the page is busy. While I'm
here, make the style of assertion used by pmap_enter() consistent
across all architectures.

On entry to pmap_remove_write(), assert that the page is neither
unmanaged nor fictitious, since we cannot remove write access to
either kind of page.

With the push down of the page queues lock, pmap_remove_write() cannot
condition its behavior on the state of the PG_WRITEABLE flag if the
page is busy. Assert that the object containing the page is locked.
This allows us to know that the page will neither become busy nor will
PG_WRITEABLE be set on it while pmap_remove_write() is running.

Correct a long-standing bug in vm_page_cowsetup(). We cannot possibly
do copy-on-write-based zero-copy transmit on unmanaged or fictitious
pages, so don't even try. Previously, the call to pmap_remove_write()
would have failed silently.

This pushes all of JC's patches that I have in place. I
am now able to run 32 cores ok.. but I still will hang
on buildworld with a NFS problem. I suspect I am missing
a patch for the netlogic rge driver.

JC check and see if I am missing anything except your
core-mask changes

Obtained from: JC

Remove some stray ';'s

Submitted by: marc balmer

Push down the page queues into vm_page_cache(), vm_page_try_to_cache(), and
vm_page_try_to_free(). Consequently, push down the page queues lock into
pmap_enter_quick(), pmap_page_wired_mapped(), pmap_remove_all(), and
pmap_remove_write().

Push down the page queues lock into Xen's pmap_page_is_mapped(). (I
overlooked the Xen pmap in r207702.)

Switch to a per-processor counter for the total number of pages cached.

Eliminate dead code.

Fix DDB backtrace involving kernel modules.

We can no longer assume that all valid program counter values reside
within the kernel object file.

Bug in the memory mapping module. The wrong
physaddr was being used in the macro (1 should be
used not 2)...

Obtained from: JC

On Alan's advice, rather than do a wholesale conversion on a single
architecture from page queue lock to a hashed array of page locks
(based on a patch by Jeff Roberson), I've implemented page lock
support in the MI code and have only moved vm_page's hold_count
out from under page queue mutex to page lock. This changes
pmap_extract_and_hold on all pmaps.

Supported by: Bitgravity Inc.

Discussed with: alc, jeffr, and kib

Adapt i386 r207205 to mips: Clearing PV_TABLE_REF and setting the page's
PG_REFERENCED flag in pmap_protect() can't really be justified, so don't
do it.

Resurrect pmap_is_referenced() and use it in mincore(). Essentially,
pmap_ts_referenced() is not always appropriate for checking whether or
not pages have been referenced because it clears any reference bits
that it encounters. For example, in mincore(), clearing the reference
bits has two negative consequences. First, it throws off the activity
count calculations performed by the page daemon. Specifically, a page
on which mincore() has called pmap_ts_referenced() looks less active
to the page daemon than it should. Consequently, the page could be
deactivated prematurely by the page daemon. Arguably, this problem
could be fixed by having mincore() duplicate the activity count
calculation on the page. However, there is a second problem for which
that is not a solution. In order to clear a reference on a 4KB page,
it may be necessary to demote a 2/4MB page mapping. Thus, a mincore()
by one process can have the side effect of demoting a superpage
mapping within another process!

Most MIPS systems have a comparatively-sparse physical memory layout. Switch
to using the sparse physseg layout in the VM system.

Build some nops into CLEAR_STATUS here to make sure that the following
instructions can't be interrupted.

Large memory mappings are always CPU local and always done with interrupts
disabled. Be doubly-sure that we don't try to do a TLB shootdown on SMP
systems for those mappings.

Submitted by: C. Jayachandran

Fix compilation error.

tick.c:298:5: error: "KDTRACE_HOOKS" is not defined

Add the necessary hooks for dtrace cyclic module.

Remove unused file.

o) Eliminate the "stand" frame and its use. Use CALLFRAME_* everywhere.
o) Use <machine/asm.h> macros for register-width, etc., rather than doing it
by hand in a few more assembly files.
o) Reduce diffs between various bits of TLB refill code in exception.S and
between interrupt processing code.
o) Use PTR_* to operate on registers that are pointers (e.g. sp).
o) Add and use a macro, CLEAR_PTE_SWBITS rather than using the
mysteriously-named WIRED_SHIFT to select bits to truncate when loading PTEs.
o) Don't doubly disable interrupts by moving zero to the status register,
especially since that has the nasty side-effect of taking us out of 64-bit
mode.
o) Use CLEAR_STATUS to disable interrupts the first time.
o) Keep SR_PX set as well as SR_[KSU]X when doing exception processing. This
is the bit that determines whether 64-bit operations are allowed.
o) Don't enable interrupts until configure_final(), like most other ports.

o) Fix XKPHYS physical address extraction. Also define cache coherency
attributes for XKPHYS.
o) Make coprocessor 0 accessor function macros for register+selector registers
take the full name so that e.g. (as done in this commit), prid selector 1
can be written through mips_wr_ebase() rather than mips_wr_prid1().
o) Allow for sign extension of 32-bit segment addresses.
o) Remove an unused MIPS-I register number.

o) Add a VM find-space option, VMFS_TLB_ALIGNED_SPACE, which searches the
address space for an address as aligned by the new pmap_align_tlb()
function, which is for constraints imposed by the TLB. [1]
o) Add a kmem_alloc_nofault_space() function, which acts like
kmem_alloc_nofault() but allows the caller to specify which find-space
option to use. [1]
o) Use kmem_alloc_nofault_space() with VMFS_TLB_ALIGNED_SPACE to allocate the
kernel stack address on MIPS. [1]
o) Make pmap_align_tlb() on MIPS align addresses so that they do not start on
an odd boundary within the TLB, so that they are suitable for insertion as
wired entries and do not have to share a TLB entry with another mapping,
assuming they are appropriately-sized.
o) Eliminate md_realstack now that the kstack will be appropriately-aligned on
MIPS.
o) Increase the number of guard pages to 2 so that we retain the proper
alignment of the kstack address.

Reviewed by: [1] alc
X-MFC-after: Making sure alc has not come up with a better interface.

o) Make pcb_onfault a pointer rather than an obscure integer value.
o) Mask off PAGE_MASK bits in pmap_update_page, etc., rather than modifying the
badvaddr in trapframe. Some nearby interfaces already did this.
o) Make PTEs "unsigned int" for now, not "unsigned long" -- we are only ready
for them to be 32-bit on 64-bit platforms.
o) Rather than using pmap_segmap and calculating the offset into the page table
by hand in trap.c, use pmap_pte().
o) Remove unused quad_syscall variable in trap.c.
o) Log things for illegal instructions like we do for bad page faults.
o) Various cast cleanups related to how to print registers.
o) When logging page faults, show the page table information not just for the
program counter, but for the fault address.
o) Modify support.S to use ABI-neutral macros for operating on pointers.
o) Consistently use CALLFRAME_SIZ rather than STAND_FRAME_SIZE, etc.
o) Remove unused insque/remque functions.
o) Remove some coprocessor 0 accessor functions implemented in assembly that
are unused and have inline assembly counterparts.

o) Add NPDEPG, like NPTEPG but for PDEs.
o) Remove NBPG, PGOFSET and PGSHIFT. Use the standard names.
o) Remove some unused macros and move things from param.h to vmparam.h that
belong in the latter. (Actually, all of the kernel segment values, virtual
addresses, etc., belong in one place, but this is a step in the right
direction.)

o) Add SMP support for Octeon using U-Boot to launch all the processors at the
same time.
o) Remove some unused trivial uart functions from octeon_machdep now that the
uart part is fully working and they are unused.
o) Use __func__ instead of __FUNCTION__.
o) Use intr_*() instead of other routines that do the same thing.
o) Remove some duplicate printfs from the Octeon port, as well as duplicate
setting of Maxmem.
o) Use the right frequency divider on Octeon.
o) Use PCPU_GET(cpuid) consistently to get the cpuid of the running core.
o) Remove some unused macros in the Octeon port.
o) Use mips_sync() around use of the global dpcpu, whose value may not be
visible to APs at first.
o) When loading the first thread's stack, use macros to make the code correct
for n64 as well.
o) Remove stub, do-nothing FAU init/enable/disable functions from the RGMX
driver.

o) Back out my previous change to SWARM; some of it was to address an issue
that turned out to be unrelated, and the rest was, as pointed out by Neel,
just wrong-headed.
o) Tweak mem.c to fix use of /dev/kmem for direct-mapped addresses.

o) Use inline functions to access coprocessor 0 registers rather than external
ones implemented using assembly.
o) Use TRAPF_USERMODE() consistently rather than USERMODE(). Eliminate
<machine/psl.h> as a result.
o) Use intr_*() rather than *intr(), consistently.
o) Use register_t instead of u_int in some trap code.
o) Merge some more endian-related macros to machine/asm.h from NetBSD.
o) Add PTR_LI macro, which loads an address with the correct sign-extension for
a pointer.
o) Restore interrupts when bailing out due to an excessive IRQ in
nexus_setup_intr().
o) Remove unused functions from psraccess.S.
o) Enter temporary virtual entries for large memory access into the page tables
rather than simply hoping they stay resident in the TLB and we don't need to
do a refill for them.
o) Abstract out large memory mapping setup/teardown using some macros.
o) Do mips_dcache_wbinv_range() when using temporary virtual addresses just
like we do when we can use the direct map.

o) Remove code related to VM_ALLOC_WIRED_TLB_PG_POOL, VM_KERNEL_ALLOC_OFFSET
and floating pages. They are unused and unsupported.

o) Use the direct map where possible for uiomove_fromphys, based on code from
sparc64.
o) Use uiomove_fromphys rather than the broken fpage mechanism for /dev/mem.
o) Update sf_buf allocator to not share buffers and to do a pmap_qremove when
done with an sf_buf so as to better track valid mappings.

Remove some unused header files.

Destroy the pmap 'pm_mutex' in pmap_release() otherwise we will panic
subsequently in pmap_pinit() with the following signature:

panic: lock "pmap" 0xc7878bc8 already initialized

This bug was uncovered by the changes made to vm_map.c in r206140.

Revert the vm_machdep.c part of r205072.

This causes a panic in vm_thread_dispose() when it tries to add this kstack
to the kstack cache. This happens only when 'td_kstack' is not (PAGE_SIZE * 2)
bytes aligned and we have unmapped the page at that address in cpu_thread_alloc.

Pointed out by: nwhitehorn@

Fix a bug where bus_dma_load_xxx() would not bounce misaligned buffers
due to rounding the buffer's physical address to the beginning of its
page. This fixes a panic in arge(4) when using PPPoE.

Reported by: Jakob van Santen <vansanten at wisc dot edu>
Reviewed by: gonzo
Obtained from: amd64

Replace sb_store64()/sb_load64() with mips3_sd()/mips3_ld() respectively.

Obtained from NetBSD.

Suggested by: jmallett@

Change the arguments of exec_setregs() so that it receives a pointer
to the image_params struct instead of several members of that struct
individually. This makes it easier to expand its arguments in the future
without touching all platforms.

Reviewed by: jhb

Fix periodic "t_delta 16.01359db7eb5eb3c0 too long" messages on the console by
accounting for the "lost time" between when the timer interrupt fired
and when clock_intr() actually started executing.

Sibyte provides a 64-bit read-only counter that counts at half the processor
frequency. This counter can be accessed coherently from both cores.

Use this as the preferred timecounter for the SWARM kernels.

The CP0 COUNT register is unusable as the timecounter on SMP platforms because
the COUNT registers on different CPUs are not guaranteed to be in sync.

Get rid of unused macro MIPS_MEM_RID.

Suggested by: Alexandr Rybalko (ray@dlink.ua)

This change enables use of physical memory that is beyond the direct
mapped kseg0 region.

The basic idea is to use KVA from the kseg2 region for mapping page
table pages that lie beyond the direct mapped region.

The TLB miss handler can now recursively fault into the TLB invalid
handler if it dereferences a kseg2 page table page address that is not
in the TLB.

Tested by: JC (c.jayachandran@gmail.com)

- Enable kernel stack guard page.

- Unmap the unused kernel stack page that we cannot use because it is
not aligned on a (PAGE_SIZE * 2) boundary.

Make the ddb command "show tlb" SMP friendly.

It now accepts an argument to dump out the tlb of a particular cpu.

Stash the context of the running thread at the time an IPI_STOP is received
in 'stoppcbs[]'. We use the 'stoppcbs[]' context to generate the backtrace
of such stopped threads.

o) Consistently use MIPS_KSEGn_TO_PHYS instead of MIPS_{,UN}CACHED_TO_PHYS etc.
Get rid of the macros that spell KSEG0 CACHED and KSEG1 UNCACHED.
o) Get rid of some nearby duplicated and unused macros.

Reviewed by: imp

Add support for CPUs with cache coherent DMA. The two main changes are:

- We don't need to fall back to uncacheable memory to satisfy BUS_DMA_COHERENT
requests on these CPUs.

- The bus_dmamap_sync() is a no-op for these CPUs.

A side-effect of this change is rename DMAMAP_COHERENT flag to
DMAMAP_UNCACHEABLE. This conveys the purpose of the flag more accurately.

Reviewed by: gonzo, imp

Looks like S8 and SP are reversed in setjmp, so longjmp doesn't work
as well as one would hope....

Submitted by: Arten Belevich

Spell START_FRAME CALLFRAME_SIZ now.

Fix DDB backtrace that includes a kernel exception frame.

The backtrace code tries to look for an instruction of the form "sw ra, x(sp)"
to figure out the program counter of the calling function. When we generate
the kernel exception frame we store the 'ra' at the time of the exception
using an instruction of the same form. The problem is that the 'ra' at the
time of the exception is not the same as the 'program counter' at the time
of the exception.

The fix is to save the 'exception program counter' register by staging
it through the 'ra' register.

Use proper structure type for 64-bit headers

# this fixes the MALTA64 build

Kernel module support for mips.

Reviewed by: gonzo

Tested by: Alexandr Rybalko (ray@dlink.ua)

Various fixes to get the SWARM config working on a big-endian Sibyte CPU.

Getting the little-endian PCI bus working on the big-endian CPU proved to be
quite challenging. We let the PCI devices be mapped in the "match byte lanes"
address window. This is where they are mapped by the CFE and DMA transfers
generated to or from addresses within this window are not subject to automatic
byte-swapping.

However any access by the driver to memory-mapped pci space is redirected
via the "match bit lanes" address window. We get the benefit of automatic
byte swapping through this address window and drivers don't need to change
to deal with CPU big-endianness.

Remove the PCI_IOSPACE_SIZE and PCI_IOSPACE_ADDR hack from nexus.c. Implement
this in the Sibyte PCI hostbridge driver instead.

The nexus driver sees resource allocation requests for memory and irq
resources only. These are legitimate resources on all MIPS platforms.

Suggested by: imp

Code cleanup:
- make some variables static
- remove unused variables.

Call profclock() and statclock() explicitly on all cpus. Prior to this
change these functions were called only on the BSP indirectly via hardclock().

top -P now shows usage statistics of all cpus.

Enable interrupts before doing AST processing to avoid a deadlock.

Specifically on an SMP kernel it was observed that if both the
processors are doing an exit1() via ast()->postsig()->sigexit()
then we will deadlock.

This happens because exit1() calls vmspace_exit() that in turn
calls pmap_invalidate_all(). This function tries to do a
smp_rendezvous() which blocks because the other processor is not
responding to IPIs - because it too is doing AST processing with
interrupts disabled.

SMP support for the mips port.

The platform that supports SMP currently is a SWARM with a dual-core Sibyte
processor. The kernel config file to use is SWARM_SMP.

Reviewed by: imp, rrs

Correct a comment - we are not setting the exception level but rather are
disabling interrupts.

Simplify register usage - we can directly load 'curpcb' into 'k1' after
interrupts are disabled. There is no need to do so indirectly through 'a1'.

Get system call tracing using ktrace working for mips.

Clean up all places in exception.S that fiddle with 'pcpup' directly. We now
use the GET_CPU_PCPU() macro exclusively.

This isolates the users of pcpu data from its implementation details.

Reviewed by: imp

Provide access to pcpu structures for SMP kernels.

The basic idea is to use a the same virtual address as a window onto
distinct physical memory locations - one per processor. The physical
address that you access through this mapping depends on which cpu you
are currently executing on. We can now use the same virtual address
on any processor to access its per-cpu area.

The details are:

- The virtual address for 'struct pcpu *pcpup' is obtained by
stealing 2 pages worth of KVA in pmap_bootstrap().

- The mapping from the constant virtual address to a distinct
physical page is done in cpu_pcpu_init() through a wired TLB entry.

- A side-effect of this is that we reserve 2 pages worth of memory
for the pcpu but in reality it needs much less than that. The unused
memory is now used as the boot stack for the BSP and APs.

Remove SMP-specific bits from locore.S. The plan is to use a separate
mpboot.S for AP bootstrap.

Discussed on: freebsd-mips

Approved by: imp (mentor)

Follow Neel's suggestion and switch to using
restoreint() in combination with saving off the
old level. That way we don't blast out the old
level.

For our memory re-mapping trick to work
interrupts must be disabled through the
page_zero's or copys etc. Note that the
temporary mapping used by panic's may
cause us pain since int's may not be disabled.
When we get dumps working we may have to revist
this. Note that with this fix the build got
much much further.. until it hung on disk IO (I
would imagine thats the rge/msgring driver acting
up).

Fix two of the extended memory hacks. The copy pages
routine in one place was setting the valid2 bit to
2 not 1. This meant the PTE was NOT valid and so
you would crash.

In Zero Page there was a incorrect setting of
the valid bit AFTER the actual zero (opps)..

Hopefully this will fix the 0xc0000000 crashes
that I have been seeing (unless of course there are
other problems with these old hacks of mine to get
to memory above 512Meg)

Adds additional hacks for proper bits so that
the RMI/XLR has the COP0 and COP2 bits enabled
Plus it needs SX too. Thanks again for JC in
catching this ;-)

Submitted by: JC (jayachandranc@netlogicmicro.com

Do not leave dirty cache lines behind if bus_dmamap_sync was called
to invalidate memory chunk that starts or ends in the middle of
cache line.

This was responsible for one half of the problem preventing umass
to work reliably on some MIPS32 platforms. USBng needs to stop
sharing cache lines between DMA-able memory and other structures
to cure the other half.

Discussed with: imp, gonzo

Add a DDB command "show trapframe" to dump out contents of the trapframe
specified by the first argument.

Approved by: imp (mentor)

Print the address of the base of the stackframe in DDB backtrace output.

Approved by: imp (mentor)

Fix a problem seen when a new process was returning to userland
through fork_trampoline.

This was caused because we were clearing the SR_INT_IE and setting
SR_EXL bits of the status register at the same time. This meant
that if an interrupt happened while this MTC0 was making its way
through the pipeline the exception processing would see the
status register with SR_EXL bit set. This in turn would mean that
the COP_0_EXC_PC would not be updated so the return from exception
would be to an incorrect address.

It is easy to verify this fix by a program that forks in a loop
and the child just exits:

while (1) {
pid_t pid = vfork();
if (pid == 0)
_exit(0);
if (pid != -1)
waitpid(pid, NULL, 0);
}

Also remove two instances where we set SR_EXL bit gratuitously in exception.S.

Approved by: imp (mentor)

Store the mutex in the correct location. Before, we were storing it
in the pcb at the td_lock offset, rather than in the struct thread at
the td_lock offset. And we were storing a pointer to the old thread
rather than to the mutex. Why this didn't always kill us, I'll never
know.

Fix an obsolete comment and update the prototype in the comments.
Also note what variables are in what registers since this function is
a little complex...

neel@ found this problem and proposed this fix. This cures a number
of different problem reports out there, and gets us booting octeon to
the login prompt...

Submitted by: neel@
Reviewed by: rrs@, gonzo@

Comment out the led wheel code for the moment. Likely it shouldn't
even be here in the first place, but it is cool to see FreeBSD
scrolling on the LED pannel of the octeon board when we're running...

- Introduce kernel_kseg0_end variable that marks first address in KSEG0
available for use. All data below this address considered to be used
by kernel. Along with kernel own data it might be symbol tables
prepeared by trampoline code, boot loader service data passed for
further analysis by kernel, etc... By default kernel_kseg0_end points
to the end of loaded kernel.

- Introduce mips_postboot_fixup function. It checks for symbol information
copied by ELF trampoline and passes it to KDB

- Copy symbol-related tables (.symtab and .strtab) to the end of
relocated kernel. We use magic number to signal kernel that
symbol data is present.

Remove Sibyte specific code from locore.S that sets the k0seg coherency.

Move it to platform_start() instead.

Approved by: imp (mentor)

Create a method of last resort for rebooting the mips processor: jump
to the reset vector. This works for many SoCs where other reset
hardware is either missing or unknown.

Remove duplicate bootverbose increment.

Add a suggested improvement.

Get rid of redundant setting of interrupt enable bit when restoring the status
register from the PCB.

Remove a couple of misleading comments while I am here. The comments are
misleading because they imply that interrupts will be enabled after the
status register is restored from the PCB. This is not the case because
the processor is at the exception level (SR_EXL is set).

Approved by: imp (mentor)

Get rid of unused function MipsTLBInvalidException().

Approved by: imp (mentor)

Make sure that interrupts are enabled when thread0 is running.

Approved by: imp (mentor)

Set the svn:eol-style = native and svn:mime-type = text/plain
properties on all files in this tree.

Submitted by: rpaulo@

Place proper svn:keywords tag on all these files. They were created
somehow without them on projects/mips, and that mistake was propigated
over to head.

Submitted by: rpaulo@

Using svn cp rather than cp to copy these files over had the benefit
of preserving their history. It had the problem that it also copied
over mergeinfo data. Since we're retiring the projects/mips branch,
I'm removing the svn:mergeinfo property from them all.

Submitted by: jhb

Cope with the octeon1->cavium move. Note: none of this junk belongs here.

Merge from projects/mips to head by hand:

sorry for the huge firehose on this commit, it would be too tedious
to do file by file

r201881 | imp | 2010-01-08 20:08:22 -0700 (Fri, 08 Jan 2010) | 3 lines
Rename mips_pcpu_init to mips_pcpu0_init since it applies only to the
BSP. Provide a missing prototype.

r201880 | neel | 2010-01-08 19:17:14 -0700 (Fri, 08 Jan 2010) | 7 lines
Compute the target of the jump in the 'J' and 'JAL' instructions
correctly. The 256MB segment is formed by taking the top 4 bits
of the address of the instruction in the "branch delay" slot
as opposed to the 'J' or 'JAL' instruction itself.

r201845 | imp | 2010-01-08 15:48:21 -0700 (Fri, 08 Jan 2010) | 2 lines
Centralize initialization of pcpu, and set curthread early...

r201770 | neel | 2010-01-07 22:53:11 -0700 (Thu, 07 Jan 2010) | 4 lines
Add a DDB command "show pcb" to dump out the contents of a thread's PCB.

r201631 | neel | 2010-01-05 23:42:08 -0700 (Tue, 05 Jan 2010) | 5 lines
Remove all CFE-specific code from locore.S. The CFE entrypoint initialization
is now done in platform-specific code.

r201563 | neel | 2010-01-04 23:58:54 -0700 (Mon, 04 Jan 2010) | 6 lines
This change increases the size of the kernel stack for thread0 from
PAGE_SIZE to (2 * PAGE_SIZE). It depends on the memory allocated by
pmap_steal_memory() being aligned to a PAGE_SIZE boundary.

r200656 | imp | 2009-12-17 16:55:49 -0700 (Thu, 17 Dec 2009) | 7 lines
Place holder ptrace mips module. Not entirely sure what's required
here yet, so I've not connected it to the build. I think that we'll
need to move something into the processor specific part of the mips
port by requiring mips_cpu_ptrace or platform_cpu_ptrace be provided
by the ports to get/set processor specific registers, ala SSE
registers on x86.

r200342 | imp | 2009-12-09 18:42:44 -0700 (Wed, 09 Dec 2009) | 4 lines
app_descriptor_addr is unused (I know it is referened still). And
unnecessary since we pass in a3 unmodified to platform_start.
Eliminate it from here and kill one more TARGET_OCTEON in the process.

r199760 | imp | 2009-11-24 10:15:22 -0700 (Tue, 24 Nov 2009) | 2 lines
Add in Cavium's CID. Report what the unknown CID is.

r199755 | imp | 2009-11-24 09:53:58 -0700 (Tue, 24 Nov 2009) | 5 lines
looks like there's more to this patch than just this one file. I'll
leave it to neel@ to get all the relevant pieces into the tree.

r199754 | imp | 2009-11-24 09:32:31 -0700 (Tue, 24 Nov 2009) | 6 lines
Include opt_cputype.h for all .c and .S files referencing TARGET_OCTEON.
Spell ld script name right.
# for the most part, we need to enhance infrastructure to obviate the need
# for such an intrusive option.

r199753 | imp | 2009-11-24 09:30:29 -0700 (Tue, 24 Nov 2009) | 3 lines
Remove a comment that's bogus.
Include opt_cputype.h since TARGET_OCTEON moved there.

r199752 | imp | 2009-11-24 09:29:23 -0700 (Tue, 24 Nov 2009) | 4 lines
Make sure kstack0 is page aligned.
# this may have been from neel@ for the sibyte stuff

r199742 | imp | 2009-11-24 01:35:11 -0700 (Tue, 24 Nov 2009) | 8 lines
Move the hard-wiring of the dcache on octeon outside of the if
statement. When no caches support was added, it looks like
TARGET_OCTEON was bogusly moved inside the if. Also, include
opt_cputype.h to make TARGET_OCTEON actually active.

# now we die in pmap init somewhere... Most likely because 32MB of RAM is
# too tight given the load address we're using.

r199741 | imp | 2009-11-24 01:21:48 -0700 (Tue, 24 Nov 2009) | 2 lines
TARGET_OCTEON reqiures opt_cputype.h.

r199736 | imp | 2009-11-24 00:40:38 -0700 (Tue, 24 Nov 2009) | 2 lines
Prefer ANSI spellings of uintXX_t, etc.

r199598 | imp | 2009-11-20 09:30:35 -0700 (Fri, 20 Nov 2009) | 3 lines
Horrible kludge to make octeon32 work. I think a better way is to
move the generic code into the config files....

r199597 | imp | 2009-11-20 09:27:50 -0700 (Fri, 20 Nov 2009) | 4 lines
cast vaddr to uintptr_t before casting it to a bus_space_handle_t.
# I'm sure this indicates a problem, but I'm not sure what...

r199496 | gonzo | 2009-11-18 15:52:05 -0700 (Wed, 18 Nov 2009) | 5 lines
- Add cpu_init_interrupts function that is supposed to
prepeare stuff required for spinning out interrupts later
- Add API for managing intrcnt/intrnames arrays
- Some minor style(9) fixes

r199246 | neel | 2009-11-13 02:24:09 -0700 (Fri, 13 Nov 2009) | 10 lines
Make pmap_copy_page() L2-cache friendly by doing the copy through the
cacheable window on physical memory (KSEG0). On the Sibyte processor
going through the uncacheable window (KSEG1) bypasses both L1 and L2
caches so we may end up with stale contents in the L2 cache.

This also makes it consistent with the rest of the function that
uses cacheable mappings to copy pages.

Approved by: imp (mentor)

r198842 | gonzo | 2009-11-02 23:42:55 -0700 (Mon, 02 Nov 2009) | 3 lines
- Handle errors when adding children to nexus. This sittuation
might occure when there is dublicate of child's entry in hints

r198669 | rrs | 2009-10-30 02:53:11 -0600 (Fri, 30 Oct 2009) | 5 lines
With this commit our friend RMI will now compile. I have
not tested it and the chances of it running yet are about
ZERO.. but it will now compile. The hard part now begins,
making it run ;-)

r198569 | neel | 2009-10-28 23:18:02 -0600 (Wed, 28 Oct 2009) | 5 lines
Deal with overflow of the COUNT register correctly. The 'cycles_per_hz'
has nothing to do with the rollover.

r198550 | imp | 2009-10-28 11:03:20 -0600 (Wed, 28 Oct 2009) | 3 lines
Remove useless for statement. i isn't used after it.
Remove needless braces.

r198534 | gonzo | 2009-10-27 21:34:05 -0600 (Tue, 27 Oct 2009) | 8 lines
- Fix busdma sync: dcache invalidation operates on cache line aligned
addresses and could modify areas of memory that share the same cache
line at the beginning and at the ending of the buffer. In order to
prevent a data loss we save these chunks in temporary buffer before
invalidation and restore them afer it.
Idea suggested by: cognet

r198531 | gonzo | 2009-10-27 18:01:20 -0600 (Tue, 27 Oct 2009) | 3 lines
- Remove bunch of declared but not defined cach-related variables
- Add mips_picache_linesize and mips_pdcache_linesize variables

r198530 | gonzo | 2009-10-27 17:45:48 -0600 (Tue, 27 Oct 2009) | 3 lines
- Replace stubs with actual cache info
- minor style(9) fix

r198355 | neel | 2009-10-21 22:35:32 -0600 (Wed, 21 Oct 2009) | 11 lines
Remove redundant instructions from tlb.S
The "_MTC0 v0, COP_0_TLB_HI" is actually incorrect because v0 has not been
initialized at that point. It worked correctly because we subsequently
did the right thing and initialized TLB_HI correctly.
The "li v0, MIPS_KSEG0_START" is redundant because we do exactly the same
thing 2 instructions down.

r198354 | neel | 2009-10-21 20:51:31 -0600 (Wed, 21 Oct 2009) | 9 lines
Get rid of the hardcoded constants to define cacheable memory:
SDRAM_ADDR_START, SDRAM_ADDR_END and SDRAM_MEM_SIZE
Instead we now keep a copy of the memory regions enumerated by
platform-specific code and use that to determine whether an address
is cacheable or not.

r198310 | gonzo | 2009-10-20 17:13:08 -0600 (Tue, 20 Oct 2009) | 5 lines
- Commit missing part of "bt" fix: store PC register in pcb_context struct
in cpu_switch and use it in stack_trace function later. pcb_regs contains
state of the process stored by exception handler and therefor is not
valid for sleeping processes.

r198264 | neel | 2009-10-19 22:36:08 -0600 (Mon, 19 Oct 2009) | 5 lines
Fix a bug where we would think that the L1 instruction and data cache are
present even though the line size field in the CP0 Config1 register is 0.

r198208 | imp | 2009-10-18 09:21:48 -0600 (Sun, 18 Oct 2009) | 3 lines
Get the PC from the trap frame, since it isn't saved as part of the
pcb regs.

r198205 | imp | 2009-10-18 08:55:55 -0600 (Sun, 18 Oct 2009) | 3 lines
Use correct signature for MipsEmulateBranch. The other one doesn't
work for 64-bit compiles.

r198182 | gonzo | 2009-10-16 18:22:07 -0600 (Fri, 16 Oct 2009) | 11 lines
- Use PC/RA/SP values as arguments for stacktrace_subr instead of trapframe.
Context info could be obtained from other sources (see below) no only from
td_pcb field
- Do not show a0..a3 values unless they're obtained from the stack. These
are only confirmed values.
- Fix bt command in DDB. Previous implementation used thread's trapframe
structure as a source info for trace unwinding, but this structure
is filled only when exception occurs. Valid register values for sleeping
processes are in pcb_context array. For curthread use pc/sp/ra for current
frame

r198181 | gonzo | 2009-10-16 16:52:18 -0600 (Fri, 16 Oct 2009) | 2 lines
- Get rid of label_t. It came from NetBSD and was used only in one place

r198066 | gonzo | 2009-10-13 19:43:53 -0600 (Tue, 13 Oct 2009) | 5 lines
- Move stack tracing function to db_trace.c
- Axe unused extern MipsXXX declarations
- Move all declarations for functions in exceptions.S/swtch.S
from trap.c to respective headers

r197796 | gonzo | 2009-10-05 17:19:51 -0600 (Mon, 05 Oct 2009) | 2 lines
- Revert part of r197685 because this change leads to wrong data in cache.

r197685 | gonzo | 2009-10-01 14:05:36 -0600 (Thu, 01 Oct 2009) | 2 lines
- Sync caches properly when dealing with sf_buf

r197014 | imp | 2009-09-08 21:57:10 -0600 (Tue, 08 Sep 2009) | 2 lines
Ugly hack to get this to compile. I'm sure there's a better way...

r197013 | imp | 2009-09-08 21:54:55 -0600 (Tue, 08 Sep 2009) | 2 lines
First half of making this 64-bit clean: fix prototypes.

r196988 | gonzo | 2009-09-08 13:15:29 -0600 (Tue, 08 Sep 2009) | 2 lines
- MFC from head@196987

r196313 | imp | 2009-08-17 06:14:40 -0600 (Mon, 17 Aug 2009) | 2 lines
suword64 and csuword64. Needed by ELF64 stuff...

r196266 | imp | 2009-08-15 16:51:11 -0600 (Sat, 15 Aug 2009) | 5 lines
(1) Fix a few 32/64-bit bugs.
(2) Also, always allocate 2 pages for the stack to optimize TLB usage.
Submitted by: neel@ (2)

r196265 | imp | 2009-08-15 16:48:09 -0600 (Sat, 15 Aug 2009) | 2 lines
Various 32/64-bit confusion cleanups.

r196264 | imp | 2009-08-15 16:45:46 -0600 (Sat, 15 Aug 2009) | 6 lines
(1) Some CPUs have a range to map I/O cyces on the pci bus. So allow
them to work by allowding the nexus to assign ports.
(2) Remove some Octeon junk that shouldn't be necessary.
Submitted by: neel@ (#1) for SB1 port.

r196061 | gonzo | 2009-08-04 11:32:55 -0600 (Tue, 04 Aug 2009) | 2 lines
- Use register_t for registers values

r195984 | gonzo | 2009-07-30 17:48:29 -0600 (Thu, 30 Jul 2009) | 4 lines
- Properly unwind stack for functions with __noreturn__ attribute
Submitted by: Neelkanth Natu <neelnatu@yahoo.com>

r195983 | gonzo | 2009-07-30 17:29:59 -0600 (Thu, 30 Jul 2009) | 4 lines
- mark map as coherent if requested by flags
- explicitly set memory allocation method in map flags instead
of duplicating conditions for malloc/contigalloc

r195584 | imp | 2009-07-10 13:09:34 -0600 (Fri, 10 Jul 2009) | 3 lines
Use PTR_* macros for pointers, and not potentially mips64 unsafe
operations.

r195583 | imp | 2009-07-10 13:08:48 -0600 (Fri, 10 Jul 2009) | 2 lines
Use PTR_* macros to deal with pointers.

r195579 | imp | 2009-07-10 13:04:32 -0600 (Fri, 10 Jul 2009) | 2 lines
use ta0-ta3 rather than t4-t7 for n32/n64 goodness.

r195511 | gonzo | 2009-07-09 13:02:17 -0600 (Thu, 09 Jul 2009) | 3 lines
- Ooops, this debug code wasn't supposed to get into
final commit. My appologises.

r195478 | gonzo | 2009-07-08 16:28:36 -0600 (Wed, 08 Jul 2009) | 5 lines
- Port busdma code from FreeBSD/arm. This is more mature version
that takes into account all limitation to DMA memory (boundaries,
alignment) and implements bounce pages.
- Add BUS_DMASYNC_POSTREAD case to bus_dmamap_sync_buf

r195438 | imp | 2009-07-08 00:00:18 -0600 (Wed, 08 Jul 2009) | 2 lines
Turns out this code was right, revert last change.

r195429 | gonzo | 2009-07-07 13:55:09 -0600 (Tue, 07 Jul 2009) | 5 lines
- Move dpcpu initialization to mips_proc0_init. It's
more appropriate place for it. Besides dpcpu_init
requires pmap module to be initialized and calling it
int pmap.c hangs the system

r195399 | imp | 2009-07-06 01:49:24 -0600 (Mon, 06 Jul 2009) | 2 lines
Prefer uintptr_t to int cast here.

r195398 | imp | 2009-07-06 01:48:31 -0600 (Mon, 06 Jul 2009) | 3 lines
Better types for 64-bit compatibility. Use %p and cast to void * and
prefer uintptr_t to other int-type casts.

r195397 | imp | 2009-07-06 01:47:39 -0600 (Mon, 06 Jul 2009) | 2 lines
No need to force mips32 here.

r195396 | imp | 2009-07-06 01:46:13 -0600 (Mon, 06 Jul 2009) | 3 lines
Pass in the uint64 value, rather than a pointer to it. that's what
the function expects...

r195395 | imp | 2009-07-06 01:45:02 -0600 (Mon, 06 Jul 2009) | 3 lines
Use ta0 instead of t4 and ta1 instead of t5. These map to the same
registers on O32 builds, but t4 and t5 don't exist on N32 or N64.

r195394 | imp | 2009-07-06 01:43:50 -0600 (Mon, 06 Jul 2009) | 3 lines
Use better casts for passing the small integer as a pointer here.
Basically, replace int with uintptr_t.

r195393 | imp | 2009-07-06 01:42:54 -0600 (Mon, 06 Jul 2009) | 5 lines
(1) Improvements for SB1. only allow real memory to be accessed.
(2) make compile n64 by using more-proper casts.
Submitted by: Neelkanth Natu (1)

r195373 | imp | 2009-07-05 09:23:54 -0600 (Sun, 05 Jul 2009) | 5 lines
(1) Use PTR_LA rather than bare la for N64 goodness (it is dla there)
(2) SB1 needs COHERENT policy, not cached for the config register
Submitted by: (2) Neelkanth Natu

r195372 | imp | 2009-07-05 09:22:22 -0600 (Sun, 05 Jul 2009) | 3 lines
use "PTR_LA" in preference to a bare la so it translates to dla on
64-bit ABIs.

r195371 | imp | 2009-07-05 09:21:35 -0600 (Sun, 05 Jul 2009) | 6 lines
Now that we define atomic_{load,store}_64 inline in atomic.h, we don't
need to define them here for the !N64 case.
We now define atomic_readandclear_64 in atomic.h, so no need to repeat
it here.

r195364 | imp | 2009-07-05 09:10:07 -0600 (Sun, 05 Jul 2009) | 5 lines
use %p in preference to 0x%08x for printing register_t values. Cast
them to void * first. This neatly solves the "how do I print a
register_t" problem because sizeof(void *) is always the same as
sizeof(register_t), afaik.

r195353 | imp | 2009-07-05 00:46:54 -0600 (Sun, 05 Jul 2009) | 6 lines
Publish PAGE_SHIFT to assembler
# we should likely phase out PGSHIFT
Submitted by: Neelkanth Natu

r195350 | imp | 2009-07-05 00:39:37 -0600 (Sun, 05 Jul 2009) | 7 lines
Switch to ABI agnostic ta0-ta3. Provide defs for this in the right
places. Provide n32/n64 register name defintions. This should have
no effect for the O32 builds that everybody else uses, but should help
make N64 builds possible (lots of other changes are needed for that).
Obtained from: NetBSD (for the regdef.h changes)

r195334 | imp | 2009-07-03 21:22:34 -0600 (Fri, 03 Jul 2009) | 6 lines
Move from using the lame invalid address I chose when trying to get
Octeon going... Turns out that you get tlb shutdowns with this...
Use PGSHIFT instead of PAGE_SHIFT.
Submitted by: Neelkanth Natu

r195147 | gonzo | 2009-06-28 15:01:00 -0600 (Sun, 28 Jun 2009) | 2 lines
- Replace casuword and casuword32 stubs with proper implementation

r195128 | gonzo | 2009-06-27 17:27:41 -0600 (Sat, 27 Jun 2009) | 4 lines
- Add support for handling TLS area address in kernel space.
From the userland point of view get/set operations are
performed using sysarch(2) call.

r195127 | gonzo | 2009-06-27 17:01:35 -0600 (Sat, 27 Jun 2009) | 4 lines
- Make cpu_set_upcall_kse conform MIPS ABI. T9 should be
the same as PC in subroutine entry point
- Preserve interrupt mask

r194938 | gonzo | 2009-06-24 20:15:04 -0600 (Wed, 24 Jun 2009) | 3 lines
- Invalidate cache in pmap_qenter. Fixes corruption of data
that comes through pipe (may be other bugs)

r194505 | gonzo | 2009-06-19 13:02:40 -0600 (Fri, 19 Jun 2009) | 5 lines
- Keep interrupts mask intact by RESTORE_CPU in MipsKernGenException
trap() function re-enables interrupts if exception happened with
interrupts enabled and therefor status register might be modified
by interrupt filters

r194277 | gonzo | 2009-06-15 20:36:21 -0600 (Mon, 15 Jun 2009) | 2 lines
- Remove debug printfs

r194275 | gonzo | 2009-06-15 19:43:33 -0600 (Mon, 15 Jun 2009) | 2 lines
- Handle KSEG0/KSEG1 addresses for /dev/mem as well. netstat requires it

r193491 | gonzo | 2009-06-05 03:21:03 -0600 (Fri, 05 Jun 2009) | 6 lines
- Status register should be set last in RESTORE_CPU in order
to prevent race over k0, k1 registers.
- Update interrupts mask in saved status register for
MipsUserIntr and MipsUserGenException. It might be
modified by intr filter or ithread.

r192864 | gonzo | 2009-05-26 16:40:12 -0600 (Tue, 26 May 2009) | 4 lines
- Replace CPU_NOFPU and SOFTFLOAT options with CPU_FPU. By default
we assume that there is no FPU, because majority of SoC does
not have it.

r192794 | gonzo | 2009-05-26 00:20:50 -0600 (Tue, 26 May 2009) | 5 lines
- Preserve INT_MASK fields in Status register across
context switches. They should be modified only by
interrupt setup/teardown and pre_ithread/post_ithread
functions

r192793 | gonzo | 2009-05-26 00:02:38 -0600 (Tue, 26 May 2009) | 2 lines
- Remove erroneus "break" instruction, it was meant for debug

r192792 | gonzo | 2009-05-26 00:01:17 -0600 (Tue, 26 May 2009) | 2 lines
- Remove now unused NetBSDism intr.h

r192791 | gonzo | 2009-05-25 23:59:05 -0600 (Mon, 25 May 2009) | 7 lines
- Provide proper pre_ithread/post_ithread functions for both
hard and soft interrupts
- Do not handle masked interrupts
- Do not write Cause register because most bytes are read-only and
writing the same byte to RW fields are pointless. And in case of
software interrupt utterly wrong

r192664 | gonzo | 2009-05-23 13:42:23 -0600 (Sat, 23 May 2009) | 4 lines
- cpu_establish_hardintr modifies INT_MASK of Status
register, so we should use disableintr/restoreintr that
modifies only IE bit.

r192655 | gonzo | 2009-05-23 12:00:20 -0600 (Sat, 23 May 2009) | 6 lines
- Remove stale comments
- Replace a1 with k1 to while restoring context. a1 was there by mistake,
interrupts are disabled at this point and it's safe to use k0, k1.
This code never was reached beacasue current Status register handling
prevented interrupta from user mode.

r192496 | gonzo | 2009-05-20 17:07:10 -0600 (Wed, 20 May 2009) | 4 lines
- Invalidate caches for respective areain KSEG0 in order
to prevent further overwriting of KSEG1 data with
writeback.

r192364 | gonzo | 2009-05-18 20:43:21 -0600 (Mon, 18 May 2009) | 6 lines
- Cleanup ticker initialization code. For some MIPS cpu Counter
register increments only every second cycle. The only timing
references for us is Count value. Therefore it's better to convert
frequencies related to it and use them. Besides cleanup this commit
fixes twice more then requested sleep interval problem.

r192176 | gonzo | 2009-05-15 20:34:03 -0600 (Fri, 15 May 2009) | 3 lines
- Add informational title for cache info lines to separate
them from environment variables dump

r192119 | gonzo | 2009-05-14 15:26:07 -0600 (Thu, 14 May 2009) | 3 lines
- Off by one check fix. Check for last address in region
to fit in KSEG1

r191841 | gonzo | 2009-05-05 20:55:43 -0600 (Tue, 05 May 2009) | 5 lines
- Use index ops in order to avoid TLBMiss exceptions when flushing caches
on mapping removal
- Writeback all VA for page that is being copied in pmap_copy_page to
guaranty up-to-date data in SDRAM

r191613 | gonzo | 2009-04-27 20:59:18 -0600 (Mon, 27 Apr 2009) | 4 lines
- When destroying va -> pa mapping writeback all caches or we may endup
with partial page content in SDRAM
- style(9) fix

r191583 | gonzo | 2009-04-27 12:46:57 -0600 (Mon, 27 Apr 2009) | 5 lines
- Use new spacebus
- Be a bit more verbose on failures
- style(9) fixes
- Use default rid value of 0 instead of MIPS_MEM_RID (0x20)

r191577 | gonzo | 2009-04-27 12:29:59 -0600 (Mon, 27 Apr 2009) | 4 lines
- Use naming convention the same as MIPS spec does: eliminate _sel1 sufix
and just use selector number. e.g. mips_rd_config_sel1 -> mips_rd_config1
- Add WatchHi/WatchLo accessors for selctors 1..3 (for debug purposes)

r191453 | gonzo | 2009-04-23 23:28:44 -0600 (Thu, 23 Apr 2009) | 4 lines
Fix cut'n'paste code. cfg3 should get the value of selector 3
Spotted by: thompa@

r191452 | gonzo | 2009-04-23 22:18:16 -0600 (Thu, 23 Apr 2009) | 2 lines
- Print supported CPU capabilities during stratup

r191448 | gonzo | 2009-04-23 21:38:51 -0600 (Thu, 23 Apr 2009) | 2 lines
- Fix whitespace to conform style(9)

r191282 | gonzo | 2009-04-19 16:02:14 -0600 (Sun, 19 Apr 2009) | 3 lines
- Make mips_bus_space_generic be of type bus_space_tag_t instead of
struct bus_space and update all relevant places.

r191084 | gonzo | 2009-04-14 20:28:26 -0600 (Tue, 14 Apr 2009) | 6 lines
Use FreeBSD/arm approach for handling bus space access: space tag is a pointer
to bus_space structure that defines access methods and hence every bus can
define own accessors. Default space is mips_bus_space_generic. It's a simple
interface to physical memory, values are read with regard to host system
byte order.

r191083 | gonzo | 2009-04-14 19:47:52 -0600 (Tue, 14 Apr 2009) | 4 lines
- Cleanout stale #ifdef'ed chunk of code
- Fix whitespaces
- Explicitly undefine NEXUS_DEBUG flag

r191079 | gonzo | 2009-04-14 16:53:22 -0600 (Tue, 14 Apr 2009) | 2 lines
- Revert changes accidentally killed by merge operation

------------------------------------------------------------------------
r187512 | gonzo | 2009-01-20 22:49:30 -0700 (Tue, 20 Jan 2009) | 4 lines
- Check if maddr/msize hints are there before setting hinted
resources to device
- Check for irq hint too

r187418 | gonzo | 2009-01-18 19:37:10 -0700 (Sun, 18 Jan 2009) | 4 lines
- Add trampoline stuff for bootloaders that do not support ELF
- Replace arm'ish KERNPHYSADDR/KERNVIRTADDR with
KERNLOADADDR/TRAMPLOADADDR and clean configs

Merge from projects/mips to head by hand:

copy over the generic bus space implementation...

Merge from projects/mips to head by hand:

Copy over elf_trampoline.c for crunched kernel support.

Merge from projects/mips to head by hand:

Copy over sys_machdep.c

Merge from projects/mips to head by hand:

Copy over inckern.S for crunched kernel support.

Merge from projects/mips to head by hand:

Placeholder for ptrace machine specific stuff... This will likely be
used to report CP2 or CP3 registers on those CPUs that have them...

Remove extraneous semicolons, no functional changes.

Submitted by: Marc Balmer <marc@msys.ch>
MFC after: 1 week

Update d_mmap() to accept vm_ooffset_t and vm_memattr_t.

This replaces d_mmap() with the d_mmap2() implementation and also
changes the type of offset to vm_ooffset_t.

Purge d_mmap2().

All driver modules will need to be rebuilt since D_VERSION is also
bumped.

Reviewed by: jhb@
MFC after: Not in this lifetime...

Simplify the invocation of vm_fault(). Specifically, eliminate the flag
VM_FAULT_DIRTY. The information provided by this flag can be trivially
inferred by vm_fault().

Discussed with: kib

Extract the code that records syscall results in the frame into MD
function cpu_set_syscall_retval().

Suggested by: marcel
Reviewed by: marcel, davidxu
PowerPC, ARM, ia64 changes: marcel
Sparc64 tested and reviewed by: marius, also sunv reviewed
MIPS tested by: gonzo
MFC after: 1 month

Unbreak booting of FreeBSD/mips by merging r195429 from projects/mips:
- Move dpcpu initialization to mips_proc0_init. It's
more appropriate place for it. Besides dpcpu_init
requires pmap module to be initialized and calling it
int pmap.c hangs the system

In r197963, a race with thread being selected for signal delivery
while in kernel mode, and later changing signal mask to block the
signal, was fixed for sigprocmask(2) and ptread_exit(3). The same race
exists for sigreturn(2), setcontext(2) and swapcontext(2) syscalls.

Use kern_sigprocmask() instead of direct manipulation of td_sigmask to
reschedule newly blocked signals, closing the race.

Reviewed by: davidxu
Tested by: pho
MFC after: 1 month

o Introduce vm_sync_icache() for making the I-cache coherent with
the memory or D-cache, depending on the semantics of the platform.
vm_sync_icache() is basically a wrapper around pmap_sync_icache(),
that translates the vm_map_t argumument to pmap_t.
o Introduce pmap_sync_icache() to all PMAP implementation. For powerpc
it replaces the pmap_page_executable() function, added to solve
the I-cache problem in uiomove_fromphys().
o In proc_rwmem() call vm_sync_icache() when writing to a page that
has execute permissions. This assures that when breakpoints are
written, the I-cache will be coherent and the process will actually
hit the breakpoint.
o This also fixes the Book-E PMAP implementation that was missing
necessary locking while trying to deal with the I-cache coherency
in pmap_enter() (read: mmu_booke_enter_locked).

The key property of this change is that the I-cache is made coherent
*after* writes have been done. Doing it in the PMAP layer when adding
or changing a mapping means that the I-cache is made coherent *before*
any writes happen. The difference is key when the I-cache prefetches.

Make sure that the primary native brandinfo always gets added
first and the native ia32 compat as middle (before other things).
o(ld)brandinfo as well as third party like linux, kfreebsd, etc.
stays on SI_ORDER_ANY coming last.

The reason for this is only to make sure that even in case we would
overflow the MAX_BRANDS sized array, the native FreeBSD brandinfo
would still be there and the system would be operational.

Reviewed by: kib
MFC after: 1 month

* Completely Remove the option STOP_NMI from the kernel. This option
has proven to have a good effect when entering KDB by using a NMI,
but it completely violates all the good rules about interrupts
disabled while holding a spinlock in other occasions. This can be the
cause of deadlocks on events where a normal IPI_STOP is expected.
* Adds an new IPI called IPI_STOP_HARD on all the supported architectures.
This IPI is responsible for sending a stop message among CPUs using a
privileged channel when disponible. In other cases it just does match a
normal IPI_STOP.
Right now the IPI_STOP_HARD functionality uses a NMI on ia32 and amd64
architectures, while on the other has a normal IPI_STOP effect. It is
responsibility of maintainers to eventually implement an hard stop
when necessary and possible.
* Use the new IPI facility in order to implement a new userend SMP kernel
function called stop_cpus_hard(). That is specular to stop_cpu() but
it does use the privileged channel for the stopping facility.
* Let KDB use the newly introduced function stop_cpus_hard() and leave
stop_cpus() for all the other cases
* Disable interrupts on CPU0 when starting the process of APs suspension.
* Style cleanup and comments adding

This patch should fix the reboot/shutdown deadlocks many users are
constantly reporting on mailing lists.

Please don't forget to update your config file with the STOP_NMI
option removal

Reviewed by: jhb
Tested by: pho, bz, rink
Approved by: re (kib)

Add a new type of VM object: OBJT_SG. An OBJT_SG object is very similar to
a device pager (OBJT_DEVICE) object in that it uses fictitious pages to
provide aliases to other memory addresses. The primary difference is that
it uses an sglist(9) to determine the physical addresses for a given offset
into the object instead of invoking the d_mmap() method in a device driver.

Reviewed by: alc
Approved by: re (kensmith)
MFC after: 2 weeks

Fix copyrights to reflect the origin of these files.

Approved by: re@ (rwatson)

Implement a facility for dynamic per-cpu variables.
- Modules and kernel code alike may use DPCPU_DEFINE(),
DPCPU_GET(), DPCPU_SET(), etc. akin to the statically defined
PCPU_*. Requires only one extra instruction more than PCPU_* and is
virtually the same as __thread for builtin and much faster for shared
objects. DPCPU variables can be initialized when defined.
- Modules are supported by relocating the module's per-cpu linker set
over space reserved in the kernel. Modules may fail to load if there
is insufficient space available.
- Track space available for modules with a one-off extent allocator.
Free may block for memory to allocate space for an extent.

Reviewed by: jhb, rwatson, kan, sam, grehan, marius, marcel, stas

Fix MALTA build; some prototypes were wrong and blew up when kobj method
signature checking was turned on.

pmap_enter() *must* set PG_WRITEABLE on the given page if it creates a
mapping that permits write access. Otherwise, pmap_remove_write() will not
remove write access from any of the page's mappings.

Give pmap_enter()'s third parameter the same name that it has on amd64 and
i386. Otherwise, my next to last commit (r192628) to this file doesn't
actually compile.

When a page is mapped for write access on a read fault, the PTE should be
configured to trap on a write access unless *all* of the page's dirty bits
are set.

Preset the modified bit in the PTE when pmap_enter() is called during a
write fault or while wiring a mapping that must support write access.

In general, this change should reduce the number of traps that occur for
the purpose of setting the modified bit. More specifically, this change
should prevent traps while holding locks in a sysctl handler. See
kern/kern_sysctl.c revisions 1.168 and 1.195 (svn r192160) for further
details.

Tested by: gonzo

Add cpu_flush_dcache() for use after non-DMA based I/O so that a
possible future I-cache coherency operation can succeed. On ARM
for example the L1 cache can be (is) virtually mapped, which
means that any I/O that uses temporary mappings will not see the
I-cache made coherent. On ia64 a similar behaviour has been
observed. By flushing the D-cache, execution of binaries backed
by md(4) and/or NFS work reliably.
For Book-E (powerpc), execution over NFS exhibits SIGILL once in
a while as well, though cpu_flush_dcache() hasn't been implemented
yet.

Doing an explicit D-cache flush as part of the non-DMA based I/O
read operation eliminates the need to do it as part of the
I-cache coherency operation itself and as such avoids pessimizing
the DMA-based I/O read operations for which D-cache are already
flushed/invalidated. It also allows future optimizations whereby
the bcopy() followed by the D-cache flush can be integrated in a
single operation, which could be implemented using on-chips DMA
engines, by-passing the D-cache altogether.

- Fix spelling.

A variety of changes:

Reimplement "kernel_pmap" in the standard way.

Eliminate unused variables. (These are mostly variables that were
discarded by the machine-independent layer after FreeBSD 4.x.)

Properly handle a vm_page_alloc() failure in pmap_init().

Eliminate dead or legacy (FreeBSD 4.x) code.

Eliminate unnecessary page queues locking.

Eliminate some excess white space.

Correct the synchronization of pmap_page_exists_quick().

Tested by: gonzo

MFamd64/i386
Introduce pmap_try_insert_pv_entry(), a function that conditionally
creates a pv entry if the number of entries is below the high water mark
for pv entries.

Introduce pmap_enter_quick_locked() and use it to reimplement
pmap_enter_object(). The old implementation was broken. For example,
it could block while holding a mutex lock.

Change pmap_enter_quick_locked() to fail rather than wait if it is
unable to allocate a page table page. This prevents a race between
pmap_enter_object() and the page daemon. Specifically, an inactive
page that is a successor to the page that was given to
pmap_enter_quick_locked() might become a cache page while
pmap_enter_quick_locked() waits and later pmap_enter_object() maps
the cache page violating the invariant that cache pages are never
mapped. Similarly, change
pmap_enter_quick_locked() to call pmap_try_insert_pv_entry() rather
than pmap_insert_entry(). Generally speaking,
pmap_enter_quick_locked() is used to create speculative mappings. So,
it should not try hard to allocate memory if free memory is scarce.

Tested by: gonzo

Fix KBI breakage by r190520 which affects older linux.ko binaries:

1) Move the new field (brand_note) to the end of the Brandinfo structure.
2) Add a new flag BI_BRAND_NOTE that indicates that the brand_note pointer
is valid.
3) Use the brand_note field if the flag BI_BRAND_NOTE is set and as old
modules won't have the flag set, so the new field brand_note would be
ignored.

Suggested by: jhb
Reviewed by: jhb
Approved by: kib (mentor)
MFC after: 6 days

Remove some pointless mergeinfo that is the result of doing a local
'svn cp' and having svn create empty mergeinfo for the file.

Implement an ipfw action to reassemble ip packets: reass.

Integrate user/mav/ata branch:

Add ch_suspend/ch_resume methods for PCI controllers and implement them
for AHCI. Refactor AHCI channel initialization according to it.

Fix Port Multipliers operation. It is far from perfect yet, but works now.
Tested with JMicron JMB363 AHCI + SiI 3726 PMP pair.
Previous version was also tested with SiI 4726 PMP.

Hardware sponsored by: Vitsch Electronics / VEHosting.nl

Implement new way of branding ELF binaries by looking to a
".note.ABI-tag" section.

The search order of a brand is changed, now first of all the
".note.ABI-tag" is looked through.

Move code which fetch osreldate for ELF binary to check_note() handler.

PR: 118473
Approved by: kib (mentor)

make loop clearer that it isn't a mistake...

Rewrite get_pv_entry() to match expectations of the rest of the
kernel. Rather than just kick off the page daemon, we actively retire
more mappings. The inner loop now looks a lot like the inner loop of
pmap_remove_all.

Also, get_pv_entry can't return NULL now, so remove panic if it did.

Reviewed by: alc@

pmap_kenrel() was recently deleted from pmap.h. It was still used
here, but inconsistently. Change all instances of it to kernel_pmap
to match rest of code.

- Fix in_cksum for big-endian MIPS: use correct compile-time check.

Retire NO_DMA completely.

Remove unused variable.
Minor style nits.

- pmap_track_modified was retired in r178606. Reintroducing it was a mistake.

Spotted by: alc@

Reduce diffs to p4 that were the result of a mismerge on my part.

MFp4:

- Add debug output
- Fix pmap_zero_page and related places: use uncached segments and invalidate
cache after zeroing memory.
- Do not test for modified bit if it's not neccessary
(merged from mips-juniper p4 branch)
- Some #includes reorganization

MFp4:

Remove Maxmem. It isn't used elsewhere in the system at this point...
realmem is used instead.

Call platform_reset() instead of looping forever on reboot.
# We likely need to have a default one of these that jumps to the rom boot
# address that's defined in the MIPS ISA.

Reverse order of dumpsys and cpu_idle_wakeup to reduce diffs to p4.

MFp4:

Remove #if'd 0 code. It is interfering with other diffs.

o Make debug output conditional

o NO_DMA is no longer required as bug with PIIX4 emulation has been fixed in GXEmul

o Move $FreeBSD$ from comment to __FBSDID macro
o Be a bit more verbose about CPU type during boot process (print
manufacturer, chip info, MMU and cache parameters)

Restore the if_*var.h and if_*reg.h to their original names, they dont need to
be different.

First pass at 64-bit elf support

Mips never had /usr/libexec/ld-elf.so.1, so remove it. We don't have
any binaries that need it. This was moved before FreeBSD 5.x.

On FreeBSD, printing the version is centralized, so no need to repeat
it here. I think that gonzo@ has also made this change to p4.

Add sv_flags field to struct sysentvec with intention to provide description
of the ABI of the currently executing image. Change some places to test
the flags instead of explicit comparing with address of known sysentvec
structures to determine ABI features.

Discussed with: dchagin, imp, jhb, peter

opt_msgbuf.h is needed for MSGBUF_SIZE overrides, if any.

Submitted by: alc@

Remove critical_enter() and critical_exit(): pmap_growkernel is called
with kernel_map->system_mtx held so these aren't needed. Add an
assertion to make sure this is the case.

Also, fix a minor style(9) nit.

Reviewed by: alc@

Remove ipi_all() and ipi_self() as the former hasn't been used at
all to date and the latter also is only used in ia64 and powerpc
code which no longer serves a real purpose after bring-up and just
can be removed as well. Note that architectures like sun4u also
provide no means of implementing IPI'ing a CPU itself natively
in the first place.

Suggested by: jhb
Reviewed by: arch, grehan, jhb

Fix comments to relate to FreeBSD's current interrupt blocking
functionality, as opposed to the legacy spl means.

Remove stale comment. splhigh -> critical_enter transition happend
some time ago, and no longer needs special attention.

Replace all calls to minor() with dev2unit().

After I removed all the unit2minor()/minor2unit() calls from the kernel
yesterday, I realised calling minor() everywhere is quite confusing.
Character devices now only have the ability to store a unit number, not
a minor number. Remove the confusion by using dev2unit() everywhere.

This commit could also be considered as a bug fix. A lot of drivers call
minor(), while they should actually be calling dev2unit(). In -CURRENT
this isn't a problem, but it turns out we never had any problem reports
related to that issue in the past. I suspect not many people connect
more than 256 pieces of the same hardware.

Reviewed by: kib

Change the static struct sysentvec and struct Elf_Brandinfo initializers
to the C99 style. At least, it is easier to read sysent definitions
that way, and search for the actual instances of sigcode etc.

Explicitely initialize sysentvec.sv_maxssiz that was missed in most
sysvecs.

No objection from: jhb
MFC after: 1 month

The kernel implemented 'memcmp' is an alias for 'bcmp'. However, memcmp
and bcmp are not the same thing. 'man bcmp' states that the return is
"non-zero" if the two byte strings are not identical. Where as,
'man memcmp' states that the return is the "difference between the
first two differing bytes (treated as unsigned char values" if the
two byte strings are not identical.

So provide a proper memcmp(9), but it is a C implementation not a tuned
assembly implementation. Therefore bcmp(9) should be preferred over memcmp(9).

MFp4: Merge interrupt code from p4

Submitted by: gonzo@

Catch up with intr_event_create() API changes.

style(9)

Style(9).

White space nit.

Document a few sysctls.

Approved by: imp

In FreeBSD 7.0 and beyond, pmap_growkernel() should pass VM_ALLOC_INTERRUPT
to vm_page_alloc() instead of VM_ALLOC_SYSTEM.

Fix spelling of "virtual".

There should be no visible change.

Reviewed by: rink

Retire pmap_addr_hint(). It is no longer used.

Add a stub for pmap_align_superpage() on machines that don't (yet)
implement pmap-level support for superpages.

These files are unused, so remove them for now. If they turn out to
be needed later, they can be restored.

Remove an XXX comment. sys/amd64/amd64/pmap.c revision 1.516 explains why
"pmap_initialized" no longer exists.

MFamd64 revision 1.544/i386 revision 1.549
Retire pmap_track_modified(). We no longer need it because we do not
create managed mappings within the clean submap. To prevent regressions,
add assertions blocking the creation of managed mappings within the clean
submap.

Approved by: imp

- Add an integer argument to idle to indicate how likely we are to wake
from idle over the next tick.
- Add a new MD routine, cpu_wake_idle() to wakeup idle threads who are
suspended in cpu specific states. This function can fail and cause the
scheduler to fall back to another mechanism (ipi).
- Implement support for mwait in cpu_idle() on i386/amd64 machines that
support it. mwait is a higher performance way to synchronize cpus
as compared to hlt & ipis.
- Allow selecting the idle routine by name via sysctl machdep.idle. This
replaces machdep.cpu_idle_hlt. Only idle routines supported by the
current machine are permitted.

Sponsored by: Nokia

This file shows signs of heavy borrowing from NetBSD's cache.c file,
so credit its authors with contributions to this file. Remove
prototype copyright notice, although one might be warranted if someone
wanted to claim it badly enough.

Noticed by: Simon Burge.

Add copyright notice and license to reflect origin of most of the
routines in this file. Remove 'place holder' copyright since the
amount that's actually original is small relative to the length of the
file. The contents of this file appear to have originated at DECWRL
by way of NetBSD.

Noticed by: Simon Burge

This isn't used, so remove it. It isn't relevant to most mips platforms.

FreeBSD/mips port. The FreeBSD/mips port targets mips32, mips64,
mips32r2 and mips64r2 (and close relatives) processors. There
presently is support for ADMtek ADM5120, A mips 4Kc in a malta board,
the RB533 routerboard (based on IDT RC32434) and some preliminary
support for sibtye/broadcom designs. Other hardware support will be
forthcomcing.

This port boots multiuser under gxemul emulating the malta board and
also bootstraps on the hardware whose support is forthcoming...

Oleksandr Tymoshenko, Wojciech Koszek, Warner Losh, Olivier Houchard,
Randall Stewert and others that have contributed to the mips2 and/or
mips2-jnpr perforce branches. Juniper contirbuted a generic mips port
late in the life cycle of the misp2 branch. Warner Losh merged the
mips2 and Juniper code bases, and others list above have worked for
the past several months to get to multiuser.

In addition, the mips2 work owe a debt to the trail blazing efforts of
the original mips branch in perforce done by Juli Mallett.