MFC r321686 and r330361:
Add inline functions to convert between sbintime_t and decimal time units.
Use them in some existing code that is vulnerable to roundoff errors.

MFC r333667: followup to r332730/r332752: set kdb_why to "trap" for fatal traps

This change updates arm, arm64 and mips achitectures. Additionally, it
removes redundant checks for kdb_active where it already results in
kdb_reenter() and adds kdb_reenter() calls where they were missing.

Some architectures check the return value of kdb_trap(), but some don't.
I haven't changed any of that.

Some trap handling routines have a return code. I am not sure if I
provided correct ones for returns after kdb_reenter(). kdb_reenter
should never return unless kdb_jmpbufp is NULL for some reason.

MFC r318353:
Avoid use of contiguous memory allocations in busdma when possible.

This patch improves the boundary checks in busdma to allow more cases
using the regular page based kernel memory allocator. Especially in
the case of having a non-zero boundary in the parent DMA tag. For
example AMD64 based platforms set the PCI DMA tag boundary to
PCI_DMA_BOUNDARY, 4GB, which before this patch caused contiguous
memory allocations to be preferred when allocating more than PAGE_SIZE
bytes. Even if the required alignment was less than PAGE_SIZE bytes.

This patch also fixes the nsegments check for using kmem_alloc_attr()
when the maximum segment size is less than PAGE_SIZE bytes.

Updated some comments describing the code in question.

Differential Revision: https://reviews.freebsd.org/D10645
Reviewed by: kib, jhb, gallatin, scottl
Sponsored by: Mellanox Technologies

MFC r293830:

Fix the spelling of fueword* to eliminate compile warnings about mismatched
begin/end symbols when the warning level is turned up.

MFC r293053, r293061, r293063, r293064, r293065, r293775, r293792:

Use 64-bit math when finding a block of ram to hold the kernel. This fixes
a problem on 32-bit systems which have ram occupying the end of the physical
address space -- for example, a block of ram at 0x80000000 with a size of
0x80000000 was overflowing 32 bit math and ending up with a calculated size
of zero.

Use 64-bit math when processing the lists of physical and excluded memory
to generate the phys_avail and dump_avail arrays.

Work around problems that happen when there is ram at the end of the
physical address space.

Cast pointer through uintptr_t on the way to uint64_t to squelch a warning.

Reword the comment to better describe what I found while researching the
problem that led to this temporary workaround (and also so I can properly
cite the PR in the commit this time).

Cast using uintfptr_t and eliminate the cast to uint64_t which is uneeded
because rounding down cannot increase the number of bits needed to express
the result.

Go back to using uintptr_t, because code that actually compiles is
infinitely less buggy than code that is theoretically correct in some
alternate universe.

PR: 201614

MFC r293045, r293046:

Make the 'env' directive described in config(5) work on all architectures,
providing compiled-in static environment data that is used instead of any
data passed in from a boot loader.

Previously 'env' worked only on i386 and arm xscale systems, because it
required the MD startup code to examine the global envmode variable and
decide whether to use static_env or an environment obtained from the boot
loader, and set the global kern_envp accordingly. Most startup code wasn't
doing so. Making things even more complex, some mips startup code uses an
alternate scheme that involves calling init_static_kenv() to pass an empty
buffer and its size, then uses a series of kern_setenv() calls to populate
that buffer.

Now all MD startup code calls init_static_kenv(), and that routine provides
a single point where envmode is checked and the decision is made whether to
use the compiled-in static_kenv or the values provided by the MD code.

The routine also continues to serve its original purpose for mips; if a
non-zero buffer size is passed the routine installs the empty buffer ready
to accept kern_setenv() values. Now if the size is zero, the provided buffer
full of existing env data is installed. A NULL pointer can be passed if the
boot loader provides no env data; this allows the static env to be installed
if envmode is set to do so.

Most of the work here is a near-mechanical change to call the init function
instead of directly setting kern_envp. A notable exception is in xen/pv.c;
that code was originally installing a buffer full of preformatted env data
along with its non-zero size (like mips code does), which would have allowed
kern_setenv() calls to wipe out the preformatted data. Now it passes a zero
for the size so that the buffer of data it installs is treated as
non-writeable.

Also, revert accidental change that snuck into r293045.

MFC r290647, r292523, r292891:

ARM: Improve robustness of locore_v6.S and fix errors.
- boot page table is not allocated in data section, so must be
cleared before use
- map only one section (1 MB) for SOCDEV mapping (*)
- DSB must be used for ensuring of finishing TLB operations
- Invalidate BTB when appropriate

Allow armv4/5 kernels to be loaded on any 2MB boundary, like armv6/7.

This eliminates the reliance on PHYSADDR and KERNPHYSADDR compile-time
symbols (except when the rom-copy code is enabled) by using the current
PC and the assumption that the entry-point routine is in the first 1MB
section of the text segment.

Other cleanups done:

- Reduce the initarm() stack size back to 2K. It got increased to
4 * 2K when this file was supporting multicore armv6, but that
support is now in locore-v6.S.

- When building the temporary startup page tables, map the entire
4GB address space as VA=PA before mapping the kernel at its loaded
location. This allows access to boot parameters stored somewhere
in ram by the bootloader, regardless of where that may be.

- When building the page table entry for supporting EARLY_PRINTF, map
the section as uncached unbuffered, since it is presumably device
registers.

Note that this restores the ability to use loader(8)/ubldr on armv4/5
kernels. That was broken in r283035, the point at which ubldr started
loading an arm kernel at any 2MB boundary.

Also note that after this, there is no reason to set KERNVIRTADDR to
anything other than 0xc0000000, and no need for PHYSADDR or KERNPHYSADDR
symbols at all.

Bring some of the recent locore-v4.S improvements into locore-V6...

- Map all 4GB as VA=PA so that args passed in from a bootloader can
be accessed regardless of where they are.
- Figure out the kernel load address by directly masking the PC rather
then by doing pc-relative math on the _start symbol.
- For EARLY_PRINTF support, map device memory as uncacheable (no-op for
ARM_NEW_PMAP because all TEX types resolve to uncacheable).

MFC r277416, r282023, r282024, r282025, r284264:

Remove the SMP code from locore-v4. These will never use the SMP code as
there is no multi-core hardware prior to ARMv6.

Remove the armv6 code from locore-v4.S, it's not needed there.

Fix the style of locore-v4.S and locore-v6.S to help find any common code.

Cleanup a little more:
- Remove whitespace at the end of lines
- Use a tab after instructions, not spaces

Fix the spelling of __ARM_ARCH >= 6 in sys/arm/arm.

MFC r289619:

Follow the advice of the misplaced comment and don't access the map struct
after freeing it. Remove the comment whose uselessness has been revealed.

MFC r293613:

Implement vsyscall hack. Prior to 2.13 glibc uses vsyscall
instead of vdso. An upcoming linux_base-c6 needs it.

MFC r283479:

The kernel sends signals to the processes via ABI specific sv_sendsig method.
Native ABI do not need signal conversion, only emulators may want this. Usually
emulators implements its own sv_sendsig method. For now only ibcs2 emulator does
not have own sv_sendsig implementation and depends on native sendsig() method.
So, remove any extra attempts to convert signal numbers from native sendsig()
methods except from i386 where ibsc2 is living.

MFC r283382:

In preparation for switching linuxulator to the use the native 1:1
threads add a hook for cleaning thread resources before the thread die.

MFC r288000:
Add support for weak symbols to the kernel linkers.

MFC:

r283014: Disable unmapped I/O: it is broken for unaligned pages
r283126: Fix comments

MFC r283033, r283062, r283066, r283069:

Do not set preload_addr_relocate for ARM.

Refactor net_getparams() to make it easier to get params from sources other
than bootp and rarp.

Add a routine to obtain netboot parameters from the U-Boot env vars

Enable the NETIF_OPEN_CLOSE_ONCE option for ubldr.

MFC r282418:

On an icache sync by address/len, round the length up if the operation
spans a cacheline boundary.

MFC r280278, r280402:

Allow to override default kernel virtual address assignment on ARM.

Do not save/restore the TLS pointer on context switch for armv6.

MFC r279810, r279811:

Clean data cache before instruction cache in armv7_icache_sync_range().

Add minimum cache line sizes to struct cpuinfo, use them in the new cache
maintenance routines. Also add a routine to invalidate the branch cache.

MFC r278518: Resolve cache line size from CP15 instead of hard-coded 32.

MFC r277098, r279235:

Introduce ofw_bus_reg_to_rl() to replace part of common bus code

Fix endianness on FDT read in ARM GIC

MFC r274249, r274484, r275583:

Avoid panic in ofwbus caused by not released resource list entry

Fix typo in ARM GIC device_printf()

Fix buffer overflow in Marvell PCI/PCIe driver

MFC r279702: Update a comment that had drifted out of date.

MFC r278770, r279114, r279215, r279338, r279543:

Add logic for handling new-style ARM cpu ID info.

Correct a comment which was exactly backwards from reality.

There is no reason to do i+dcache writeback and invalidate when changing
the translation table (this may be left over from armv5 days). It's
especially bad to do so using a cache operation that isn't coherent on
SMP systems.

Add casting to make atomic ops work for pointers. (Apparently nobody has
ever done atomic ops on pointers before now on arm).

Revert incorrect casting.

MFC r282120:
The add_bounce_page() function can be called when loading physical
pages which pass a NULL virtual address. If the BUS_DMA_KEEP_PG_OFFSET
flag is set, use the physical address to compute the page offset
instead. The physical address should always be valid when adding
bounce pages and should contain the same page offset like the virtual
address.

Submitted by: Svatopluk Kraus <onwahe@gmail.com>
Reviewed by: jhb@

Merge r263233 from HEAD to stable/10:

Update kernel inclusions of capability.h to use capsicum.h instead; some
further refinement is required as some device drivers intended to be
portable over FreeBSD versions rely on __FreeBSD_version to decide whether
to include capability.h.

Sponsored by: Google, Inc.

MFC r279312:

Fix a number of -Wcast-qual warnings under sys/arm. No functional
change.

Submitted by: andrew

MFC r277643:
Remove Giant from /dev/mem and /dev/kmem.

MFC r277743:
Arm: ensure that _tmppt KVA is used exclusively.

MFC r278031: Remove a stale comment.

MFC r277523: Add last_fault_code when DEBUG is defined.

MFC r277532, r277533: Add Maxmem global for arm.

MFC r277454, r277460, r277465, r277466, r277467, r277469, r277470, r277471,
r277472, r277473, r277474, r277475, r277476, r277477, r277478, r277479,
r277480, r277512, r277516:

Add inline implementations of arm bus_space_read/write_N().

Revise the arm bus_space implementation to avoid dereferencing the tag on
every operation to retrieve the bs_cookie value almost nothing actually uses.

Use the explicit member initializer style to init the bus_space struct.

Use arm/bus_space-v6.c for all armv6 systems

Consolidate many identical implementations of bus_space to a single
common tag and implementation shared by armv4 and armv6.

Micro-optimize the new arm inline bus_space implementation by grouping all
the data the inline functions access together at the start of the bus_space
struct so that they all fit in a single cache line.

MFC r277305: Minor cleanups, comment changes.

MFC r266083, r267597:

Give suitably-endowed ARMs a register similar to the x86 TSC register.

MFC r272356, r275639, r276638:

Split syscall handling out to a separate file.

Include sys/kernel.h to pick up the definition of hz in syscall.c

Add a new trap-v6.c which has support for all armv7 exceptions.

MFC r276525, r276596:

Put in a workaround for bug 196407 (arm modules cause crashes & panics).
(Don't allow movw/movt insn in modules.)

Fix alignment directives in arm asm code after clang 3.5 import.

MFC r276519:

Define a WFI macro that expands to the right form of wait-for-interrupt
depending on the architecture.

MFC r276518: Rework vfp code so it will compile on clang 3.4 and 3.5.

MFC r276444, r276445, r276470:

Fix a paste-o in dcache_inv_pou_all().

Change the order of operations for the initial cache setup.

A couple small fixes to make clang 3.5 happy... Move END(sigcode)
and other misplace ENTRY/END macros.

MFC r276394, r276397:

Add armv6 implementations of the startup-time cache maintenence functions.

Create a new locore.S that includes locore-v4 or locore-v6 as needed.

MFC r276395, r276396: Rename locore.S to locore-v4.S and add a new locore-v6.S.

MFC r276247, r276333, r276334, r276335, r276336, r276340, r276350:

Include acle-compat.h directly rather than getting it via sysreg.h.

Add new code to read and parse cpu identification data using the new CPUID
mechanism defined for armv7.

Add new TLB and cache maintainence functions for armv6 and armv7.

Eliminate an unused macro whose name clashes now with a function in the
new cpu-v6.h

Add cache maintenance functions which will be used by startup code to
initially set up the MMU.

Fix a "decl is not a prototype" error noticed by gcc (but not clang).

Update comments (r4 is not used anywhere), use non-profiling entry macros.

Add arm option ARM_NEW_PMAP, to allow us to begin adding the new pmap
code alongside the existing implementation

Add armv6 implementations of cache operations to avoid duplication

MFC r272209, r272300, r276212, r276213:

Add machine/sysreg.h to simplify accessing the system control coprocessor
registers and use it in the ARMv7 CPU functions.

Add macros for asm barrier instructions with arch-specific implementations.

Define only the CP15 register operations that are valid for the architecture.

MFC r276206:

For data and instruction prefetch aborts, call the same handler in the C
code, passing a 0/1 flag that indicates which type of abort it was. This
sets the stage for unifying the handling of page faults in a single routine.

MFC r276196, r276197, r276198, r276202, r276203, r276204:

Change the style of the DO_AST macro to match the others

Remove _PROF_PROLOGUE from the EENTRY() macros.

Stylish changes... put tabs where they need to be in macros, move lines
around so that related things are more grouped together, rewrite comments.

Fix the GLOBAL macro so it works (upper vs lowercase X), use it in _EENTRY.

Create 'L' variants of all the ENTRY macros for file-static/local symbols.

MFC r276191: Display correct value for cache level-of-coherency (needs +1).

MFC r276187, r276190, r271422:

Rename pmap_kenter_temp to pmap_kenter_temporary to be consistent with the
other architectures with this function.

Eliminate unnecessary references to pte.h internals by using the standard
pmap_kenter_temporary() to map pages while dumping.

Cleanup up ARM *frame structures.

MFC r271394, r271398:

Add more register values to armreg.h and remove CPU_CONTROL_32BP_ENABLE
from asm.h as they were already defined in armreg.h.

Unify interrupts bit definition and usage. While here remove PSR_C_bit.

MFC r278001:
Do not qualify the mcontext_t *mcp argument for set_mcontext(9) as const.

MFC r273599:

Fix a bug where DMA maps created with bus_dmamap_create() won't increment
the map count and without being able to keep track of the current map
allocation, bus_dma_tag_destroy() will fail to proceed and will return
EBUSY even after all the maps have been correctly destroyed with
bus_dmamap_destroy().

Found while testing the detach method of a NIC.

MFC r273701, r274556
By the time that pmap_init() runs, vm_phys_segs[] has been initialized.
Obtaining the end of memory address from vm_phys_segs[] is a little
easier than obtaining it from phys_avail[].

Enable the use of VM_PHYSSEG_SPARSE on amd64 and i386, making it the
default on i386 PAE. (The use of VM_PHYSSEG_SPARSE on i386 PAE saves
us some precious kernel virtual address space that would have been
wasted on unused vm_page structures.)

MFC r274538, r274545, r274596, r274602, r274603, r274604, r274605, r274839:

When doing busdma sync ops for BUSDMA_COHERENT memory, there is no need
for cache maintenance operations, but ensure that all prior writes have
reached memory when doing a PREWRITE sync.

Do not do a cache invalidate on a PREREAD sync that is also a PREWRITE sync.

Do the cache invalidate sequence from the outermost to innermost, required
for correct operation.

Correct the sequence of busdma sync ops involved with PRE/POSTREAD syncs.

When doing a PREREAD sync of an mbuf-type dma buffer, do a writeback of
the first cacheline if the buffer start address is not on a cacheline
boundary.

MFC 275564:

Use the unified syntax when generating assembly for clang. The clang 3.5
integrated assembler only accepts it.

Sponsored by: ABT Systems Ltd

Clean up our ARM assembly:

MFC 275256:

Switch to the ARM unified assembly language as the clang integrated as only
supports it. Binutils supports it when the ".syntax unified" directive
is set.

Sponsored by: ABT Systems Ltd

MFC 275264:

Update _ENTRY to use _EENTRY to reduce the common code.

MFC 275321:

Remove extra labels, ENTRY_NP already provides them.

Sponsored by: ABT Systems Ltd

MFC 275322:

Correctly a few incorrect uses of ENTRY/EENTRY and END/EEND

Sponsored by: ABT Systems Ltd

MFC 275416:

Fix the name of the coprocessor to include the "p" prefix, the clang
integrated assembler expects this.

Sponsored by: ABT Systems Ltd

MFC 275418:

Switch to unified syntax so these can be built with clang 3.5.

Sponsored by: ABT Systems Ltd

MFC 275519:

Add missing END macros to some of the xscale functions.

Sponsored by: ABT Systems Ltd

MFC 275520:

Use the unified syntax in a few more assembly files

Sponsored by: ABT Systems Ltd

MFC 275521:

Set the alignment to 4-bytes after a string as clang 3.5 can switch to
thumb mode if this is incorrect.

Sponsored by: ABT Systems Ltd

MFC 275522:

Place the literal pool after a RET otherwise clang 3.5 tries to put it too
far away from a ldr psuedo instruction. With this clang will place the
literal value here where it's close enough to be loaded.

Sponsored by: ABT Systems Ltd

MFC 275523:

Switch to an armv6k cpu, without this clang 3.5 complains "bx lr" is
unsupported as it needs a newer cpu.

Sponsored by: ABT Systems Ltd

MFC 275524:

Switch to a .cpu directive. These will work when clang 3.5 is imported
where the .arch directive is a nop.

Sponsored by: ABT Systems Ltd

MFC 275207:

We don't use the hypervisor interrupt, make it optional in the device tree.

Submitted by: Julien Grall <julien.grall AT linaro.org>

MFC 275208:

Some device tree configurations place the generic timer under the root
of the tree and not under simplebus. Update the driver to handle this.

Submitted by: Julien Grall <julien.grall AT linaro.org>

MFC r273703:

Remove the ARM_DEVICE_MULTIPASS option and make its effect be the default.

MFC r273288:

Allow the armv6 kernel to be build with PHYSADDR undefined. The kernel
will now find the virtual to physical mapping for libkvm to use at
runtime. This makes PHYSADDR redundant, however keep it around to give
everyone a chance to update their libkvm.

MFC r273284:

Allow libkvm to get the kernel va to pa delta without the need for
physaddr. This should allow for a kernel where PHYSADDR and KERNPHYSADDR
are both undefined.

For now libkvm will use the old method of reading physaddr and kernaddr
to allow it to work with old kernels. This could be removed in the future
when enough time has passed.

MFC r269956:

From https://sourceware.org/ml/newlib/2014/msg00113.html
By Richard Earnshaw at ARM
>
>GCC has for a number of years provides a set of pre-defined macros for
>use with determining the ISA and features of the target during
>pre-processing. However, the design was always somewhat cumbersome in
>that each new architecture revision created a new define and then
>removed the previous one. This meant that it was necessary to keep
>updating the support code simply to recognise a new architecture being
>added.
>
>The ACLE specification (ARM C Language Extentions)
>(http://infocenter.arm.com/help/index.jsp?topic=/com.arm.doc.set.swdev/index.html)
>provides a much more suitable interface and GCC has supported this
>since gcc-4.8.
>
>This patch makes use of the ACLE pre-defines to map to the internal
>feature definitions. To support older versions of GCC a compatibility
>header is provided that maps the traditional pre-defines onto the new
>ACLE ones.

Stop using __FreeBSD_ARCH_armv6__ and switch to __ARM_ARCH >= 6 in the
couple of places in tree. clang already implements ACLE. Add a define
that says we implement version 1.1, even though the implementation
isn't quite complete.

MFC r263710, r273377, r273378, r273423 and r273455:

- De-vnet hash sizes and hash masks.
- Fix multiple issues related to arguments passed to SYSCTL macros.

Sponsored by: Mellanox Technologies

MFC r273590: Accept documented compatible string for PL310 cache controller/

MFC r272333: Honor exclusion flags when building the memory lists.

MFC r271906:

Make the ARM MPCore Timer driver work with published standard FDT bindings.

MFC r271595, r271601, r271607, r271630:

Add compat strings for all the flavors of GIC this driver should support.
Also allow the driver to attach to ofwbus as well as simplebus, some FDT
data puts the root interrupt controller on the root bus.

Add a common routine for parsing FDT data describing an ARM GIC interrupt.

Use gic_decode_fdt() rather than a local routine to parse fdt interrupt
properties. Move fdt_pic_table and fdt_fixup_table into imx6_machdep.c,
which means imx6 doesn't need imx_common.c anymore.

The private peripheral interrupts start at offset 16, not 0. Also, use
names rather than inline mystery constants for these offsets.

MFC r271594:

Fix an undefined variable that was accidentally not causing an error.

MFC r270945:

Rename OF_xref_phandle() to OF_node_from_xref() and add a new function
that provides the inverse translation, OF_xref_from_node().

MFC r270862, r270878: MMU fixes for kernel startup.

Fix the handling of MMU type in the AP entry code. The ARM_MMU_V6/V7
symbols are always #defined to 0 or 1, so use #if SYM not #if defined(SYM).
Also, it helps if you include the header file that defines the symbols.

The Marvell PJ4B cpu family is armv7, not armv6.

Approved by: re(gjb)

Fix a leak of the wired pages when unwiring of the PROT_NONE-mapped
wired region. Rework the handling of unwire to do the it in batch,
both at pmap and object level.

All commits below are by alc.

MFC r268327:
Introduce pmap_unwire().

MFC r268591:
Implement pmap_unwire() for powerpc.

MFC r268776:
Implement pmap_unwire() for arm.

MFC r268806:
pmap_unwire(9) man page.

MFC r269134:
When unwiring a region of an address space, do not assume that the
underlying physical pages are mapped by the pmap. This fixes a leak
of the wired pages on the unwiring of the region mapped with no access
allowed.

MFC r269339:
In the implementation of the new function pmap_unwire(), the call to
MOEA64_PVO_TO_PTE() must be performed before any changes are made to the
PVO. Otherwise, MOEA64_PVO_TO_PTE() will panic.

MFC r269365:
Correct a long-standing problem in moea{,64}_pvo_enter() that was revealed
by the combination of r268591 and r269134: When we attempt to add the
wired attribute to an existing mapping, moea{,64}_pvo_enter() do nothing.
(They only set the wired attribute on newly created mappings.)

MFC r269433:
Handle wiring failures in vm_map_wire() with the new functions
pmap_unwire() and vm_object_unwire().
Retire vm_fault_{un,}wire(), since they are no longer used.

MFC r269438:
Rewrite a loop in vm_map_wire() so that gcc doesn't think that the variable
"rv" is uninitialized.

MFC r269485:
Retire pmap_change_wiring().

Reviewed by: alc

Merge the changes to pmap_enter(9) for sleep-less operation (requested
by flag). The ia64 pmap.c changes are direct commit, since ia64 is
removed on head.

MFC r269368 (by alc):
Retire PVO_EXECUTABLE.

MFC r269728:
Change pmap_enter(9) interface to take flags parameter and superpage
mapping size (currently unused).

MFC r269759 (by alc):
Update the text of a KASSERT() to reflect the changes in r269728.

MFC r269822 (by alc):
Change {_,}pmap_allocpte() so that they look for the flag
PMAP_ENTER_NOSLEEP instead of M_NOWAIT/M_WAITOK when deciding whether
to sleep on page table page allocation.

MFC r270151 (by alc):
Replace KASSERT that no PV list locks are held with a conditional
unlock.

Reviewed by: alc
Approved by: re (gjb)
Sponsored by: The FreeBSD Foundation

MFC r269646: Use a SYSINIT to init the array of interrupt names on arm.

MFC r269594, r269596, r269597, r269598, r269605, r269606:

Set ofwbus and simplebus to attach during BUS_PASS_BUS.

Define names that drivers can use to adjust their position relative to
other drivers within a BUS_PASS

Adjust ofwbus and simplebus to attach at BUS_PASS_ORDER_MIDDLE, so that
a platform can attach some other bus first if necessary.

Set the pl310 L2 cache driver to attach during the middle of BUS_PASS_CPU.

Attach arm generic interrupt and timer drivers in the middle of
BUS_PASS_INTERRUPT and BUS_PASS_TIMER, respectively.

Add an arm option, ARM_DEVICE_MULTIPASS, used to opt-in to multi-pass
device attachment on arm platforms. If this is defined, nexus attaches
early in BUS_PASS_BUS, and other busses and devices attach later, in the
pass number they are set up for. Without it defined, nexus attaches in
BUS_PASS_DEFAULT and thus so does everything else, which is status quo.

MFC r269390: Fix unwind info in hand-written asm (avoid nested functions).

MFC r269206, r269207, r269208, r269209, r269210, r269211, r269212, r269213,
r269214, r269215, r269216, r269217, r269221:

busdma-v6 improvements, primarily:
- Allocate the temporary segments array per-map rather than per-tag.
- Avoid needlessly bouncing IO for mbufs and buffers allocated by
bus_dmamem_alloc() (in both situations we known they're allocated
on cacheline boundaries and don't need bouncing).
- Various minor reformatting and cleanups.

MFC r256691, r256748: casuword fixes

Use unsigned compare against KERNBASE addr.
Use atomic ops on armv6.

MFC r266565, r266651:

Map device memory using PTE_DEVICE attributes, and also ensure that the
shared flag is set on normal-memory mappings made via pmap_kenter() for SMP.

The "shared flag" part of this change isn't obvious from the diff, here's
the deal... by using the array of preformatted page table entry templates
instead of constructing the PTE from scratch, we automatically get the
right attribute bits set for both caching and shared.

Fix whitespace glitches.

MFC r267213 (by alc):
Add a page size field to struct vm_page.

Approved by: alc

MFC 266621: Eliminte spurious interrupts caused by ARM weak memory ordering.

MFC 258287: Implement pmap_align_superpage().

MFC 256941: make sure the frame is indeed in the kernel memory.

MFC 265913, 265914:

Interrupts need to be disabled on entry to cpu_sleep() for ARM. Given
that and the need to be in a critical section when switching to idleclock
mode for event timers, use spinlock_enter()/exit() to achieve both needs.

Clean up some style nits.

MFC 265861, 265870:

Make the hardware memory and instruction barrier functions work on armv4
and armv5 as well.

Add cpu_l2cache_drain_writebuf(), use it to implement generic_bs_barrier().

MFC 265694, 265705, 265784:

Move the mptramp code which is specific to the Marvell ArmadaXP SoC out of
the common locore.S file and into the mv/armadaxp directory.

Consolidate all the AP core startup stuff under a single #ifdef SMP block

Call idcache_inv_all from the AP core entry code before turning on the MMU.
Also, enable instruction and branch caches, which should be safe now that
they're properly initialized/invalidated first.

MFC 265440, 265441, 265444, 265445, 265446, 265447:

Move the pl310.enabled tunable to hw.pl310.enabled. Clean up a few minor
style(9) nits. Use DEVMETHOD_END.

Break out the code that figures out the L2 cache geometry to its own
routine, so that it can be called from multiple places in upcoming changes.

Call platform_pl310_init() before enabling the controller, and handle the
case where the controller is already enabled.

Add defines for the bits in the PL310 debug control register.

Add a public routine to set the L2 cache ram latencies. This can be
called by platform init routines to fine-tune cache performance.

Enable PL310 power-saving modes and tune the cache ram latencies for imx6.

MFC 265035: Move duplicated code to print l2 config into the common code.

MFC 265023, 265024, 265036:

There is no difference between IPI_STOP and IPI_STOP_HARD on ARM, so
map them both to the same interrupt number like other arches do.

Flush and invalidate caches on each CPU as part of handling IPI_STOP.

Don't use multiprocessing-extensions instruction on processors that don't
support SMP.

MFC 264990, 264994, 265020, 265025:

Call cpu_icache_sync_range() rather than sync_all since we know the range
and flushing the entire icache is needlessly expensive.

Provide a proper armv7 implementation of icache_sync_all rather than
using armv7_idcache_wbinv_all, because wbinv_all doesn't broadcast the
operation to other cores. In elf_cpu_load_file() use icache_sync_all()
and explain why it's needed (and why other sync operations aren't).

Remove cpu_idcache_wbinv_all() from kdb_cpu_trap(), it's no longer needed.

Explain why wbinv_all is SMP-safe when dumping, and add a missing l2 cache
flush. (Either it was missing here, or it isn't needed in the minidump
case. Adding it here seems like the safer path to consistancy.)

MFC 264702: Remove uncessary armv6 cache and TLB maintenance ops.

MFC 264160: Remove code under PMAP_CACHE_VIVT, not compiled on armv6.

MFC 264183: Add a couple more required TLB flushes.

MFC 264128, 264129, 264130, 264135,

Fix TTB set operation for armv7. Perform sychronization (by "isb" barrier)
after TTB is set.

Fix TLB maintenance issues for armv6 and armv7.

- Add cpu_cpwait to comply with the convention.
- Add missing TLB invalidations, especially in pmap_kenter & pmap_kremove
with distinguishing between D and ID pages.
- Modify pmap init/bootstrap invalidations to ID, just to be safe.
- Fix TLB-inv and PTE_SYNC ordering.

Allocate per-cpu resources for doing pmap_zero_page() and pmap_copy_page().
This is performance enhancement rather than bugfix.

We don't support any ARM systems with an ISA bus and don't need a freelist
of memory to support ISA addressing limitations.

MFC 264052, 264057, 264065, 264094, 264103, 264120

Actually save the mpcore clock frequency retrieved from fdt data.

imx6..
- Don't call sdhci_init_slot() until after handling the FDT properties
related to detecting card presence.
- Flag several sysctl variables as tunables.
- Rework the cpu frequency management code for imx6 to add "operating
points" and min/max frequency controls.

generic timer...
- Setup both secure and non-secure timer IRQs.
We don't know our ARM security state, so one of them will operate.
- Don't set frequency, since it's unpossible in non-secure state.
Only rely on DTS clock-frequency value or get clock from timer.

MFC 264019, 264041, 264048, 264049, 264050, 264051

Add support for event timers whose clock frequency can change while running.

Apparently all ARM configs build kern_et.c, but only a few of them also
build kern_clocksource.c, un-break the build by not referencing functions in
kern_clocksource if NO_EVENTTIMERS is defined.

Add variable-frequency support to the arm mpcore eventtimer driver.

mpcore_timer: Disable the timer and clear any pending bit, then setup the
new counter register values, then restart the timer. Also re-nest the parens
properly for casting the result of converting time and frequency to a count.

MFC 263910, 263913, 263914, 263933, 263934, 263935, 263936, 263981, 263982,

Add more flags for the fpexc register from the ARM1176JZF-S Manual

Initialise fpscr to a sane value when we create the pcb. This sets NaNs to
be the default NaN and for denormalised numbers to be flushed to zero.

VFP fixes/cleanups for ARM11:
* Save the required VFP registers on context switch. If the exception bit
is set we need to save and restore the FPINST register, and if the fp2v
bit is also set we need to save and restore FPINST2.
* Move saving and restoring the floating point control registers to C.
* Clear the fpexc exception and fp2v flags on a floating-point exception.
* Signal a SIGFPE if the fpexc exception flag is set on an undefined
instruction. This is how the ARM core signals to software there is a
floating-point exception.

Add Cortex-A15 cpu id revisions.

Exynos/Arndale...
- Merge SoC-common parts
- Enable iicbus device
- Directly call kmem_alloc_contig to allocate framebuffer memory
and pass VM_MEMATTR_UNCACHEABLE (no-cache, no-buffer).
This fixes screen refreshing problem when data is updated too slowly.
- Add support for keyboard used in Samsung Chromebook (ARM machine)
Support covers device drivers for:
- Interrupt Combiner
- gpio/pad, External Interrupts Controller (pad)
- I2C Interface
- Chrome Embedded Controller
- Chrome Keyboard
- Use new gpio dev class in EHCI driver
- Expand device tree information
- Release i2c bus on detach.

MFC 263250, 263251, 263424, 263425, 263426, 263427, 263430, 263431

Use the same cache terminology as the ARM docs in comments. No
functional changes.

Use armv7 TLB flush code, not arm11, for cortex-a processors.

Exynos/ Arndale...
- Disable debugging by default.
- Add display-related and clk devices to the tree
- Prevent resources intersection with EHCI driver
- Add display-related and clk devices to the tree
- Prevent resources intersection with EHCI driver
- Add driver for Display Controller.
- Add support for Samsung Chromebook (ARM Cortex A15 machine).
- Rename mct and ehci drivers files to match common naming.

MFC 263036, 263059: delete advertising clause in licenses, renumber.

MFC 262952, 262958, 262966, 262979, 262980, 262986, 262987, 262995, 262997,
263030, 263033, 263034, 263056, 263057,

Remove all the redundant external declarations of exception vectors and
runtime setting of the pointers that's scattered around various places.

Remove all traces of support for ARM chips prior to the arm9 series.

Make the default exception handler vectors point to where I thought they
were already pointing: the default handlers (not a panic that says there
is no default handler).

Eliminate irq_dispatch.S. Move the data items it contained into
arm/intr.c and the functionality it provided into arm/exception.S.

Move the exception vector table (so-called "page0" data) into exception.S
and eliminate vectors.S.

Change the way the asm GET_CURTHREAD_PTR() macro is defined so that code
using it doesn't have to have an "AST_LOCALS" macro somewhere in the file.

Arrange for arm fork_trampoline() to return to userland via the standard
swi_exit code in exception.S instead of having its own inline expansion
of the DO_AST and PULLFRAME macros.

Now that the PUSHFRAME and PULLFRAME macros are used only in the swi
entry/exit code, they don't need to be macros. Except that didn't work
and the whole change was reverted.

Remove some unnecessary indirection and jump right to the handler functions.

Use panic rather than printf to "handle" an arm26 address exception
(should never happen on arm32).

Remove the unreferenced DATA() macro.

Remove #include <machine/asmacros.h> from files that don't need it.

MFC 257774, 256760, 262916, 262905, 262918, 262919, 262920, 262921, 262924,
262925, 262929, 262932, 262935, 262940, 262941, 262942, 262948, 262949,
262950

Strip arm/conf/DEFAULTS down to just items that are mandatory for running
the architecture.

Move all the files named foo/common.c to foo/foo_common.c

Initial cut for DTS on the hl201 board.

Add commented out dts for sam9260ek as well as early printf support.

Make clock optional on uart nodes, then back it out ("I don't know what I
was thinking, but it is lame.")

Set the baud rate if it isn't 0

Make at91_soc_id() public.

Properly round at91 resource on unmapping.

Move AT91 AIC related stuff to own file.

Fix another bug in multicast filtering. i.MX uses 6 bits from MSB in
LE CRC32 for the hash value, not the lowest 6 bits in BE CRC32.

Follow r262916 with one more config file that references a renamed common.c

Remove bogus AT91 define that causes compile errors. Most of the defines
for SAM9X are going away soonish anyway (once FDT works), but until
then...

Remove all dregs of a per-thread undefined-exception-mode stack.

Rework the VFP code that handles demand-based save and restore of state.

Always call vfp_discard() on thread death.

When a thread begins life it doesn't own the VFP hardware state on any cpu.

Make undefined exception entry MP-safe.

MFC 262695, 262708, 262709, 262710, 262711, 262728, 262870, 262877, 262880,
262885, 262891, 262903,

imx6: Add a tunable to set the number of active cores, enable SMP by default.

ffec: Fix multicast filtering.

Allwinner a10/a20...
- Add gpio and clock bits for A10/A20's EMAC ethernet controller driver
- EMAC gpio configuration
- EMAC clock activation
- Add Static Random Access Memory controller driver for A10/A20.
A10/A20's SRAM is used by devices, such as CPU, EMAC,
for extra fast memory or as cache.
- Add EMAC 10/100 Ethernet controller driver for A10/A20.
It is available mostly in A10 devices like Hackberry, Marsboard,
Mele A1000, A2000, A100 HTPC, cubieboard1 and A20 device
like cubieboard2.
TX performance can be improved using both channels 0 and 1.
RX performance is poor and needs improvement with the assistance of
external DMA controller in case there
- Add EMAC and SRAM controller entries to FDT.
- Add EMAC device to kernel config files and enable EMAC, SRAM drivers.

OMAP: When calculating the MPU freq, make sure not to overflow.

Vybrid:
- Add driver for Port control and interrupts (PORT).
- Export panel info to DTS
- Reset all the layers before setup first one
- Enable display

nandfs: Slight code reordering to make error branch last.

Add option TMPFS to arm/conf/DEFAULTS, remove it from the few configs
that have it individually. Concensus on freebsd-arm@ is that it should
be included in all ARM kernels.

Fix the arm sys_sigreturn(): its argument is a struct ucontext, not a
struct sigframe containing the struct ucontext.

MFC r262534, r262548, r262549, r262552, r262568, r262581, r262583, r262584,
r262585, r262587, r262696, r262712

Replace many pasted identical definitions of cpu_initclocks() with a common
implementation in arm/machdep.c.

aicasm: Don't complain about missing prototypes to ease bootstrap issues.

Vybrid: Add driver for Inter-Integrated Circuit (I2C).

imx6: Initialize the Low Power Mode bits to keep the ARM cores running
during WFI.

All our current ARM multi-core systems have all cores in one package with
a shared L2 cache, reflect that in the common cpu_topo() routine.

mpcore timer: Supply a DELAY() implementation via weak linkage, so that
SoC-specific code can supply a better implementation.

imx6: Add some rudimentary voltage control.

Add an armv7 implementation of cpu_sleep().

Add __used attribute so that the DELAY implementation doesn't get
optimized away as unreferenced, causing linker errors when trying to
resolve the weak reference to the missing function.

MFC r262409, r262411, r262413, r262420, r262426, r262427, r262440, r262456,
r262482, r262483, r262531,

Move the declaration for mpentry() into a header file instead of pasting
it into a bunch of different .c files.

If the L2 cache type is PIPT, pass a physical address for a flush.

Actually set the proper bit to indicate TTB shared memory.

Add a new cache maintenance function, idcache_inv_all, to the table, and
implementations for each of the chips we support.

Invalidate caches immediately upon entry to init_secondary(). Also set
the Bufferable bit in the PDE entries of the secondary processor startup
pagetables.

Add the bits needed to run SMP on imx6.

Invalidate the SCU cache tag ram on all 4 cores, not just 1-3.

Minor tweaks to the imx GPT timer

Vybrid enhancements...
- Pin configuration is a complete iomux register now and includes
drive strength, pull mode, mux mode, speed, etc.
- Add i2c devices to the tree
- Add IPG clock
- Add support for Quartz Module.
- Pin configuration is a complete iomux register now and includes
drive strength, pull mode, mux mode, speed, etc.
- Add i2c devices to the tree
- Add IPG clock

MFC r261982, r261987, r262123, r262244, r262278, r262280, r262353, r262354,
r262355, r262419,

Add Vybrid driver for Synchronous Audio Interface (SAI).

Decrease SAI buffer size. Handle eDMA interrupt on running channel only.

Give the physmem fdt helper routines static linkage since no global
definition of them is provided anywhere.

Add imx6 early printf support, wrapped in #if 0 because it's rarely needed.

Add basic cpu frequency control and temperature monitoring to imx6_anatop.

Add the FREEBSD_BOOT_LOADER option so that a loaded DTB passed in from
ubldr will actually get used.

Create a generic IMX6 kernel config, then fix it to have an ident line.

Don't force imx6 bootverbose on anymore, it can be set from ubldr now.

MFC r261917, r261918, r261919, r261920, r261921, r261922

Always clear L1 PTE descriptor when removing superpage on ARM

Invalidate L1 PTE regardles of existance of the corresponding l2_bucket.

Ensure proper TLB invalidation on superpage promotion and demotion on ARM

Base pages within newly created superpage need to be invalidated so that
new mapping is "visible" immediately after creation.

Fix superpage promotion on ARM with respect to RO/RW and wired attributes

Avoid redundant superpage promotion attempts on ARM

Remove spurious assertion from pmap_extract_locked() on ARM

Handle pmap_enter() on already promoted mappings for ARMv6/v7

MFC r261803, r261808, r261814, r261815, r261816, r261817, r261818, r261826,
r261848, r261855

On armv6 and later, use the WriteNotRead bit of the fault status register
to decide what protections are required by the faulting access.

Use the right symbols for determining arm architecture. Include the
necessary header file which has the new FAULT_WNR symbol defined in it.

Allow the kernel to be loaded at any 1MiB address. This requirement is
because we use the 1MiB section maps as they only need a single pagetable.

Add function for configuring Vybrid PLL4 (Audio) clock frequency output.

imx6 changes ...

- Fix the definition of the SDHCI_STATE_DAT and SDHCI_STATE_CMD fields,
and add SDHCI_RETUNE_REQUEST. None of these are actually used in the
code yet.

- Write translation code for the SDHCI_PRESENT_STATE register.
Freescale moved some bits around in their version of the register,
adjust things so that the sdhci code sees the standard layout.

- Add standard non-removable and cd-gpios properties to the usdhc
devices. That generates references to gpio devices, so uncomment them
even though there isn't a gpio driver to do anything with them yet.

- Add handling of standard "non-removable" property, and also some
workaround code so that if card detect is wired to a gpio pin, for now
we just treat it the same as non-removable (because there isn't a gpio
driver yet).

- Enable both sdcard slots, but not the sdio-based wifi that we don't
yet have a driver for.

- Remove a couple obsolete function declarations.

MFC r261786, r261789

Rework the EARLY_PRINTF mechanism. Instead of defining a special eprintf()
routine, now a platform can provide a pointer to an early_putc() routine
which is used instead of cn_putc(). Control can be handed off from early
printf support to standard console support by NULLing out the pointer
during standard console init.

Convert two while(1); statements into proper panics.

MFC r261681, r261682, r261683, r261684, r261685, r261686, r261687, r261688,
r261689, r261690, r261783, r261791, r261836, r261837, r261841,

Add FDT matching code to AT91 device drivers.

Better nomatch messages: include compat string. Also, flag devices as
disabled in the successful probe message, but leave what that means to
the actual driver (no semantic changes).

Fix Embest board name and id.

Honor the disabled status by only grabbing resources and returning
when running under FDT in the AT91 SPI driver.

MFC r261643, r261646, r261648, r261649, r261651, r261656, r261657, r261663,
r261676, r261677, r261698, r261778

Consolidate code related to setting up physical memory configuration into
a new physmem.c file.

Replace compile-time constant KERNPHYSADDR with abp_physaddr

Calculate the kernel's load address from the PC in the elf / gzip
trampoline instead of relying on KERNPHYSADDR as a compile-time constant.

It turns out a global variable is the only straightforward way to
communicate the kernel's physical load address from where it's known in
initarm() into cpu_mp_start() which is called from non-arm code and
takes no parameters.

Remove the now unused MMU_INIT macro.

Use vm_paddr_t, not vm_offset_t, when dealing with physical addresses.

No need to set physmem in each initarm() instance anymore, it's handled
in common code now.

Pass the pagetable used from locore.S to initarm to allow it to map data
in as required.

Fix the physmem exclude-region clipping logic for the edge-trim case.

Add some extra debugging output when DEBUG is defined.

Update legacy platforms to use new arm_physmem helper routines.

MFC r256672

If we avoid to use the page at addr 0, we should adjust the size to reflect it.

MFC r257549, r261642

Don't create a distinct free page pool for segregating allocations that are
accessed through the direct map unless the kernel configuration actually
includes a direct map. Only a few configurations do, and for the rest the
unnecessary free page pool is a small pessimization.

Remove the ARM_USE_SMALL_ALLOC option and code related to it.

MFC r261423, r261424, r261516, r261513, r261562, r261563, r261564, r261565,
r261596, r261606

Add the imx sdhci controller.

Move Open Firmware device root on PowerPC, ARM, and MIPS systems to
a sub-node of nexus (ofwbus) rather than direct attach under nexus. This
fixes FDT on x86 and will make coexistence with ACPI on ARM systems easier.
SPARC is unchanged.

Add the missing ')' at end of sentence. Reword it to use a more common idiom.

Pass the kernel physical address to initarm through the boot param struct.

Make functions only used in vfp.c static, and remove vfp_enable.

Fix __syscall on armeb EABI. As it returns a 64-bit value it needs to
place 32-bit data in r1, not r0. 64-bit data is already packed correctly.

Use abp_physaddr for the physical address over KERNPHYSADDR. This helps us
remove the need to load the kernel at a fixed address.

Remove references to PHYSADDR where it's used only in debugging output.

Dynamically generate the page table. This will allow us to detect the
physical address we are loaded at to change the mapping.

MFC r261414, r261415, r261417, r261418, r261419

Don't call device_set_ivars() for the mmchs

Change the way pcpu and curthread are stored per-core

Invalidate cachelines for bounce pages on PREREAD too, there may still be
stale entries from a previous transfer.

Only use the CPU ID register if SMP is defined. Some non-MPCore armv6 cpu,
such as the one found in the RPi, don't have it, and just hang when we try
to access it.

MFC r261410

Follow r261352 by updating all drivers which are children of simplebus
to check the status property in their probe routines.

MFC r261137, r261393

Correct the alignment of sp through functions that use UNWINDSVCFRAME.

Update all arm code that manipulates the PSR registers to use modern syntax.

MFC r261351, r261352, r261355, r261396, r261397, r261398, r261403, r261404,
r261405

Open Firmware interrupt specifiers can consist of arbitrary-length byte
strings and include arbitrary information (IRQ line/domain/sense). When the
ofw_bus_map_intr() API was introduced, it assumed that, as on most systems,
these were either 1 cell, containing an interrupt line, or 2, containing
a line number plus a sense code. It turns out a non-negligible number of
ARM systems use 3 (or even 4!) cells for interrupts, so make this more
general.

Provide a simpler and more standards-compliant simplebus implementation to
get the Routerboard 800 up and running with the vendor device tree. This
does not implement some BERI-specific features (which hopefully won't be
necessary soon), so move the old code to mips/beri, with a higher attach
priority when built, until MIPS interrupt domain support is rearranged.

Allow nesting of simplebuses.

Add a set of helpers (ofw_bus_get_status() and ofw_bus_status_okay()) to
process "status" properties of OF nodes.

Fix one remnant endian flaw in nexus.

MFC r261252, r261279, r261304, r261305, r261322, r261336, r261337, r261338,
r261353

Fix the name of the dts file for the HL201...

When mapping an address, the bsh needs the same offset we do for other things.

Add explicit depends on bus_if.h and device_if.h to avoid a
chicken and egg problem in some compilation environments.

Switch to using PAs rather than VAs for the addresses we map for
devices. This is a nop, except for what's reported by atmelbus for the
resources.

Comment cleanups. Move things around for diff reduction against FDT work.

MFC r260886, r261165, r261171, r261172, r261214

Fix gcc with -Wstrict-prototypes by telling it bi_emac takes no
parameters.

Bus space handles need to be the VA of the requested resource, not the
rounded page VA. Correct so the DBGU device can be mapped for FDT
console since it isn't on a page boundary.

Make early printf output nicer by inserting a carriage return before
any linefeeds that are output.

Before resetting the USART, delay a bit to allow the transmitter to
finish the current character to drain to avoid glitching. Also,
simplify the code a smidge.

Remove extra parens to silence clang warning.

MFC r261038, r261039, r261040, r261041

Implement generic support for early printf.

MFC r260440, r260441, r260447, r260490, r260493

Add option USB_HOST_ALIGN to configs that contain 'device usb'.

Update dts files of Cubieboard1,2 to use 1GB memory.

Add a function to print the contents of the static device mapping table,

MFC r257738, r259202, r258410, r260288, r260292, r260294, r260320, r260323,
r260326, r260327, r260331, r260333, r260340, r260371, r260372, r260373,
r260374, r260375

Add common bus_space tag definition shared for most supported ARMv6/v7 SoCs.
Correct license statements to reflect the fact that these files were all
derived from sys/arm/mv/bus_space.c.

In pmap_unmapdev(), remember the size, and use that as an argument to
kva_free(), or we'd end up always passing it a size of 0

In pmap_mapdev(), first check whether a static mapping exists,

Convert TI static device mapping to use the new arm_devmap_add_entry(),

Use the common armv6 fdt_bus_tag defintion for tegra instead of a local copy.

Eliminate use of fdt_immr_addr(), it's not needed for tegra

Convert lpc from using fdt_immr style to arm_devmap_add_entry() to make
static device mappings.

Retire machine/fdt.h as a header used by MI code, as its function is now
obsolete. This involves the following pieces:
- Remove it entirely on PowerPC, where it is not used by MD code either
- Remove all references to machine/fdt.h in non-architecture-specific code
(aside from uart_cpu_fdt.c, shared by ARM and MIPS, and so is somewhat
non-arch-specific).
- Fix code relying on header pollution from machine/fdt.h includes
- Legacy fdtbus.c (still used on x86 FDT systems) now passes resource
requests to its parent (nexus). This allows x86 FDT devices to allocate
both memory and IO requests and removes the last notionally MI use of
fdtbus_bs_tag.
- On those architectures that retain a machine/fdt.h, unused bits like
FDT_MAP_IRQ and FDT_INTR_MAX have been removed.

Add #include <machine/fdt.h> to a few files that used to get it via pollution

Enable the mv cesa security/crypto device by providing the required property
in the dts source, and adding the right devices to the kernel config.

Remove dev/fdt/fdt_pci.c, which was code specific to Marvell ARM SoCs,
related to setting up static device mappings. Since it was only used by
arm/mv/mv_pci.c, it's now just static functions within that file, plus
one public function that gets called only from arm/mv/mv_machdep.c.

Switch RPi to using arm_devmap_add_entry() to set up static device mapping.

Allow 'no static device mappings' to potentially work.

Don't try to find a static mapping before calling pmap_mapdev(), that logic
is now part of pmap_mapdev() and doesn't need to be duplicated here.

Switch a10 to using arm_devmap_add_entry() to set up static device mapping.

MFC r260281, r260282, r260283, r260285

Implement OFW_BUS_MAP_INTR() in terms of the FDT PIC table

Reimplement fdt_intr_to_rl() in terms of OFW_BUS_MAP_INTR() and
OFW_BUS_CONFIG_INTR().

Use bus_space_map() rather than pmap_mapdev() in nexus_activate_resource(),
when running on FDT systems. Unmap memory in nexus_deactivate_resource().

Remove fdt_pic_table code from MIPS, PowerPC, and x86, as it is no longer
used by anything.

MFC r260161, r260163, r260165, r260166, r260189

Add polarity and level support to ARM GIC

Do not attach to PCI bridges in AHCI driver

Use only mapped BIOs on ARM

Fix race condition in DELAY for SP804 timer.

MFC r258359, r258742, r258845, r259936, r259640

Apply access flags for managed and unmanaged pages properly on ARMv6/v7

Set the PGA_WRITEABLE flag when the protections indicate write access, not
just when the current access is a write.

Enable missing Access Flag for secondary cores on ARMv6/v7

Add identification and necessary type checks for Krait CPU cores.

MFC r256707, r256708, r257291, r258358

Switch to use WBWA mappings for page tables on armv6, this is needed for SMP.
Fix PTE_SYNC() for PIPT L2 caches, using the virtual address wasn't so useful.
Use PTE_SYNC() for >= armv6
Spell cpu_l2cache_wb_range correctly.

Fix condition that determines PMAP_NEEDS_PTE_SYNC value for ARM

Use values of the correct defines to determine statement's result.
ARM_ARCH_ symbols are always defined, hence only values are relevant.

Avoid clearing EXEC permission bit when setting the page RW on ARMv6/v7

When emulating modified bit the executable attribute was cleared by
mistake when calling pmap_set_prot().

MFC r257170, r257171, r257172, r257240, r257278, r257279, r257280, r257281,
r257282, r257332

Wait for DesignWare UART transfers completion before accessing line control

Enable UART busy detection handling for Armada XP - based board

Enable SATA interface on Armada XP
Run mvs SATA driver on Armada XP instead of old mv_sata

Retire arm_remap_nocache() and the data and constants associated with it.

Remove hard-coded mappings related to Armada XP support

Fix-up DTB for Armada XP registers' base according to the actual settings

Change Armada XP kernel load address to the u-boot's end address

Remove not working and deprecated PJ4Bv6 support

Switch off explicit broadcasting of the TLB flush operations for PJ4B CPU

Add missing ARMv6 CPU functions to ARM Makefile

MFC r257702, r257745, r257746, r257747, r257751, r257791, r257793,
r257794, r257795, r257992

Teach nexus(4) about Open Firmware (e.g. FDT) on ARM and MIPS, retiring
fdtbus in most cases.

Make OF_nextprop() work correctly for FDT by using the libfdt
fdt_next_property_offset() API.

Do not panic if pmap_mincore() is called.

An addendum: it is possible, though of questionable utility, for a node
to have no properties at all.
Add definition for the Atheros 8021 gigabit PHY.

Consolidate Apple firmware hacks and improve them by switching on the
presence of mac-io devices in the tree, which uniquely identifies Apple
hardware.

Allow OF_decode_addr() to also be able to map resources on big-endian
devices.

Make tsec work with the device tree present on the RB800.

Be more flexible about which compatible strings to accept. This brings up
the PCI Express bus on the RB800 using the firmware device tree.

Rename the "bare" platform "mpc85xx", which is what it actually is, and
add actual platform probing based on PVR.

Merge r264984

Retire smp_active. It was racey and caused demonstrated problems with
the cpufreq code. Replace its use with smp_started. There's at least
one userland tool that still looks at the kern.smp.active sysctl, so
preserve it but point it to smp_started as well.

Obtained from: Netflix, Inc.

MFC r258787:
r258780 should not have applied to .S files.

MFC r258779,r258780,r258787,r258822:

Fix undefined behavior: (1 << 31) is not defined as 1 is an int and this
shifts into the sign bit. Instead use (1U << 31) which gets the
expected result.

Similar to the (1 << 31) case it is not defined to do (2 << 30).

This fix is not ideal as it assumes a 32 bit int, but does fix the issue
for most cases.

A similar change was made in OpenBSD.

Revert r252694 from stable/10 to fix instabilities seen with jemalloc + dhclient/sshd.

This is a direct commit to stable/10 as the VM code has changed
since the stable/10 branch.

PR: kern/185046

MFC r257228:
Add bus_dmamap_load_ma() function to load map with the array of
vm_pages.

MFC r258392, r258412:

Call cpu_setup() immediately after the page tables are installed. This
enables data cache and other chip-specific features. It was previously
done via an early SYSINIT, but it was being done after pmap and vm setup,
and those setups need to use mutexes. On some modern ARM platforms,
the ldrex/strex instructions that implement mutexes require the data cache
to be enabled.

Call cpu_setup() from the initarm() routine on platforms that don't use
the common FDT-aware initarm() in arm/machdep.c.

MFC r258240:

In the data abort handler, don't panic if kdb is available and says it
handled the condition.

MFC r257669, r257672, r257673, r257676, r257678:

Call initarm_lastaddr() later in the init sequence, after establishing
static device mappings, rather than as the first of the initializations
that a platform can hook into. This allows a platform to allocate KVA
from the top of the address space downwards for things like static device
mapping, and return the final "last usable address" result after that and
other early init work is done.

Because some platforms were doing work in initarm_lastaddr() that needs to
be done early, add a new initarm_early_init() routine and move the early
init code to that routine on those platforms.

Make PTE_DEVICE a synonym for PTE_NOCACHE on armv4, to make it easier to
share the same code on both architectures.

Add new helper routines for arm static device mapping. The new code
allocates kva space from the top down for the device mappings and builds
entries in an internal table which is automatically used later by
arm_devmap_bootstrap(). The platform code just calls the new
arm_devmap_add_entry() function as many times as it needs to (up to 32
entries allowed; most platforms use 2 or 3 at most).

Remove imx local devmap code and use the essentially identical common
code that got moved from imx_machdep.c to arm/devmap.c.

MFC r257648, r257649, r257660:

Begin reducing code duplication in arm pmap.c and pmap-v6.c by factoring
out common code related to mapping device memory into a new devmap.c file.

Remove the growing duplication of code that used pmap_devmap_find_pa() and
then did some math with the returned results to generate a virtual address,
and likewise in reverse to get a physical address. Now there are a pair
of functions, arm_devmap_vtop() and arm_devmap_ptov(), to do that. The
bus_space_map() implementations are rewritten in terms of these.

Move remaining code and data related to static device mapping into the
new devmap.[ch] files. Emphasize the MD nature of these things by using
the prefix arm_devmap_ on the function and type names (already a few of
these things found their way into MI code, hopefully it will be harder to
do by accident in the future).

MFC r257419:

Do not EOI an interrupt until the point after the filter handlers / before
threaded handlers.

MFC r257203: Eliminate a compiler warning about extraneous parens.

MFC r257201, r257202

Retire arm_remap_nocache() and the data and constants associated with it.

MFC r257199, r257200, r257217:

Remove all #include <machine/pmap.h> from arm code. It's already
included by vm/pmap.h, which is a prerequisite for arm/machine/pmap.h
so there's no reason to ever include it directly.

Remove #include <machine/frame.h> from all the arm code that doesn't
really need it. That would be almost everywhere it was included. Add
it in a couple files that really do need it and were previously getting
it by accident via another header.

Remove the last dregs of trapframe_t. It turns out only arm was using
this type, so remove it to make arm code more consistant with other
platforms.

MFC r256647:

Invalidate the entire L2 cache before enabling it. Say whether it
has been enabled or disabled.

MFC r256638:

Add cases for the combinations of busdma sync op flags that we handle
correctly by doing nothing, then add a panic for the default case, because
that implies that some driver asked for a sync (probably incorrectly) and
nothing was done.

MFC r256637:

When calculating the number of bounce pages needed, round the maxsize
up to a multiple of PAGE_SIZE, and add one page because there can always
be one more boundary crossing than the number of pages in the transfer.

MFC r256628:
Fix a register name typo. The effect was that CPU_CONTROL_AFLT_ENABLE
wasn't being set, but it was almost assuredly already turned on anyway
by the bootloader.

Copy head (r256279) to stable/10 as part of the 10.0-RELEASE cycle.

Approved by: re (implicit)
Sponsored by: The FreeBSD Foundation

- Create kern.ipc.sendfile namespace, and put the new "readhead" OID
there as "kern.ipc.sendfile.readahead".
- Push all nsfbuf related tunables into MD code. Don't move them
to new namespace in favor of POLA.

Reviewed by: scottl
Approved by: re (gjb)

Fix GCC build for all ARMs. Revert bug introduced in r255613.

Previous change applied in r255613 fixed build for ARMv6 but
broke it for previous architecture revisions. This commit
eventually fixes GCC build for all ARM revisions.

Approved by: cognet (mentor)
Approved by: re (kib)

The pmap function pmap_clear_reference() is no longer used. Remove it.

pmap_clear_reference() has had exactly one caller in the kernel for
several years, more precisely, since FreeBSD 8. Now, that call no
longer exists.

Approved by: re (kib)
Sponsored by: EMC / Isilon Storage Division

Fix panic in ktrcapfail() when no capability rights are passed.
While here, correct all consumers to pass NULL instead of 0 as we pass
capability rights as pointers now, not uint64_t.

Reported by: Daniel Peyrolon
Tested by: Daniel Peyrolon
Approved by: re (marius)

Fix GCC build error when building for ARMv6

Apply theravens's idea to move __strong_reference
macros into the proper ifdef section.

Approved by: cognet (mentor)
Approved by: re

Implement pmap_advise() for ARMv6/v7 pmap module

Apply the given advice to the specified range of addresses within the
given pmap. Depending on the advice, clear the referenced and/or
modified flags in each mapping. Superpage within the given range will
be demoted or destroyed.

Reviewed by: alc
Approved by: cognet (mentor)
Approved by: re

Write protect base page after superpage demotion so that it may repromote

When clearing the modification status of the superpage, one of the
base pages produced during demotion should be marked as write disabled.
The intention is that subsequent write access may repromote.
In the current implementation this was done wrong as write permission was
granted instead of forbidden.

Approved by: cognet (mentor)
Approved by: re

Unconditionally compile the __sync_* atomics support functions into compiler-rt
for ARM.
This is quite ugly, because it has to work around a clang bug that does not
allow built-in functions to be defined, even when they're ones that are
expected to be built as part of a library.

Reviewed by: ed

Fix a typo in a comment.

Significantly reduce the cost, i.e., run time, of calls to madvise(...,
MADV_DONTNEED) and madvise(..., MADV_FREE). Specifically, introduce a new
pmap function, pmap_advise(), that operates on a range of virtual addresses
within the specified pmap, allowing for a more efficient implementation of
MADV_DONTNEED and MADV_FREE. Previously, the implementation of
MADV_DONTNEED and MADV_FREE relied on per-page pmap operations, such as
pmap_clear_reference(). Intuitively, the problem with this implementation
is that the pmap-level locks are acquired and released and the page table
traversed repeatedly, once for each resident page in the range
that was specified to madvise(2). A more subtle flaw with the previous
implementation is that pmap_clear_reference() would clear the reference bit
on all mappings to the specified page, not just the mapping in the range
specified to madvise(2).

Since our malloc(3) makes heavy use of madvise(2), this change can have a
measureable impact. For example, the system time for completing a parallel
"buildworld" on a 6-core amd64 machine was reduced by about 1.5% to 2.0%.

Note: This change only contains pmap_advise() implementations for a subset
of our supported architectures. I will commit implementations for the
remaining architectures after further testing. For now, a stub function is
sufficient because of the advisory nature of pmap_advise().

Discussed with: jeff, jhb, kib
Tested by: pho (i386), marcel (ia64)
Sponsored by: EMC / Isilon Storage Division

Introduce superpages support for ARMv6/v7.

Promoting base pages to superpages can increase TLB coverage and allow for
efficient use of page table entries. This development provides FreeBSD/ARM
with superpages management mechanism roughly equivalent to what we have for
i386 and amd64 architectures.

1. Add mechanism for automatic promotion of 4KB page mappings to 1MB section
mappings (and demotion when not needed, respectively).

2. Managed and non-kernel mappings are now superpages-aware.

3. The functionality can be enabled by setting "vm.pmap.sp_enabled" tunable to
a non-zero value (either in loader.conf or by modifying "sp_enabled"
variable in pmap-v6.c file). By default, automatic promotion is currently
disabled.

Submitted by: Zbigniew Bodek <zbb@semihalf.com>
Reviewed by: alc
Sponsored by: The FreeBSD Foundation, Semihalf

Add missing TAILQ initializer (omitted in r250634).

Submitted by: Zbigniew Bodek <zbb@semihalf.com>
Reviewed by: alc
Sponsored by: The FreeBSD Foundation, Semihalf

Revert r251370 as it contains a deadlock.

Add the frame information to cpu_switch to allow us to unwind out of it,
for example when dumping threads in the kernel debugger.

Add the unwind information to irq_entry so we can pass through it when
unwinding the stack.

Revert r254501. Instead, reuse the type stability of the struct pmap
which is the part of struct vmspace, allocated from UMA_ZONE_NOFREE
zone. Initialize the pmap lock in the vmspace zone init function, and
remove pmap lock initialization and destruction from pmap_pinit() and
pmap_release().

Suggested and reviewed by: alc (previous version)
Tested by: pho
Sponsored by: The FreeBSD Foundation

Do not use pv_kva on ARMv6/v7 and save some space on each vm_page. It's only
relevant for older ARM variants (with virtual cache).

Submitted by: Zbigniew Bodek <zbb@semihalf.com>
Reviewed by: gber
Sponsored by: The FreeBSD Foundation, Semihalf

Simplify and clean up pmap_clearbit()

There is no need for calling vm_page_dirty() when clearing "modified" flag as
it is already set for that page in pmap_fault_fixup() or pmap_enter() thanks
to "modified" bit emulation.

Also, there is no need for checking PTE "referenced" or "writeable" flags. If
there is a request to clear a particular flag we should just do it.

Submitted by: Zbigniew Bodek <zbb@semihalf.com>
Reviewed by: gber
Sponsored by: The FreeBSD Foundation, Semihalf

Fix ARMv6/v7 mapping's wired status.

Last input argument in pmap_modify_pv() should be a mask of flags to be set.
In pmap_change_wiring() however, the straight wired status was used, which
does not represent valid flags (and is of type boolean).

This commit fixes the issue so that wired flag is passed to pmap_modify_pv()
properly.

Submitted by: Zbigniew Bodek <zbb@semihalf.com>
Reviewed by: gber
Sponsored by: The FreeBSD Foundation, Semihalf

Clear all L2 PTE protection bits before their configuration.

Revise L2_S_PROT_MASK to include all of the protection bits. Notice that
clearing these bits does not always take away the corresponding permissions
(for example, permission is granted when the bit is cleared). The bits are
cleared but are to be set or left cleared accordingly in pmap_set_prot(),
pmap_enter_locked(), etc.

Clear L2_XN along with L2_S_PROT_MASK in pmap_set_prot() so that all
permissions related bits are cleared before actual configuration.

Submitted by: Zbigniew Bodek <zbb@semihalf.com>
Reviewed by: gber
Sponsored by: The FreeBSD Foundation, Semihalf

Simplify pv_entry removal or ARMv6/v7:

- PGA_WRITEABLE indicates that there *might be* a writable mapping for the
particular page, so to avoid frequent sweeping of the pv_entries whenever
pmap_nuke_pv(), pmap_modify_pv(), etc. is called, it is sufficient to
clear that flag if there are no managed mappings for that page anymore
(notice that only pmap_enter is authorized to set this flag).
- Avoid redundant checking for PVF_WIRED flag when this flag cannot be set
anyway.
- Clear PGA_WRITEABLE only once for each vm_page instead of multiple,
redundant clearing it in loop when there are no writeable mappings
to that page anymore.

Submitted by: Zbigniew Bodek <zbb@semihalf.com>
Reviewed by: gber
Sponsored by: The FreeBSD Foundation, Semihalf

Rename device vfp to option VFP and retire the ARM_VFP_SUPPORT option. This
simplifies enabling as previously both options were required to be enabled,
now we only need a single option.

While here enable VFP on the PandaBoard.

Remove the ARMFPE option. It is unsupported, and appears to be broken as
arm_fpe_core_changecontext is not a function.

Remove unused FPE code. This is not enabled anywhere as it is the only
file I can find containing FAST_FPE. It appears this would not work as
want_resched is not defined anywhere.

Only allocate 2 bounce pages for maps that can only use them for buffers that
are unaligned on cache lines boundary, as we will never need more.

- The address lies in the bus space handle, not in the cookie
- Use the right address when calling kva_free()

The soft and hard busy mechanism rely on the vm object lock to work.
Unify the 2 concept into a real, minimal, sxlock where the shared
acquisition represent the soft busy and the exclusive acquisition
represent the hard busy.
The old VPO_WANTED mechanism becames the hard-path for this new lock
and it becomes per-page rather than per-object.
The vm_object lock becames an interlock for this functionality:
it can be held in both read or write mode.
However, if the vm_object lock is held in read mode while acquiring
or releasing the busy state, the thread owner cannot make any
assumption on the busy state unless it is also busying it.

Also:
- Add a new flag to directly shared busy pages while vm_page_alloc
and vm_page_grab are being executed. This will be very helpful
once these functions happen under a read object lock.
- Move the swapping sleep into its own per-object flag

The KPI is heavilly changed this is why the version is bumped.
It is very likely that some VM ports users will need to change
their own code.

Sponsored by: EMC / Isilon storage division
Discussed with: alc
Reviewed by: jeff, kib
Tested by: gavin, bapt (older version)
Tested by: pho, scottl

Don't bother trying to work around buffers which are not aligned on a cache
line boundary. It has never been 100% correct, and it can't work on SMP,
because nothing prevents another core from accessing data from an unrelated
buffer in the same cache line while we invalidated it. Just use bounce pages
instead.

Reviewed by: ian
Approved by: mux (mentor) (implicit)

Replace kernel virtual address space allocation with vmem. This provides
transparent layering and better fragmentation.

- Normalize functions that allocate memory to use kmem_*
- Those that allocate address space are named kva_*
- Those that operate on maps are named kmap_*
- Implement recursive allocation handling for kmem_arena in vmem.

Reviewed by: alc
Tested by: pho
Sponsored by: EMC / Isilon Storage Division

We no longer need to align the stack before calling swi_handler as it is
already aligned correctly in the PUSHFRAME macro.

Let the platform calculate the timer frequency at runtime, and use that for
the omap4, instead of relying on the (wrong) value provided in the dts.

When entering exception handlers we may not have an aligned stack. This is
because an exception may happen at any time. The stack alignment rules on
ARM EABI state the only place the stack must be 8-byte aligned is on a
function boundary.

If an exception happens while a function is setting up or tearing down it's
stack frame it may not be correctly aligned. There is also no requirement
for it to be when the function is a leaf node.

The fix is to align the stack after we have stored a backup of the old stack
pointer, but before we have stored anything in the trapframe. Along with
this we need to adjust the size of the trapframe by 4 bytes to ensure the
stack below it is also correctly aligned.

Only receive the interrupts on the first core, to avoid duplicate interrupts.

Add identification for Cortex-A7 (R0) cores.

Reviewed by: cognet@

The errata 727915 requires a different workaround for r2p0, we have to
explicitely clean/invalidate every cache line using way/set operations.

Remove useless cache operations.

Fix vfp:
- We should check is_d32 to see howmany registers we have
- In vfp_restore mark vfpscr as an output register

Without the second part it appears we can return the incorrect value from
vfp_bounce if the VFP condition flags are set as it may override the
register holding the return value.

Introduce new structure sfstat for collecting sendfile's statistics
and remove corresponding fields from struct mbstat. Use PCPU counters
and SFSTAT_INC() macro for update these statistics.

Discussed with: glebius

Indent the "scp=... rlv=..." to make it easier to read the backtrace.

Remove trailing whitespaces.

Remove extraneous format string converison specifier

Submitted by: wxs@

Another warning.

armadaxp_idcache_wbinv_all() is in this file.

Fix all the compiler warnings in elf_trampoline.c.

Fix the build with gcc.

Gcc outputs pre-UAL asm and expects the ldcl instruction with a condition
in the form ldc<c>l, where the code produces the instruction in the UAL
form ldcl<c>. Work around this by checking if we are using clang or gcc and
adjusting the instruction.

While here correct the cmp instruction's value to include the # before the
immediate value.

o Make fields names short.
o Slim down reg fields comments.

Remove redundant clearing of the PGA_WRITEABLE flag in
pmap_remove_all()

This flag should already be cleared by pmap_nuke_pv()

Submitted by: Zbigniew Bodek <zbb@semihalf.com>
Sponsored by: The FreeBSD Foundation, Semihalf

Fix modified bit emulation for ARMv6/v7

When doing pmap_enter_locked(), enable write permission only when access
type indicates attempt to write. Otherwise, leave the page read only but
mark it writable in pv_flags.

This will result in:
1. Marking page writable during pmap_enter() but only when ensured that it
will be written right away so that we will not get redundant permissions
fault on write attempt.
2. Keeping page read only when it is permitted to be written but there was
no actual write attempt. Hence, we will get permissions fault on write
access and mark page writable in pmap_fault_fixup() what will indicate
modification status.

Submitted by: Zbigniew Bodek <zbb@semihalf.com>
Sponsored by: The FreeBSD Foundation, Semihalf

Fix one of INVARIANTS-related UMA panics on ARM

Force UMA zone to allocate service structures like slabs using own
allocator. uma_debug code performs atomic ops on uma_slab_t fields
and safety of this operation is not guaranteed for write-back caches

Replace some spaces to tab.

Decrypt magic numbers - define names for fields of Generic Timer's CNTKCTL reg.

Submitted by: Ruslan Bukin <br@bsdpad.com>

In generic_bs_map(), use kmem_alloc_nofault() instead of kmem_alloc(), as we
only need virtual addresses.

Submitted by: alc

Add ARM Generic Timer driver.

Submitted by: Ruslan Bukin <br@bsdpad.com>

o Initialize interrupt groups as Group 0 (secure interrupts).
o Minor cleanup.

Submitted by: Ruslan Bukin <br@bsdpad.com>

Add identification for Cortex-A15 (R0) cores.

Submitted by: Ruslan Bukin <br@bsdpad.com>

Support reading registers r0-r3 when unwinding. There is a seperate
instruction to load these. We only hit it when unwinding past an trap frame
as in C r0-r3 would never have been saved onto the stack.

Add UNWINDSVCFRAME to provide the unwind pseudo ops to allow us to unwind
past a trapframe.

Use this macro in exception_exit as it is the function the unwinder enters
as the functions that store the frame setting lr to point to it.

Make support for atomics on ARM complete.

Provide both __sync_*-style and __atomic_*-style functions that perform
the atomic operations on ARMv5 by using Restartable Atomic Sequences.

While there, clean up some pieces of code where it's sufficient to use
regular uint32_t to store register contents and don't need full reg_t's.
Also sync this back to the MIPS code.

Fix the vfp code to work with the 16 register variants of the VFP unit. We
check which variant we are on, and if it is a VFPv3 or v4, and has 32
double registers we save these. This fixes VFP support on Raspberry Pi.

While here clean fmrx and fmxr up to use the register names from vfp.h
as opposed to the raw register names.

Add C11 atomic fallbacks for ARM.

Basically the situation is as follows:

- When using Clang + armv6, we should not need any intrinsics. It should
support it, even though due to a target misconfiguration it does not.
We should fix this in Clang.
- When using Clang + noarmv6, provide __atomic_* functions that disable
interrupts.
- When using GCC + armv6, we can provide __sync_* intrinsics, similar to
what we did for MIPS. As ARM and MIPS are quite similar, simply base
this implementation on the one I did for MIPS.
- When using GCC + noarmv6, disable the interrupts, like we do for
Clang.

This implementation still lacks functions for noarmv6 userspace. To be
done.

Implement pmap_copy() for ARMv6/v7.

Copy the given range of mappings from the source map to the
destination map, thereby reducing the number of VM faults on fork.

Submitted by: Zbigniew Bodek <zbb@semihalf.com>
Sponsored by: The FreeBSD Foundation, Semihalf

Rework and organize pmap_enter_locked() function.

pmap_enter_locked() implementation was very ambiguous and confusing.
Rearrange it so that each part of the mapping creation is separated.
Avoid walking through the redundant conditions.
Extract vector_page specific PTE setup from normal PTE setting.

Submitted by: Zbigniew Bodek <zbb@semihalf.com>
Sponsored by: The FreeBSD Foundation, Semihalf

Stop using PVF_MOD, PVF_REF & PVF_EXEC flags in pv_entry, use PTE.

Using PVF_MOD, PVF_REF and PVF_EXEC is redundant as we can get the proper
info from PTE bits.
When the mapping is marked as executable and has been referenced we assume
that it has been executed. Similarly, when the mapping is set to be writable
and is referenced, it must have been due to write access to it.
PVF_MOD and PVF_REF flags are kept just for pmap_clearbit() usage,
to pass the information on which bit should be cleared.

Submitted by: Zbigniew Bodek <zbb@semihalf.com>
Sponsored by: The FreeBSD Foundation, Semihalf

Improve, optimize and clean-up ARMv6/v7 memory management related code.

Use pmap_find_pv if needed instead of multiplying its code throughout
pmap-v6.

Avoid possible NULL pointer dereference in pmap_enter_locked()
When trying to get m->md.pv_memattr, make sure that m != NULL,
in particular that vector_page is set to be NULL.

Do not set PGA_REFERENCED flag in pmap_enter_pv().
On ARM any new page reference will result in either entering the new
mapping by calling pmap_enter, etc. or fixing-up the existing mapping in
pmap_fault_fixup().
Therefore we set PGA_REFERENCED flag in the earlier mentioned cases and
setting it later in pmap_enter_pv() is just waste of cycles.

Delete unused pm_pdir pointer from the pmap structure.

Rearrange brackets in the fault cause detection in trap.c
Place the brackets correctly in order to see course of the conditions
instantaneously.

Unify naming in pmap-v6.c and improve style
Use naming common for whole pmap and compatible with other pmaps,
improve style where possible:
pm -> pmap
pg -> m
opg -> om
*pt -> *ptep
*pte -> *ptep
*pde -> *pdep

Submitted by: Zbigniew Bodek <zbb@semihalf.com>
Sponsored by: The FreeBSD Foundation, Semihalf

Switch to AP[2:1] access permissions model. Store "referenced"
bit in PTE.

Enable Access Flag in CPU control. With AF enabled each valid mapping
needs to have referenced bit in PTE set in order to be able to cache
it in the TLB.

AP[0] bit is to be used as reference flag.
All access permissions are encoded by AP[2:1] wherein AP[1] is in fact
"user enable" and AP[2](APX) is "write disable".

All mappings are always set to be valid. Reference emulation is performed
by setting/clearing reference flag in PTE.

md.pvh_attrs are no longer necessary however pv_flags are still being used
for now.

Marking vm_page as "dirty" or "referenced" is being performed on:
- page or flag fault servicing in pmap_fault_fixup(), basing on the fault
type
- vm_fault servicing in pmap_enter() according to the desired protections
and faulty access type
Redundant page marking has been removed as on ARM we know exactly when the
particular page is referenced or is going to be written.

Submitted by: Zbigniew Bodek <zbb@semihalf.com>
Sponsored by: The FreeBSD Foundation, Semihalf

o Relax locking assertions for vm_page_find_least()
o Relax locking assertions for pmap_enter_object() and add them also
to architectures that currently don't have any
o Introduce VM_OBJECT_LOCK_DOWNGRADE() which is basically a downgrade
operation on the per-object rwlock
o Use all the mechanisms above to make vm_map_pmap_enter() to work
mostl of the times only with readlocks.

Sponsored by: EMC / Isilon storage division
Reviewed by: alc

Add a comment explaining why stack_capture is empty for EABI and clang.

While here add a comment pointing out that, while r11 is not the frame
pointer on EABI as there is no frame pointer, it's value is unused so is
safe.

Fix L2 cache write-back invalidate for Sheeva core.

Submitted by: Michal Dubiel
Obtained from: Netasq, Semihalf

Port the new PV entry allocator from amd64/i386/mips to armv6/v7.

PV entries are now roughly half the size.
Instead of using a shared UMA zone for 28 byte pv entries
(two 8-byte tailq nodes, a 4 byte pointer, a 4 byte address and 4 byte
flags), we allocate a page at a time per process.
This provides 252 pv entries per process (actually, per pmap address space)
and eliminates one of the 8-byte tailq entries since we now can track
per-process pv entries implicitly.
The pointer to the pmap can be eliminated by doing address arithmetic to
find the metadata on the page headers to find a single pointer shared by
all 252 entries. There is an 8-int bitmap for the freelist of those 252
entries.
When in serious low memory condition, allocation of another pv_chunk is
possible by freeing some pages in pmap_pv_reclaim().

Added pv_entry/pv_chunk related statistics to pmap.
pv_entry/pv_chunk statistics can be accessed via sysctl vm.pmap.

Ported PTE freelist of KVA allocation and maintenance from i386.
Using an idea from Stephan Uphoff, use the empty pte's that correspond
to the unused kva in the pv memory block to thread a freelist through.
This allows us to free pages that used to be used for pv entry chunks
since we can now track holes in the kva memory block.

As both ARM pmap.c and pmap-v6.c use the same header and pv_entry, pmap and
md_page structures are different, it was needed to separate code designed
for ARMv6/7 from the one for other ARMs.

Submitted by: Zbigniew Bodek <zbb@semihalf.com>
Reviewed by: alc
Sponsored by: The FreeBSD Foundation, Semihalf

Don't use the old stack-walking code with
EABI ARM kernels or clang-compiled ARM kernels.

This fixes a crash seen in clang-compiled ARM
kernels that include WITNESS.

This code could be easily modified to walk the stack
for current clang-generated code (including EABI)
but Andrew Turner has raised concerns that the
stack frame currently emitted by clang isn't actually
required by EABI so such a change might cause problems
down the road.

In case anyone wants to experiment, the change
to support current clang-compiled kernels
involves simply setting FR_RFP=0 and FR_SCP=1.

Fix page reference emulation on ARMv6 and v7

Submitted by: Zbigniew Bodek
Obtained from: Semihalf

Fix L2 PTE access permissions management.

Keep following access permissions:

APX AP Kernel User
1 01 R N
1 10 R R
0 01 R/W N
0 11 R/W R/W

Avoid using reserved in ARMv6 APX|AP settings:
- In case of unprivileged (user) access without permission to write,
the access permission bits were being set to reserved for ARMv6
(but valid for ARMv7) value of APX|AP = 111.

Fix-up faulting userland accesses properly:
- Wrong condition statement in pmap_fault_fixup() caused that
any genuine, unprivileged access was being fixed-up instead of
just skip doing anything and return. Staring from now we ensure
proper reaction for illicit user accesses.

L2_S_PROT_R and L2_S_PROT_U names might be misleading as they do not
reflect real permission levels. It will be clarified in following
patches (switch to AP[2:1] permissions model).

Obtained from: Semihalf

Correct comment about initial VA=>PA mapping

Avoid calling pcpu_init() simultaneously.

pcpu_init() updates queue, so cannot be called by multiple cores
at the same time

Obtained from: Semihalf

Properly initialize Armada XP MP subsystem.

- correct setting of Auxiliary Control Register for MP mode
- correct setting of Auxiliarty Debug registers
- cleanup management of memory contains bootup code
- early initialization of Coherency Fabric (MP and not-MP mode)
- enable Snoop Filtering

Obtained from: Semihalf

Make a debugging printf a little more useful.

Fix comment block formatting.

Insert STOP_UNWINDING directives in the _start (kernel entry point) and
fork_trampoline (thread entry point) assembler routines, because it's
not possible to unwind beyond those points.

Also insert STOP_UNWINDING in the exception_exit routine, to prevent an
unwind-loop at that point. This is just a stopgap until we get around
to instrumenting all assembler functions with proper unwind metadata.

EABI unwinder enhancements... When it's time to stop unwinding, don't
exit the loop until after printing info about the current frame. Also,
if executing the unwind function for a frame doesn't change the values of
any registers, log that and exit the loop rather than looping endlessly.

Add Xilinx Zynq ARM/FPGA SoC support to FreeBSD/arm port.

Submitted by: Thomas Skibo <ThomasSkibo (at) sbcglobal.net>
Tested by: wkoszek (ZedBoard)
Reviewed by: wkoszek, freebsd-arm@ (no objections raised)

Initialize GIC_PMRR register on ARM GIC.

Provided by: Thomas Skibo

Add the hw.floatingpoint sysctl to ARM to tell us if we have vfp support
in the kernel and the hardware includes a vfp unit.

When running on armv6, set alignment checking to modulo-4 mode rather
than modulo-8, because clang emits ldrd and strd instructions for
addresses that are only 4-byte aligned.

Don't check and warn about pmap mismatch on every call to busdma sync.
With some recent busdma refactoring, sometimes it happens that a sync
op gets called when bus_dmamap_load() never got called, which results
in a spurious warning about a map mismatch when no sync operations will
actually happen anyway. Now the check is done only if a sync operation
is actually performed, and the result of the check is a panic, not just
a printf.

Reviewed by: cognet (who prevented me from donning a point hat)

Implement the concept of the unmapped VMIO buffers, i.e. buffers which
do not map the b_pages pages into buffer_map KVA. The use of the
unmapped buffers eliminate the need to perform TLB shootdown for
mapping on the buffer creation and reuse, greatly reducing the amount
of IPIs for shootdown on big-SMP machines and eliminating up to 25-30%
of the system time on i/o intensive workloads.

The unmapped buffer should be explicitely requested by the GB_UNMAPPED
flag by the consumer. For unmapped buffer, no KVA reservation is
performed at all. The consumer might request unmapped buffer which
does have a KVA reserve, to manually map it without recursing into
buffer cache and blocking, with the GB_KVAALLOC flag.

When the mapped buffer is requested and unmapped buffer already
exists, the cache performs an upgrade, possibly reusing the KVA
reservation.

Unmapped buffer is translated into unmapped bio in g_vfs_strategy().
Unmapped bio carry a pointer to the vm_page_t array, offset and length
instead of the data pointer. The provider which processes the bio
should explicitely specify a readiness to accept unmapped bio,
otherwise g_down geom thread performs the transient upgrade of the bio
request by mapping the pages into the new bio_transient_map KVA
submap.

The bio_transient_map submap claims up to 10% of the buffer map, and
the total buffer_map + bio_transient_map KVA usage stays the
same. Still, it could be manually tuned by kern.bio_transient_maxcnt
tunable, in the units of the transient mappings. Eventually, the
bio_transient_map could be removed after all geom classes and drivers
can accept unmapped i/o requests.

Unmapped support can be turned off by the vfs.unmapped_buf_allowed
tunable, disabling which makes the buffer (or cluster) creation
requests to ignore GB_UNMAPPED and GB_KVAALLOC flags. Unmapped
buffers are only enabled by default on the architectures where
pmap_copy_page() was implemented and tested.

In the rework, filesystem metadata is not the subject to maxbufspace
limit anymore. Since the metadata buffers are always mapped, the
buffers still have to fit into the buffer map, which provides a
reasonable (but practically unreachable) upper bound on it. The
non-metadata buffer allocations, both mapped and unmapped, is
accounted against maxbufspace, as before. Effectively, this means that
the maxbufspace is forced on mapped and unmapped buffers separately.
The pre-patch bufspace limiting code did not worked, because
buffer_map fragmentation does not allow the limit to be reached.

By Jeff Roberson request, the getnewbuf() function was split into
smaller single-purpose functions.

Sponsored by: The FreeBSD Foundation
Discussed with: jeff (previous version)
Tested by: pho, scottl (previous version), jhb, bf
MFC after: 2 weeks

Move the __aeabi_unwind_cpp_pr{0,1,2} functions to libkern so they can be
referenced in a non-debug kernel.

Implement the required but unused __aeabi_unwind_cpp_* functions in the
trampoline kernel.

Add an END macro to ARM. This is mostly used to tell gas where the bounds
of the functions are when creating the EABI unwind tables.

Add pmap function pmap_copy_pages(), which copies the content of the
pages around, taking array of vm_page_t both for source and
destination. Starting offsets and total transfer size are specified.

The function implements optimal algorithm for copying using the
platform-specific optimizations. For instance, on the architectures
were the direct map is available, no transient mappings are created,
for i386 the per-cpu ephemeral page frame is used. The code was
typically borrowed from the pmap_copy_page() for the same
architecture.

Only i386/amd64, powerpc aim and arm/arm-v6 implementations were
tested at the time of commit. High-level code, not committed yet to
the tree, ensures that the use of the function is only allowed after
explicit enablement.

For sparc64, the existing code has known issues and a stab is added
instead, to allow the kernel linking.

Sponsored by: The FreeBSD Foundation
Tested by: pho (i386, amd64), scottl (amd64), ian (arm and arm-v6)
MFC after: 2 weeks

Fix a typo where db_printf was spelt printf.

Update how we read the stack pointer to work on both GCC and clang.

Tell the unwinder we can't unwind swi_entry. This fixes an infinite loop
when the kernel attempts to unwind through this function.

The .fnstart and .fnend in this function should be moved to macros but we
are currently missing an END macro on ARM.

__FreeBSD_ARCH_armv6__ is undefined on clang. We can use __ARM_ARCH in
it's place. This makes 'uname -p' correctly output 'armv6' on a kernel
built with clang.

Switch the vm_object mutex to be a rwlock. This will enable in the
future further optimizations where the vm_object lock will be held
in read mode most of the time the page cache resident pool of pages
are accessed for reading purposes.

The change is mostly mechanical but few notes are reported:
* The KPI changes as follow:
- VM_OBJECT_LOCK() -> VM_OBJECT_WLOCK()
- VM_OBJECT_TRYLOCK() -> VM_OBJECT_TRYWLOCK()
- VM_OBJECT_UNLOCK() -> VM_OBJECT_WUNLOCK()
- VM_OBJECT_LOCK_ASSERT(MA_OWNED) -> VM_OBJECT_ASSERT_WLOCKED()
(in order to avoid visibility of implementation details)
- The read-mode operations are added:
VM_OBJECT_RLOCK(), VM_OBJECT_TRYRLOCK(), VM_OBJECT_RUNLOCK(),
VM_OBJECT_ASSERT_RLOCKED(), VM_OBJECT_ASSERT_LOCKED()
* The vm/vm_pager.h namespace pollution avoidance (forcing requiring
sys/mutex.h in consumers directly to cater its inlining functions
using VM_OBJECT_LOCK()) imposes that all the vm/vm_pager.h
consumers now must include also sys/rwlock.h.
* zfs requires a quite convoluted fix to include FreeBSD rwlocks into
the compat layer because the name clash between FreeBSD and solaris
versions must be avoided.
At this purpose zfs redefines the vm_object locking functions
directly, isolating the FreeBSD components in specific compat stubs.

The KPI results heavilly broken by this commit. Thirdy part ports must
be updated accordingly (I can think off-hand of VirtualBox, for example).

Sponsored by: EMC / Isilon storage division
Reviewed by: jeff
Reviewed by: pjd (ZFS specific review)
Discussed with: alc
Tested by: pho