arm64 pmap: Enable L3C promotions by pmap_enter_quick()

More precisely, implement L3C (64KB/2MB, depending on base page size)
promotion in pmap_enter_quick()'s helper function,
pmap_enter_quick_locked(). At the same time, use the recently
introduced flag VM_PROT_NO_PROMOTE from pmap_enter_object() to
pmap_enter_quick_locked() to avoid L3C promotion attempts that will
fail.

Reviewed by: kib
Differential Revision: https://reviews.freebsd.org/D45445

physmem: zero entire array

As a convenience to callers, who might allocate the array on the stack.
An empty/zero-valued range indicates the end of the physmap entries.

Remove the now-redundant calls to bzero() at the call site.

Reviewed by: andrew
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D45318

pmap: Skip some superpage promotion attempts that will fail

Implement a simple heuristic to skip pointless promotion attempts by
pmap_enter_quick_locked() and moea64_enter(). Specifically, when
vm_fault() calls pmap_enter_quick() to map neighboring pages at the end
of a copy-on-write fault, there is no point in attempting promotion in
pmap_enter_quick_locked() and moea64_enter(). Promotion will fail
because the base pages have differing protection.

Reviewed by: kib
Differential Revision: https://reviews.freebsd.org/D45431
MFC after: 1 week

arm64 pmap: Convert panic()s to KASSERT()s

There is no reason for the ATTR_SW_NO_PROMOTE checks in
pmap_update_{entry,strided}() to be panic()s instead of KASSERT()s.

Requested by: markj
Reviewed by: markj
Differential Revision: https://reviews.freebsd.org/D45424

Adjust comments referencing vm_mem_init()

I cannot find a time where the function was not named this.

Reviewed by: kib, markj
MFC after: 3 days
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D45383

arm64: set ATTR_CONTIGUOUS on the DMAP's L2 blocks

On systems configured with 16KB pages, this change creates 1GB page
mappings in the direct map where possible. Previously, the largest page
size that was used to implement the direct map was 32MB. Similarly, on
systems configured with 4KB pages, this change creates 32MB page
mappings, instead of 2MB, in the direct map where 1GB is too large.

Implement demotion on L2C (32MB/1GB) page mappings within the DMAP.

Update sysctl vm.pmap.kernel_maps to report on L2C page mappings.

Reviewed by: markj
Tested by: gallatin, Eliot Solomon <ehs3@rice.edu>
Differential Revision: https://reviews.freebsd.org/D45224

arm64 pmap: eliminate a redundant variable

Moreover, if we attempt an L2 promotion on the kernel pmap from
pmap_enter_quick_locked(), this change eliminates the recomputation of
the L2 entry's address.

MFC after: 1 week

arm64_pmap: narrow scope of bti_same test

The pmap_bti_same test in pmap_enter_l3c only happens in the
!ADDR_IS_KERNEL case; in the other case, a KASSERT fails. So move the
test into that case to save a bit of time when ADDR_IS_KERNEL.

Reviewed by: andrew
Differential Revision: https://reviews.freebsd.org/D45160

arm64: add page-to-pte convenience macros

Define macros to perform pte to vm_page and vm_page to pte conversions
without composing two macros, and use the convenience macros wherever
possible.

Reviewed by: alc
Differential Revision: https://reviews.freebsd.org/D44699

arm64 pmap: Add ATTR_CONTIGUOUS support [Part 3]

Introduce L3C promotion of base page mappings. When the base page size
is 4KB, use ATTR_CONTIGUOUS to promote 16 aligned, contiguous base page
mappings to a 64KB mapping. Alternatively, when the base page size is
16KB, use ATTR_CONTIGUOUS to promote 128 aligned, contiguous base page
mappings to a 2MB mapping.

Given the frequency of L3C counter updates, switch to per-CPU counters
to avoid cache line ping ponging.

Revise the L3C counter descriptions to reflect the fact that the size
of an L3C mapping varies depending on the base page size.

Co-authored-by: Eliot Solomon <ehs3@rice.edu>
Reviewed by: markj
Differential Revision: https://reviews.freebsd.org/D44983

arm64 pmap: Add ATTR_CONTIGUOUS support [Part 2]

Create ATTR_CONTIGUOUS mappings in pmap_enter_object(). As a result,
when the base page size is 4 KB, the read-only data and text sections
of large (2 MB+) executables, e.g., clang, can be mapped using 64 KB
pages. Similarly, when the base page size is 16 KB, the read-only
data section of large executables can be mapped using 2 MB pages.

Rename pmap_enter_2mpage(). Given that we have grown support for 16 KB
base pages, we should no longer include page sizes that may vary, e.g.,
2mpage, in pmap function names. Requested by: andrew

Co-authored-by: Eliot Solomon <ehs3@rice.edu>
Differential Revision: https://reviews.freebsd.org/D44575

arm64: correctly handle a failed BTI check in pmap_enter_l2()

If pmap_enter_l2() does not create a mapping because the BTI check
fails, then we should release the reference on the page table page
acquired from pmap_alloc_l2(). Otherwise, the page table page will
never be reclaimed.

arm64 pmap: Add ATTR_CONTIGUOUS support [Part 1]

The ATTR_CONTIGUOUS bit within an L3 page table entry designates that
L3 page as being part of an aligned, physically contiguous collection
of L3 pages. For example, 16 aligned, physically contiguous 4 KB pages
can form a 64 KB superpage, occupying a single TLB entry. While this
change only creates ATTR_CONTIGUOUS mappings in a few places,
specifically, the direct map and pmap_kenter{,_device}(), it adds all
of the necessary code for handling them once they exist, including
demotion, protection, and removal. Consequently, new ATTR_CONTIGUOUS
usage can be added (and tested) incrementally.

Modify the implementation of sysctl vm.pmap.kernel_maps so that it
correctly reports the number of ATTR_CONTIGUOUS mappings on machines
configured to use a 16 KB base page size, where an ATTR_CONTIGUOUS
mapping consists of 128 base pages.

Additionally, this change adds support for creating L2 superpage
mappings to pmap_kenter{,_device}().

Reviewed by: markj
Tested by: gallatin
Differential Revision: https://reviews.freebsd.org/D42737

arm64: Switch the address argument to cpu_*cache* to a pointer

No functional change, but this reduces diffs with CheriBSD downstream.

Reviewed by: andrew
Sponsored by: University of Cambridge, Google, Inc.
Differential Revision: https://reviews.freebsd.org/D44342

arm64: Add BTI support to pmap

Add a rangeset to the arm64 pmap to describe which address space needs
the Branch Target Identification (BTI) Guard Page flag set in the page
table.

On hardware that supports BTI the Guard Page flag tells the hardware
to raise an exception if the target of a BR* and BLR* instruction is
not an appropriate landing pad instruction.

To support this in userspace we need to know which address space
should be guarded. For this add a rangeset to the arm64 pmap when the
hardware supports BTI. The kernel can then use pmap_bti_set and
pmap_bti_clear mark and unmark which address space is guarded.

Sponsored by: Arm Ltd
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D42328

arm64: Add pmap integration for KMSAN

- In pmap_bootstrap_san(), allocate the root PTPs for the shadow maps.
(For KASAN, this is done earlier since we need to do some special
bootstrapping for the kernel stack.)
- Adjust ifdefs to include KMSAN.
- Expand the shadow maps when pmap_growkernel() is called.

MFC after: 2 weeks
Sponsored by: Klara, Inc.
Sponsored by: Juniper Networks, Inc.
Differential Revision: https://reviews.freebsd.org/D43405

arm64: Simplify and improve KASAN shadow map bootstrapping

- Move pmap_bootstrap_allocate_kasan_l2() close to the place where it is
actually used.
- Simplify pmap_bootstrap_allocate_kasan_l2() a bit: eliminate some
unneeded variables and zero and exclude each 2MB mapping as we go
rather than doing that all at once. Excluded regions will be
coalesced.
- As a consequence of the previous point, ensure that we do not zero a
preexisting 2MB mapping.
- Simplify pmap_bootstrap_san() and prepare it to work with KMSAN.

MFC after: 2 weeks
Sponsored by: Klara, Inc.
Sponsored by: Juniper Networks, Inc.
Differential Revision: https://reviews.freebsd.org/D43404

arm64: Disable kernel superpage promotion when KMSAN is configured

The break-before-make operation required to promote or demote a
superpage leaves a window where the KMSAN runtime can trigger a fatal
data abort. More specifically, the code in pmap_update_entry() which
executes after ATTR_DESCR_VALID is cleared may implicitly attempt to
access KMSAN context via curthread, but we may be promoting or demoting
a 2MB page containing the curthread structure.

Reviewed by: imp
Sponsored by: Klara, Inc.
Sponsored by: Juniper Networks, Inc.
Differential Revision: https://reviews.freebsd.org/D43158

arm64: Define shadow maps for KMSAN

Both are the same size as the kernel map.

Reviewed by: imp
MFC after: 2 weeks
Sponsored by: Klara, Inc.
Sponsored by: Juniper Networks, Inc.
Differential Revision: https://reviews.freebsd.org/D43154

pmap: Convert boolean_t to bool.

Reviewed by: kib (older version)
Differential Revision: https://reviews.freebsd.org/D39921

arm64: Remove pmap_san_bootstrap() and call kasan_init_early() directly

pmap_san_bootstrap() doesn't really do much, and it was hard-coding the
the bootstrap stack size defined in locore.S. Moreover, the name is a
bit confusing given the existence of pmap_bootstrap_san(). Just remove
it and call kasan_init_early() directly like we do on amd64. It will
not be used by KMSAN in a forthcoming patch series.

No functional change intended.

MFC after: 1 week
Sponsored by: Klara, Inc.
Sponsored by: Juniper Networks, Inc.
Differential Revision: https://reviews.freebsd.org/D43403

arm64: Remove an unused global variable

No functional change intended.

MFC after: 1 week
Sponsored by: Klara, Inc.
Sponsored by: Juniper Networks, Inc.

sys: Remove ancient SCCS tags.

Remove ancient SCCS tags from the tree, automated scripting, with two
minor fixup to keep things compiling. All the common forms in the tree
were removed with a perl script.

Sponsored by: Netflix

arm64: Set the Guarded Page flag in the kernel

Now the kernel and modules are built with branch protection we can
enablethe Guarded Page flag in the page tables. This causes indirect
branches to a location without a correct landing pad instruction to
raise an exception.

This should help mitigate some attacks where a function pointer is
changed to point somewhere other than the start of the function,
however it doesn't stop an attacker pointing it to an unintended
function.

Reviewed by: alc, scottph (both earlier version), markj
Sponsored by: Arm Ltd
Sponsored by: The FreeBSD Foundation (earlier version)
Differential Revision: https://reviews.freebsd.org/D42080

arm64: Make kern_delta unneeded in the boot params

Use pmap_early_vtophys to translate from a virtual to physical where
we were previously using the calculated delta. This means that, while
we still calculate it, we don't need to pass it to initarm or either
pmap bootstrap functions.

While here remove an unneeded printf that indirectly used it or was
related to the previous printf.

Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D42567

arm64: Use pmap_early_vtophys in pmap_bootstrap_san

Use pmap_early_vtophys to find the physical address of the kernel base
rather than using the calculated offset as it will be removed in a
latter commit.

Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D42566

pmap: add pmap_kextract(9) man page

Add a man page for pmap_kextract(9), with alias to vtophys(9). This man
page is based on pmap_extract(9).

Add it as cross reference in pmap(9), and add comments above the
function implementations.

Co-authored-by: Graham Perrin <grahamperrin@gmail.com>
Co-authored-by: mhorne
Sponsored by: The FreeBSD Foundation
Pull Request: https://github.com/freebsd/freebsd-src/pull/827

arm64: Add a leaf PTP when pmap_enter(psind=1) creates a wired mapping

Let pmap_enter_l2() create wired mappings. In particular, allocate a
leaf PTP for use during demotion. This is a step towards reverting
commit 64087fd7f372.

Reviewed by: alc, markj
Sponsored by: Google, Inc. (GSoC 2023)
MFC after: 2 weeks
Differential Revision: https://reviews.freebsd.org/D41634

sys: Remove $FreeBSD$: one-line .c pattern

Remove /^[\s*]*__FBSDID\("\$FreeBSD\$"\);?\s*\n/

arm64 pmap: Eliminate unnecessary TLB invalidations

Eliminate unnecessary TLB invalidations by pmap_kenter(),
pmap_qenter(), and pmap_mapbios() when the old page table entries
were invalid.

While I'm here, correct some nearby whitespace issues.

MFC after: 2 weeks

arm64 pmap: Retire PMAP_INLINE

Neither of the remaining callers to pmap_kremove() warrant inlining.
Those calls rarely occur. In other words, we were optimizing for the
uncommon case.

MFC after: 1 week

arm64 pmap: Eliminate some duplication of code

pmap_unmapbios() can simply call pmap_kremove_device() rather than
duplicating its code.

While I'm here, add a comment to pmap_kremove_device() explaining its
proper use, and fix a whitespace issue.

MFC after: 1 week

arm64/riscv pmap: Initialize the pmap's pm_pvchunk field

I believe that there are two reasons that the missing TAILQ
initialization operations haven't caused a problem. First, the TAILQ
head's first field is being initialized to zeroes elsewhere. Second,
the first access to the TAILQ head's last field is by
TAILQ_INSERT_HEAD(), which assigns to the last field without reading
it when the first field is NULL.

Reviewed by: kib, markj
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D41118

vm_radix_init: use initializer

Several vm_radix tries are not initialized with vm_radix_init. That
works, for now, since static initialization zeroes the root field
anyway, but if initialization changes, these tries will fail. Add
missing initializer calls.

Reviewed by: alc, kib, markj
Differential Revision: https://reviews.freebsd.org/D40971

arm64 pmap: Tidy up pmap_promote_l2() calls

Since pmap_ps_enabled() is true by default, check it inside of
pmap_promote_l2() instead of at every call site.

Modify pmap_promote_l2() to return true if the promotion succeeded and
false otherwise.

(A similar change was applied to the amd64 pmap in 0d2f98c2f092.)

Reviewed by: kib, markj
Differential Revision: https://reviews.freebsd.org/D40781

vm_phys: add binary segment search

Replace several sequential searches for a segment that contains a
phyiscal address with a call to a function that does it by binary
search. In vm_page_reclaim_contig_domain_ext, find the first segment
to reclaim from, and reclaim from each subsequent appropriate segment.
Eliminate vm_phys_scan_contig.

Reviewed by: alc, markj
Differential Revision: https://reviews.freebsd.org/D40058

amd64/arm64 pmap: Stop requiring the accessed bit for superpage promotion

Stop requiring all of the PTEs to have the accessed bit set for superpage
promotion to occur. Given that change, add support for promotion to
pmap_enter_quick(), which does not set the accessed bit in the PTE that
it creates.

Since the final mapping within a superpage-aligned and sized region of a
memory-mapped file is typically created by a call to pmap_enter_quick(),
we now achieve promotions in circumstances where they did not occur
before, for example, the X server's read-only mapping of libLLVM-15.so.

See also https://www.usenix.org/system/files/atc20-zhu-weixi_0.pdf

Reviewed by: kib, markj
MFC after: 2 weeks
Differential Revision: https://reviews.freebsd.org/D40478

arm64: Correct a pmap unlock in pmap_stage2_fault

This is used by bhyve so was not an issue as it is still in development.
Sponsored by: Arm Ltd

arm64 pmap: Make VM_PAGE_TO_PV_LIST_LOCK() a constant-time operation

The prior implementation of VM_PAGE_TO_PV_LIST_LOCK() performed a
linear-time search of the vm_phys_segs[] array. However, in contrast to
PHYS_TO_PV_LIST_LOCK(), that search is unnecessary because every (non-
fictitious) vm_page contains the index of the vm_phys_seg in which it
resides.

Change most of the remaining uses of CHANGE_PV_LIST_LOCK_TO_PHYS() and
PHYS_TO_PV_LIST_LOCK() to CHANGE_PV_LIST_LOCK_TO_VM_PAGE() and
VM_PAGE_TO_PV_LIST_LOCK(), respectively.

Collectively, these changes also reduce the size of a GENERIC-NODEBUG
kernel's pmap.

Before:

text data bss dec hex filename
70144 3200 2248 75592 0x12748 pmap.o

After:

text data bss dec hex filename
69192 3200 2248 74640 0x12390 pmap.o

Reviewed by: kib, markj
Differential Revision: https://reviews.freebsd.org/D40306

arm64 pmap: Eliminate an unused global variable

The global variable "pmap_last_pa" was copied from the amd64 pmap as a
part of commit c15085278cb5 "arm64 pmap: implement per-superpage locks"
but it is neither used nor needed by the arm64 pmap.

arm64 pmap: introduce PHYS_TO_PTE macro

Introduce macro for PHYS_TO_PTE, setting the groundwork for future
support of various Arm VMSA extensions.

For extensions such as 52-bit VA/PA (FEAT_LPA2), the representation of
an address between a PTE and PA are not equivalent. This macro will
allow converting between the different representations.

Currently PHYS_TO_PTE is a NOP. Replace all instances where we go from
PA to PTE with new PHYS_TO_PTE macro.

Reviewed by: markj
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D39828

arm64 pmap: introduce PTE_TO_PHYS macro

Introduce macro for PTE_TO_PHYS, setting the groundwork for future
support of various Arm VMSA extensions.

For extensions such as 52-bit VA/PA (FEAT_LPA2), the representation of
an address between a PTE and PA are not equivalent. This macro will
allow converting between the different representations.

Currently going from PTE to PA is achieved by masking off the upper and
lower attributes. Retain this behaviour but replace all instances with
the new macro instead.

Reviewed by: alc, markj
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D39827

pmap_{un}map_io_transient: Use bool instead of boolean_t.

Reviewed by: imp, kib
Differential Revision: https://reviews.freebsd.org/D39920

Split out pmap_map_delete on arm64

This will be used when supporting some extensions, e.g. Branch Target
Identification (BTI).

Sponsored by: Arm Ltd

Remove unneeded masks from the arm64 KASAN shadow

When mapping the arm64 KASAN shadow map we use Ln_TABLE_MASK to align
physical addresses, however these should already be aligned either
by rounding to a greater alignment, or the VM subsystem is giving us
a correctly aligned page.

Remove these extra alignment masks.

Reviewed by: kevans
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D39752

Create pmap_mask_set_locked on arm64

Create a locked version of pmap_mask_set. We will need this for BTI
support.

Sponsored by: Arm Ltd

Add non-posted device memory to the arm64 mem map

Add VM_MEMATTR_DEVICE_NP to the arm64 vm.pmap.kernel_maps sysctl.

Reviewed by: markj
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D39371

arm64: add KASAN support

This entails:
- Marking some obvious candidates for __nosanitizeaddress
- Similar trap frame markings as amd64, for similar reasons
- Shadow map implementation

The shadow map implementation is roughly similar to what was done on
amd64, with some exceptions. Attempting to use available space at
preinit_map_va + PMAP_PREINIT_MAPPING_SIZE (up to the end of that range,
as depicted in the physmap) results in odd failures, so we instead
search the physmap for free regions that we can carve out, fragmenting
the shadow map as necessary to try and fit as much as we need for the
initial kernel map. pmap_bootstrap_san() is thus after
pmap_bootstrap(), which still included some technically reserved areas
of the memory map that needed to be included in the DMAP.

The odd failure noted above may be a bug, but I haven't investigated it
all that much.

Initial work by mhorne with additional fixes from kevans and markj.

Reviewed by: andrew, markj
Sponsored by: Juniper Networks, Inc.
Sponsored by: Klara, Inc.
Differential Revision: https://reviews.freebsd.org/D36701

arm64: Support stage 2 mappings in pmap_remove_all

This has been hit when testing bhyve.

Sponsored by: Arm Ltd

Support the arm64 pmap_remove_write for stage 2

The fields we need to adjust are different in stage 1 and stage 2
tables. Handle this by adding variables to hold the bits to check,
set, and clear.

Reviewed by: alc
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D37399

Support arm64 stage2 TLB invalidation

To invalidate stage 2 mappings on arm64 we may need to call into the
hypervisor so add a function pointer that bhyve can use to implement
this.

Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D37254

arm64 pmap: Fix a buffer overrun initializing per-superpage locks.

pmap_init_pv_table makes a first pass over the memory segments to
compute the amount of address space needed to allocate per-superpage
locks. It then makes a second pass over each segment allocating
domain-local memory to back the pages for the locks belonging to each
segment. This second pass rounds each segment's allocation up to a
page size since the domain-local allocation has to be a multiple of
pages. However, the first pass was only doing a single round of the
total page counts up at the end not accounting for the padding present
in each segment. To fix, apply the rounding in each segment in the
first pass instead of just at the end.

While here, tidy the second pass a bit by trimming some
not-quite-right logic copied from amd64. In particular, compute pages
directly at the start of the loop iteration to more closely match the
first loop. Then, drop an always-false condition as 'end' was
computed as 'start + pages' where 'start == highest + 1'. Thus, the
actual condition being tested was 'if (highest >= highest + 1 +
pages)'. Finally, remove 'highest' entirely by keep the result of the
'pvd' increment in the existing loop.

Reported by: CHERI (overflow)
Reviewed by: markj
Sponsored by: DARPA
Differential Revision: https://reviews.freebsd.org/D38377

pmap: standardize promotion conditions between amd64 and arm64

On amd64, don't abort promotion due to a missing accessed bit in a
mapping before possibly write protecting that mapping. Previously,
in some cases, we might not repromote after madvise(MADV_FREE) because
there was no write fault to trigger the repromotion. Conversely, on
arm64, don't pointlessly, yet harmlessly, write protect physical pages
that aren't part of the physical superpage.

Don't count aborted promotions due to explicit promotion prohibition
(arm64) or hardware errata (amd64) as ordinary promotion failures.

Reviewed by: kib, markj
MFC after: 2 weeks
Differential Revision: https://reviews.freebsd.org/D36916

arm64/pmap: freed only used for PV_STATS

When PV_STATS is defined, freed is used. Otherwise it isn't. Mark it as
__pvused and define __pvused appropriately.

Sponsored by: Netflix
Reviewed by: tsoome, rpokala, andrew
Differential Revision: https://reviews.freebsd.org/D37438

arm64 pmap: implement per-superpage locks

As with amd64 pmap introduce per-superpage locks backed by pages
allocated by their respective domains.

This significiantly reduces lock contantion from pmap when running
poudriere on a 160 core Ampere Altra server

Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D36310

Rename the arm64 pmap_invalidate_* functions

These all work on stage 1 tables. Rename them so we can add similar
functions that operate on stage 2 tables.

Reviewed by: alc, markj, kib
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D37302

Disable superpage use for stage 2 arm64 mappings

When modifying a stage 2 mapping we may need to call into the
hypervisor to invalidate the TLB. Until it is known if the cost of
this operation is less than the performance gains superpages offers
disable their use.

Reviewed by: kib. markj
Sponsored by: Innovate UK
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D37299

arm64: Disable per-thread stack-smashing protection in data_abort()

With PERTHREAD_SSP configured, the compiler's stack-smashing protection
uses a per-thread canary value instead of a global value. The value is
stored in td->td_md.md_canary; the sp_el0 register always contains a
pointer to that value, and certain functions selected by the compiler
will store the canary value on the stack as a part of the function
prologue (and will verify the copy as part of the epilogue). In
particular, the thread structure may be accessed.

This happens to occur in data_abort(), which leads to the same problem
addressed by commit 2c10be9e06d4 ("arm64: Handle translation faults for
thread structures"). This commit fixes that directly, by disabling SSP
in data_abort() and a couple of related functions by using a function
attribute. It also moves the update of sp_el0 out of C code in case
the compiler decides to start checking the canary in pmap_switch()
someday.

A different solution might be to move the canary value to the PCB, which
currently lives on the kernel stack and isn't subject to the same
problem as thread structures (if only because guard pages inhibit
superpage promotion). However, there isn't any particular reason the
PCB has to live on the stack today; on amd64 it is embedded in struct
thread, reintroducing the same problem. Keeping the reference canary
value at the top of the stack is also rather dubious since it could be
clobbered by a sufficiently large stack overflow.

A third solution could be to go back to the approach of commit
5aa5420ff2e8, and modify UMA to use the direct map for thread structures
even if KASAN is enabled. But, transient promotions and demotions in
the direct map are possible too.

Reviewed by: alc, kib, andrew
MFC after: 1 month
Sponsored by: Juniper Networks, Inc.
Sponsored by: Klara, Inc.
Differential Revision: https://reviews.freebsd.org/D37255

sys: Consolidate common implementation details of PV entries.

Add a <sys/_pv_entry.h> intended for use in <machine/pmap.h> to
define struct pv_entry, pv_chunk, and related macros and inline
functions.

Note that powerpc does not yet use this as while the mmu_radix pmap
in powerpc uses the new scheme (albeit with fewer PV entries in a
chunk than normal due to an used pv_pmap field in struct pv_entry),
the Book-E pmaps for powerpc use the older style PV entries without
chunks (and thus require the pv_pmap field).

Suggested by: kib
Reviewed by: kib
Sponsored by: DARPA
Differential Revision: https://reviews.freebsd.org/D36685

amd64, arm64 pmap: fix a comment typo

There is no such error code.

Fixes: 1d5ebad06c20b ("pmap: optimize MADV_WILLNEED on existing superpages")

arm64: Simplify initialization of pc_freemask.

Sponsored by: DARPA
Differential Revision: https://reviews.freebsd.org/D36502

pmap: optimize MADV_WILLNEED on existing superpages

Specifically, avoid pointless calls to pmap_enter_quick_locked() when
madvise(MADV_WILLNEED) is applied to an existing superpage mapping.

Reported by: mhorne
Reviewed by: kib, markj
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D36801

arm64 pmap: per-domain pv chunk list

As with amd64 use a per-domain pv chunk lock to reduce contention as
chunks get created and removed all the time.

Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D36307

arm64 pmap: batch chunk removal in pmap_remove_pages

As with amd64 batch chunk removal in pmap_remove_pages to move it out
of the pv list lock. This is one of the main contested locks when
running poudriere on a 160 core Ampere Altra server.

Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D36305

Reorder pmap_bootstrap_state to reduce holes

Reduce holes in pmap_bootstrap_state by moving freemempos after the
pointers as they are more likely to change size in any future ABI.

Sponsored by: The FreeBSD Foundation

Remove unneeded variables in the arm64 pmap bootstrap

These are now unneeded after cleaning up the pmap bootstrap process.
Remove them and the variables that set them.

Sponsored by: The FreeBSD Foundation

Remove duplicate arm64 pmap bootstrap code

The table bootstrap functions can now be used for non-DMAP uses. Use
them for all early page table creation.

Sponsored by: The FreeBSD Foundation

Allow changing arm64 table attributes when bootstrapping pmap

Only the DMAP region can be mapped with PXN. Allow the table attributes
to be changed for other mappings.

Sponsored by: The FreeBSD Foundation

Allow the arm64 pmap table bootstrap to work in more places

Rework the pmap_bootstrap table generation so we can use it with
partially filled out page tables after the DMAP has been bootstrapped.
This allows it to be reused by the later bootstrap code.

Sponsored by: The FreeBSD Foundation

Make the arm64 pmap bootstrap state global

So it can be reused by KASAN

Sponsored by: The FreeBSD Foundation

Rename arm64 pmap bootstrap to not be dmap specific

This will be used by KASAN and possibly the kernel in general, rename
so we aren't confused by it.

Sponsored by: The FreeBSD Foundation

arm64: Fix an assertion in pmap_enter_largepage()

The intent is to assert that either no mapping exists at the given VA,
or that the existing L1 block mapping maps the same PA.

Fixes: 36f1526a598c ("Add experimental 16k page support on arm64")
Reviewed by: alc
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D36698

arm64: Handle 1GB mappings in pmap_enter_quick_locked()

Reviewed by: alc, kib
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D36697

arm64: Ignore 1GB mappings in pmap_advise()

For the same reason as commit 4c224f8e5f36cfad5a9af8db7c7acdecc3d4c7b5.

MFC after: 1 week

kmem_malloc/free: Use void * instead of vm_offset_t for kernel pointers.

Reviewed by: kib, markj
Sponsored by: DARPA
Differential Revision: https://reviews.freebsd.org/D36549

pmap_unmapdev/bios: Accept a pointer instead of a vm_offset_t.

This matches the return type of pmap_mapdev/bios.

Reviewed by: kib, markj
Sponsored by: DARPA
Differential Revision: https://reviews.freebsd.org/D36548

pmap: don't recompute mpte during promotion

When attempting to promote 4KB user-space mappings to a 2MB user-space
mapping, the address of the struct vm_page representing the page table
page that contains the 4KB mappings is already known to the caller.
Pass that address to the promotion function rather than making the
promotion function recompute it, which on arm64 entails iteration over
the vm_phys_segs array by PHYS_TO_VM_PAGE(). And, while I'm here,
eliminate unnecessary arithmetic from the calculation of the first PTE's
address on arm64.

MFC after: 1 week

arm64 pmap: Convert PC_IS_FREE into an inline function.

This permits inlining the comparisons even in the 16K page case.
Note that since PC_FREEN is -1, values can be compared efficiently
without having to fetch words of pc_freemask from memory via the
'cmn <reg>, #0x1' instruction.

Reviewed by: markj
Sponsored by: DARPA
Differential Revision: https://reviews.freebsd.org/D36218

arm64 pmap: Simplify logic around pv_chunk sizes.

- Define PC_FREEL and _NPCM in terms of _NPCPV rather than via magic
numbers.

- Remove assertions about _NPC* values from pmap.c. This is less
relevant now that PC_FREEL and _NPCM are derived from _NPCPV.

- Add a helper inline function pc_is_full() which uses a loop to check
if pc_map is all zeroes. Use this to replace three places that
check for a full mask assuming there are only 3 entries in pc_map.

Reviewed by: markj
Sponsored by: DARPA
Differential Revision: https://reviews.freebsd.org/D36217

Disable promotion on pcpu memory on arm64

We need to be careful to not promote or demote the memory containing
the per-CPU structures as the exception handlers will dereference it
so any time it's invalid may cause recursive exceptions.

Add a new pmap function to set a flag in the pte marking memory that
cannot be promoted or demoted and use it to mark pcpu memory.

Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D35434

Add experimental 16k page support on arm64

Add initial 16k page support on arm64. It is considered experimental,
with no guarantee of compatibility with a userspace or kernel modules
built with the current a 4k page size as code will likely try to pass
in a too small size when working with APIs that take a multiple of a
page, e.g. mmap.

As this is experimental, and because userspace and the kernel need to
have the PAGE_SIZE macro kept in sync there is no kernel option to
enable this. To test a new image should be built with the
PAGE_{SIZE,SHIFT,MASK} macros changed to the 16k versions.

There are currently known issues with loading modules from an old
loader as it can misalign them to load on a non-16k boundary.

Testing has shown good results in kernel workloads that allocate and
free large amounts of memory as only a quarter of the number of calls
into the VM subsystem are needed in the best case.

Reviewed by: markj
Tested by: gallatin
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D34793

Add arm64 CnP support

Set the Common not Private bit in the ttbr registers when supported on
arm64. This tells the hardware it can share the translation table
entries on multiple CPUs.

Reviewed by: alc, kib
Sponsored by: The FreeBSD Foundation

Move the arm64 DMAP creation to C

To simplify the creation of the direct map (DMAP) region on arm64 move
it from the pre-C code into pmap. This simplifies the DMAP creation
as we can use the notmal index macros, and should reduce the number
of pages needed to hold the level 1 tables to just those needed.

Reviewed by: alc, dch
Tested by: dch, kevans
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D34568

arm64: pmap: Mask VA operand in TLBI instructions

Bits 43:0 of the TLBI operand are bits 55:12 of the VA. Leaving
bits 63:55 of the VA in bits 51:44 of the operand might wind up
setting the TTL field (47:44) and accidentally restricting which
translation levels are flushed in the TLB.

Reviewed By: andrew
MFC after: 3 days
Sponsored by: Ampere Computing
Differential Revision: https://reviews.freebsd.org/D34664

Fix arm64 TLB invalidation with non-4k pages

When using 16k or 64k pages atop will shift the address by more than
the needed amount for a tlbi instruction. Replace this with a new macro
to shift the address by 12 and use PAGE_SIZE in the for loop to let the
code work with any page size.

Reviewed by: alc, markj
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D34516

Remove a redundant L1_BLOCK == L2_BLOCK check

We have two checks for L1_BLOCK == L2_BLOCK. Remove one.

Sponsored by: The FreeBSD Foundation

Fix calculating l0index in _pmap_alloc_l3 on arm64

When moving from the l1 index to l0 index we need to use the l1 shift
value not the l0 shift value. With 4k pages they are identical, however
with 16k pages we only have 2 l0 entries so the shift value is incorrect.

Reviewed by: alc, markj
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D34517

Remove an unneeded memset from the arm64 pmap

There is no need to zero pagetable_dmap as we already did it when
creating the page tables in locore.S

Sponsored by: The FreeBSD Foundation

Split out creating the arm64 L2 dmap entries

When creating the DMAP region we may need to create level 2 page table
entries at the start and end of a block of memory. The code to do this
was almost identical so we can merge into a single function.

Sponsored by: The FreeBSD Foundation

Remove l1ptfrom pmap_early_vtophys on arm64

The first argument was unused as we use an address translation
instruction to get the physical address.

Sponsored by: The FreeBSD Foundation

Add arm64 pointer authentication support

Pointer authentication allows userspace to add instructions to insert
a Pointer Authentication Code (PAC) into a register based on an address
and modifier and check if the PAC is correct. If the check fails it will
either return an invalid address or fault to the kernel.

As many of these instructions are a NOP when disabled and in earlier
revisions of the architecture this can be used, for example, to sign
the return address before pushing it to the stack making Return-oriented
programming (ROP) attack more difficult on hardware that supports them.

The kernel manages five 128 bit signing keys: 2 instruction keys, 2 data
keys, and a generic key. The instructions then use one of these when
signing the registers. Instructions that use the first four store the
PAC in the register being signed, however the instructions that use the
generic key store the PAC in a separate register.

Currently all userspace threads share all the keys within a process
with a new set of userspace keys being generated when executing a new
process. This means a forked child will share its keys with its parent
until it calls an appropriate exec system call.

In the kernel we allow the use of one of the instruction keys, the ia
key. This will be used to sign return addresses in function calls.
Unlike userspace each kernel thread has its own randomly generated.

Thread0 has a static key as does the early code on secondary CPUs.
This should be safe as there is minimal user interaction with these
threads, however we could generate random keys when the Armv8.5
Random number generation instructions are present.

Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D31261

arm64: Implement final level only TLB invalidations

A feature of arm64's instruction for TLB invalidation is the ability
to determine whether cached intermediate entries, i.e., L{0,1,2}_TABLE
entries, are invalidated in addition to the final entry, e.g., an
L3_PAGE entry.

Update pmap_invalidate_{page,range}() to support both types of
invalidation, allowing the caller to determine which type of
invalidation is performed.

Update the callers to request the appropriate type of invalidation.

Eliminate redundant TLB invalidations in pmap_abort_ptp() and
pmap_remove_l3_range().

Add a comment to pmap_invalidate_all() making clear that it always
invalidates entries at all levels.

As expected, these changes result in a tiny yet measurable
performance improvement.

Reviewed by: kib, markj
MFC after: 3 weeks
Differential Revision: https://reviews.freebsd.org/D33705

arm64: Simplify pmap_ts_referenced

Use pmap_pte_exists() in place of multiple KASSERT()s.

Eliminate an unnecessary NULL check.

MFC after: 1 week

arm64: Enhance pmap_pte_exists()'s error reporting

Report the descriptor type and level at which the page table does not
match the caller's expectations.

MFC after: 1 week

Fix pmap_is_prefaultable() on arm64 and riscv

The current implementations never correctly return TRUE. In all cases,
when they currently return TRUE, they should have returned FALSE. And,
in some cases, when they currently return FALSE, they should have
returned TRUE. Except for its effects on performance, specifically,
additional page faults and pointless calls to pmap_enter_quick() that
abort, this error is harmless. That is why it has gone unnoticed.

Add a comment to the amd64, arm64, and riscv implementations
describing how their return values are computed.

Reviewed by: kib, markj
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D33659

arm64: Fix "set-but-not-used" warnings in the pmap

MFC after: 1 week

arm64: Introduce and use pmap_pte_exists()

Use pmap_pte_exists() instead of pmap_pte() when the caller expects a
mapping to exist at a particular level. The caller benefits in two
ways from using pmap_pte_exists(). First, because the level is
specified to pmap_pte_exists() as a constant, rather than returned, the
compiler can specialize the implementation of pmap_pte_exists() to the
caller's exact needs, i.e., generate fewer instructions. Consequently,
within a GENERIC-NODEBUG kernel, 704 bytes worth of instructions are
eliminated from the inner loops of various pmap functions. Second,
suppose that the mapping doesn't exist. Rather than requiring every
caller to implement its own KASSERT()s to report missing mappings, the
caller can optionally have pmap_pte_exists() provide the KASSERT().

Reviewed by: andrew, kib
Tested by: andrew (an earlier version)
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D33597

Don't fail changing props for unmapped DMAP memory

When recursing in pmap_change_props_locked we may fail because there is
no pte. This shouldn't be considered a fail as it may happen in a few
cases, e.g. there are multiple normal memory ranges with device memory
between them.

Reported by: cperciva
Tested by: cperciva
Reviewed by: alc, kib
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D33459

Only change DMAP props on DMAP covered memory

When changing memory properties in the arm64 pmap we need to keep both
the kernel address and DMAP mappings in sync.

To keep the kernel and DMAP memory in sync we recurse when updating the
former to also update the latter. There was insuffucuent checking around
this recursion. It would check if the virtual address is not within the
DMAP region, but not if the physical address is covered.

Add the missing check as without it the recursion may return an error.

Sponsored by: The FreeBSD Foundation

Fix a set but not used warning in the arm64 pmap

In pmap_ts_referenced we read the virtual address from pv->pv_va,
but then continue to use the pv struct to get the same value later
in the function.

Use the virtual address value we initially read rather than loading
from the pv struct each time.

Handle table attributes in the arm64 kernel map

When getting the arm64 kernel maps sysctl we should look at the table
attributes as well as the block/page attributes. These attributes are
different to the last level attributes so need to be translated.

The previous code assumed the table and block/page attributes are
identical, however this is not the case. Handle the difference by
extracting the code into new helper functions & calling them as needed
based on the entry type being checked.

Reviewed by: markj
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D33321

Teach vm.pmap.kernel_maps about both XN bits

The arm64 vm.pmap.kernel_maps sysctl would only check the kernel XN bit
when printing the kernel mapping. It can also be useful to check none
of the mappings allow userspace to execute from a given virtual address.
To check for this add the user XN bit when getting the kernel maps.

While here fix the ATTR_S1_AP_USER check to use ATTR_S1_AP to shift the
value to the correct offset.

Reviewed by: kib (earlier version), markj
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D33304

Per-thread stack canary on arm64

With the update to llvm 13 we are able to tell the compiler it can find
the SSP canary relative to the register that holds the userspace stack
pointer. As this is unused in most of the kernel it can be used here
to point to a per-thread SSP canary.

As the kernel could be built with an old toolchain, e.g. when upgrading
from 13, add a warning that the options was enabled but the compiler
doesn't support it to both the build and kernel boot.

Discussed with: emaste
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D33079

Use the vm_radix_init() helper when initializing pmaps

No functional change intended.

Reviewed by: alc, kib
MFC after: 1 week
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D32527

Convert vm_page_alloc() callers to use vm_page_alloc_noobj().

Remove page zeroing code from consumers and stop specifying
VM_ALLOC_NOOBJ. In a few places, also convert an allocation loop to
simply use VM_ALLOC_WAITOK.

Similarly, convert vm_page_alloc_domain() callers.

Note that callers are now responsible for assigning the pindex.

Reviewed by: alc, hselasky, kib
MFC after: 1 week
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D31986

Only demote when needed in the arm64 pmap_change_props_locked

When changing page table properties there is no need to demote a
level 1 or level 2 block if we are changing the entire memory range the
block is mapping. In this case just change the block directly.

Reported by: alc, kib, markj
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D32339

Add pmap_change_prot on arm64

Support changing the protection of preloaded kernel modules by
implementing pmap_change_prot on arm64 and calling it from
preload_protect.

Reviewed by: alc (previous version)
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D32026

Add the arm64 table attributes and use them

Add the table page table attributes on arm64 and use them to add
restrictions to the block and page entries below them. This ensures
we are unable to increase the permissions in these last level entries
without also changing them in the upper levels.

Use the attributes to ensure the kernel can't execute from userspace
memory and vice versa, userspace has no access to read or write kernel
memory, and that the DMAP region is non-executable.

Reviewed by: alc, kib
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D32081

Start to clean up arm64 address space selection

On arm64 we should use bit 55 of the address to decide if aan address
is a user or kernel address. Add a new macro with this check and a
second to ensure the address is in teh canonical form, i.e.
the top bits are all zero or all one.

This will help with supporting future cpu features, including Top
Byte Ignore, Pointer Authentication, and Memory Tagging.

Reviewed by: kib
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D31179

arm64: Sync icache when creating executable superpage mappings

Reviewed by: andrew, kib, markj
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D31181

Assert that valid PTEs are not overwritten when installing a new PTP

amd64 and 32-bit ARM already had assertions to this effect. Add them to
other pmaps.

Reviewed by: alc, kib
MFC after: 2 weeks
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D31171

Clear the accessed bit when copying a managed superpage mapping

pmap_copy() is used to speculatively create mappings, so those mappings
should not have their access bit preset.

Reviewed by: kib, markj
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D31162

pmap: Micro-optimize pmap_remove_pages() on amd64 and arm64

Reduce the live ranges for three variables so that they do not span the
call to PHYS_TO_VM_PAGE(). This enables the compiler to generate
slightly smaller machine code.

Reviewed by: kib, markj
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D31161

arm64: Simplify fcmpset failure in pmap_promote_l2()

When the initial fcmpset in pmap_promote_l2() fails, there is no need
to repeat the check for the physical address being 2MB aligned or for
the accessed bit being set. While the pmap is locked the hardware can
only transition the accessed bit from 0 to 1, and we have already
determined that it is 1 when the fcmpset fails.

MFC after: 1 week

On a failed fcmpset don't pointlessly repeat tests

In a few places, on a failed compare-and-set, both the amd64 pmap and
the arm64 pmap repeat tests on bits that won't change state while the
pmap is locked. Eliminate some of these unnecessary tests.

Reviewed by: andrew, kib, markj
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D31014

arm64: a few simplications to pmap_remove_{all,write}

Eliminate some unnecessary unlocking and relocking when we have to retry
the operation to avoid deadlock. (All of the other pmap functions that
iterate over a PV list already implemented retries without these same
unlocking and relocking operations.)

Avoid a pointer dereference by using an existing local variable that
already holds the desired value.

Eliminate some unnecessary repetition of code on a failed fcmpset.
Specifically, there is no point in retesting the DBM bit because it
cannot change state while the pmap lock is held.

Reviewed by: kib, markj
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D30931

arm64: fix a potential KVA leak in pmap_demote_l1()

In the unlikely event that the 1 GB page mapping being demoted is used
to access the L1 page table page containing the 1 GB page mapping and
the vm_page_alloc() to allocate a new L2 page table page fails, we
would leak a page of kernel virtual address space. Fix this leak.

MFC after: 1 week

arm64: make it possible to define PV_STATS

Remove an #if 0 that results in a compilation error if PV_STATS is
defined. Aside from this #if 0, there is nothing wrong with the
PV_STATS code.

MFC after: 1 week

arm64: replace pa_to_pvh() with page_to_pvh() in pmap_remove_l2()

Revise pmap_remove_l2() to use the constant-time function page_to_pvh()
instead of the linear-time function pa_to_pvh().

Reviewed by: kib, markj
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D30876

arm64: remove an unneeded test from pmap_clear_modify()

The page table entry for a 4KB page mapping must be valid if a PV entry
for the mapping exists, so there is no point in testing each page table
entry's validity when iterating over a PV list.

Reviewed by: kib, markj
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D30875

arm64: Use page_to_pvh() when the vm_page_t is known

When support for a sparse pv_table was added, the implementation of
pa_to_pvh() changed from a simple constant-time calculation to iterating
over the array vm_phys_segs[]. To mitigate this issue, an alternative
function, page_to_pvh(), was introduced that still runs in constant time
but requires the vm_page_t to be known. However, three cases where the
vm_page_t is known were not converted to page_to_pvh(). This change
converts those three cases.

Reviewed by: kib, markj
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D30832

arm64: Fix pmap_copy()'s handling of 2MB mappings

When copying mappings from parent to child, we clear the accessed and
dirty bits. This is done for both 4KB and 2MB PTEs. However,
pmap_demote_l2() asserts that writable superpages must be dirty. This
is to avoid races with the MMU setting the dirty bit during promotion
and demotion. pmap_copy() can create clean, writable superpage
mappings, so it violates this assertion.

Modify pmap_copy() to preserve the accessed and dirty bits when copying
2MB mappings, like we do on amd64.

Fixes: ca2cae0b4dd
Reported by: Jenkins via mhorne
Reviewed by: alc, kib
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D30643

arm64: Use the right PTE when downgrading perms in pmap_promote_l2()

When promoting a run of small mappings to a superpage, we have to
downgrade clean, writable mappings to read-only, to handle the
possibility that the MMU will concurrently mark one of the mappings as
dirty.

The code which performed this operation for the first PTE in the run
used the wrong PTE pointer. As a result, the comparison would always
fail, aborting the promotion. This only occurs when promoting writable,
clean mappings.

Fixes: ca2cae0b4dd
Reviewed by: alc, kib
MFC after: 1 week
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D30642

Clean up early arm64 pmap code

Early in the arm64 pmap code we need to translate between a virtual
address and a physical address. Rather than manually walking the page
table we can ask the hardware to do it for us.

Reviewed by: kib, markj
Sponsored by: Innovate UK
Differential Revision: https://reviews.freebsd.org/D30357

Move IOMMU code to a separate pmap module and switch ARM System MMU
driver to use it.

Add ARM Mali Txxx (Midgard), Gxx (Bifrost) GPU page management code.

Sponsored by: UKRI

Add a VA_IS_CLEANMAP() macro.

This macro returns true if a provided virtual address is contained
in the kernel's clean submap.

In CHERI kernels, the buffer cache and transient I/O map are allocated
as separate regions. Abstracting this check reduces the diff relative
to FreeBSD. It is perhaps slightly more readable as well.

Reviewed by: kib
Obtained from: CheriBSD
Sponsored by: DARPA
Differential Revision: https://reviews.freebsd.org/D28710

Extract the logic from pmap_kextract

This allows us to use it when we only need to check if the virtual address
is valid. For example when checking if an address in the DMAP region is
mapped.

Reviewed by: kib, markj
Sponsored by: Innovate UK
Differential Revision: https://reviews.freebsd.org/D27621

Skip the vm.pmap.kernel_maps sysctl by default.

This sysctl node can generate very verbose output, so don't trigger it
for sysctl -a or sysctl vm.pmap.

Reviewed by: markj, kib
Differential Revision: https://reviews.freebsd.org/D27504

Allow the creation of 3 level page tables on arm64

The stage 2 arm64 page tables may need to start at a lower level. This
is because we may only be able to map a limited IPA range and trying
to use a full 4 levels will cause the CPU to fault in an unrecoverable
way.

To simplify the code we still allocate the full 4 levels, however level 0
will only ever be used to find the level 1 table used as the base. Handle
this by creating a dummy entry in the level 0 table to point to the level 1
table.

Sponsored by: Innovate UK
Differential Revision: https://reviews.freebsd.org/D26066

Add routines for ARM System MMU (SMMU) pmap management.

Reviewed by: markj
Discussed with: kib
Sponsored by: DARPA, Innovate UK
Differential Revision: https://reviews.freebsd.org/D26877

arm64: Remove a racy KASSERT from pmap_remove_pages()

PCPU_GET(curpmap) expands to multiple instructions on arm64, and if the
current thread is migrated in between execution of those instructions, a
stale value may be used in the assertion condition.

Diagnosed by: mmel
Reported by: mmel, Bob Prohaska <fbsd@www.zefox.net>
Submitted by: alc
MFC after: 1 week

Avoid dump_avail[] redefinition.

Move dump_avail[] extern declaration and inlines into a new header
vm/vm_dumpset.h. This fixes default gcc build for mips.

Reviewed by: alc, scottph
Tested by: kevans (previous version)
Sponsored by: The FreeBSD Foundation
Differential revision: https://reviews.freebsd.org/D26741

Add largepage support to the arm64 pmap.

Reviewed by: alc, kib
Sponsored by: Juniper Networks, Inc., Klara, Inc.
Differential Revision: https://reviews.freebsd.org/D26466

arm64/pmap: Sparsify pv_table

Reviewed by: markj, kib
Approved by: scottl (implicit)
MFC after: 1 week
Sponsored by: Ampere Computing, Inc.
Differential Revision: https://reviews.freebsd.org/D26132

Weaken assertions in pmap_l1_to_l2() and pmap_l2_to_l3().

pmap_update_entry() will temporarily clear the valid bit of page table
entries in order to satisfy the arm64 pmap's break-before-make
constraint. pmap_kextract() may operate concurrently on kernel page
table pages, introducing windows where the assertions added in r365879
may fail incorrectly since they implicitly assert that the valid bit is
set. Modify the assertions to handle this.

Reviewed by: andrew, mmel (previous version)
Reviewed by: alc, kib
Reported by: mmel, scottph
MFC with: r365879

Assert we are not traversing through superpages in the arm64 pmap.

Reviewed by: alc, andrew
MFC after: 1 week
Sponsored by: Juniper Networks, Inc., Klara, Inc.
Differential Revision: https://reviews.freebsd.org/D26465

Include the psind in data returned by mincore(2).

Currently we use a single bit to indicate whether the virtual page is
part of a superpage. To support a forthcoming implementation of
non-transparent 1GB superpages, it is useful to provide more detailed
information about large page sizes.

The change converts MINCORE_SUPER into a mask for MINCORE_PSIND(psind)
values, indicating a mapping of size psind, where psind is an index into
the pagesizes array returned by getpagesizes(3), which in turn comes
from the hw.pagesizes sysctl. MINCORE_PSIND(1) is equal to the old
value of MINCORE_SUPER.

For now, two bits are used to record the page size, permitting values
of MAXPAGESIZES up to 4.

Reviewed by: alc, kib
Sponsored by: Juniper Networks, Inc.
Sponsored by: Klara, Inc.
Differential Revision: https://reviews.freebsd.org/D26238

arm64: clean up empty lines in .c and .h files

Support stage 2 arm64 pmap in more places

Add support for stage 2 pmap to pmap_pte_dirty, pmap_release, and more
of pmap_enter. This adds support in all placess I have hit while testing
bhyve ehile faulting pages in as needed.

Sponsored by: Innovate UK
Differential Revision: https://reviews.freebsd.org/D26065

Use EFI memory map to determine attributes for Acpi mappings on arm64.

AcpiOsMapMemory is used for device memory when e.g. an _INI method wants
to access physical memory, however, aarch64 pmap_mapbios is hardcoded to
writeback. Search for the correct memory type to use in pmap_mapbios.

Submitted by: Greg V <greg_unrelenting.technology>
Differential Revision: https://reviews.freebsd.org/D25201

Support pmap_extract_and_hold on arm64 stage 2 mappings

Sponsored by: Innovate UK
Differential Revision: https://reviews.freebsd.org/D24469

Support creating and using arm64 pmap at stage 2

Add minimal support for creating stage 2 IPA -> PA mappings. For this we
need to:

- Create a new vmid set to allocate a vmid for each Virtual Machine
- Add the missing stage 2 attributes
- Use these in pmap_enter to create a new mapping
- Handle stage 2 faults

The vmid set is based on the current asid set that was generalised in
r358328. It adds a function pointer for bhyve to use when the kernel needs
to reset the vmid set. This will need to call into EL2 and invalidate the
TLB.

The stage 2 attributes have been added. To simplify setting these fields
two new functions are added to get the memory type and protection fields.
These are slightly different on stage 1 and stage 2 tables. We then use
them in pmap_enter to set the new level 3 entry to be stored.

The D-cache on all entries is cleaned to the point of coherency. This is
to allow the data to be visible to the VM. To allow for userspace to load
code when creating a new executable entry an invalid entry is created. When
the VM tried to use it the I-cache is invalidated. As the D-cache has
already been cleaned this will ensure the I-cache is synchronised with the
D-cache.

When the hardware implements a VPIPT I-cache we need to either have the
correct VMID set or invalidate it from EL2. As the host kernel will have
the wrong VMID set we need to call into EL2 to clean it. For this a second
function pointer is added that is called when this invalidation is needed.

Sponsored by: Innovate UK
Differential Revision: https://reviews.freebsd.org/D23875

Simplify arm64's pmap_bootstrap() a bit.

locore constructs an L2 page mapping the kernel and preloaded data
starting a KERNBASE (the same as VM_MIN_KERNEL_ADDRESS on arm64).
initarm() and pmap_bootstrap() use the preloaded metadata to
tell it where it can start allocating from.

pmap_bootstrap() currently iterates over the L2 page to find the last
valid entry, but doesn't do anything with the result. Remove the loop
and zap some now-unused local variables.

MFC after: 2 weeks
Sponsored by: Juniper Networks, Klara Inc.
Differential Revision: https://reviews.freebsd.org/D24559

Convert arm's physmem interface to MI code

The arm_physmem interface found in arm's MD code provides a convenient
set of routines for adding/excluding physical memory regions and
initializing important kernel globals such as Maxmem, realmem,
phys_avail[], and dump_avail[]. It is especially convenient for FDT
systems, since we can use FDT parsing functions and pass the result
directly to one of these physmem routines. This interface is already in
use on arm and arm64, and can be used to simplify this early
initialization on RISC-V as well.

This requires only a couple trivial changes:
- Move arm_physmem_kernel_addr to arm/machdep.c. It is unused on arm64,
and manipulated entirely in arm MD code.
- Convert arm32_btop/arm64_btop to atop. This is equivalently defined
on all architectures.
- Drop the "arm" prefix.

Reviewed by: manu, emaste ("looks reasonable")
MFC after: 2 weeks
Differential Revision: https://reviews.freebsd.org/D24153

Mark more nodes as CTLFLAG_MPSAFE or CTLFLAG_NEEDGIANT (17 of many)

r357614 added CTLFLAG_NEEDGIANT to make it easier to find nodes that are
still not MPSAFE (or already are but aren’t properly marked).
Use it in preparation for a general review of all nodes.

This is non-functional change that adds annotations to SYSCTL_NODE and
SYSCTL_PROC nodes using one of the soon-to-be-required flags.

Mark all obvious cases as MPSAFE. All entries that haven't been marked
as MPSAFE before are by default marked as NEEDGIANT

Approved by: kib (mentor, blanket)
Commented by: kib, gallatin, melifaro
Differential Revision: https://reviews.freebsd.org/D23718

Generalise the arm64 ASID allocator.

The requirements of an Address Space ID allocator and a Virtual Machine ID
allocator are similar. Generalise the former code so it can be used with
the latter.

Reviewed by: alc (previous version)
Sponsored by: Innovate UK
Differential Revision: https://reviews.freebsd.org/D23831

Start to support multiple stages in the arm64 pmap.

On arm64 the stage 1 and stage 2 pte formats are similar enough we can
reuse the pmap code for both. As they are only similar and not identical
we need to know if we are managing stage 1 or stage 2 tables.

Add an enum to store this information and a check to make sure it is
set to stage 1 when we manage stage 1 pte fields.

Sponsored by: Innovate UK
Differential Revision: https://reviews.freebsd.org/D23830

Split out the stage 1 pte bits and add the stage 2 bits

In preperation for adding bhyve support to arm64 we need to split the
stage 1 and stage 2 pte fields to allow future changes to create stage 2
page tables.

MFC after: 1 month
Sponsored by: Innovate UK
Differential Revision: https://reviews.freebsd.org/D23669

sysctl: mark more nodes as MPSAFE

vm.kvm_size and vm.kvm_free are read only and marked as MPSAFE on i386
already. Mark them as that on amd64 and arm64 too to avoid locking Giant.

Reviewed by: kib (mentor)
Approved by: kib (mentor)
Differential Revision: https://reviews.freebsd.org/D23039

When a copy-on-write fault occurs, pmap_enter() is called on to replace the
mapping to the old read-only page with a mapping to the new read-write page.
To destroy the old mapping, pmap_enter() must destroy its page table and PV
entries and invalidate its TLB entry. This change simply invalidates that
TLB entry a little earlier, specifically, on amd64 and arm64, before the PV
list lock is held.

Reviewed by: kib, markj
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D23027

Determine whether the MMU hardware is capable of updating a page table
entry's access flag and dirty state, and enable this feature when it's
available.

Ensure that we don't overlook a dirty state update that is concurrent
with a call to pmap_enter(). (Previously, all dirty state updates would
have occurred with the containing pmap's lock held, so a page table entry's
dirty state could not have changed while pmap_enter() held that same lock.)

Reviewed by: andrew, markj
MFC after: 2 weeks
Differential Revision: https://reviews.freebsd.org/D22907

Correct a mistakenly inverted condition in r355833.

Noticed by: kib
X-MFC with: r355833

When pmap_enter_{l2,pde}() are called to create a kernel mapping, they are
incrementing (and decrementing) the ref_count on kernel page table pages.
They should not do this. Kernel page table pages are expected to have a
fixed ref_count. Address this problem by refactoring pmap_alloc{_l2,pde}()
and their callers. This also eliminates some duplicated code from the
callers.

Correctly implement PMAP_ENTER_NOREPLACE in pmap_enter_{l2,pde}() on kernel
mappings.

Reduce code duplication by defining a function, pmap_abort_ptp(), for
handling a common error case.

Handle a possible page table page leak in pmap_copy(). Suppose that we are
determining whether to copy a superpage mapping. If we abort because there
is already a mapping in the destination pmap at the current address, then
simply decrementing the page table page's ref_count is correct, because the
page table page must have a ref_count > 1. However, if we abort because we
failed to allocate a PV entry, this might be a just allocated page table
page that has a ref_count = 1, so we should call pmap_abort_ptp().

Simplify error handling in pmap_enter_quick_locked().

Reviewed by: kib, markj (an earlier)
MFC after: 2 weeks
Differential Revision: https://reviews.freebsd.org/D22763

Apply a small optimization to pmap_remove_l3_range(). Specifically, hoist a
PHYS_TO_VM_PAGE() operation that always returns the same vm_page_t out of
the loop. (Since arm64 is configured as VM_PHYSSEG_SPARSE, the
implementation of PHYS_TO_VM_PAGE() is more costly than that of
VM_PHYSSEG_DENSE platforms, like amd64.)

MFC after: 1 week

Introduce vm_page_astate.

This is a 32-bit structure embedded in each vm_page, consisting mostly
of page queue state. The use of a structure makes it easy to store a
snapshot of a page's queue state in a stack variable and use cmpset
loops to update that state without requiring the page lock.

This change merely adds the structure and updates references to atomic
state fields. No functional change intended.

Reviewed by: alc, jeff, kib
Sponsored by: Netflix, Intel
Differential Revision: https://reviews.freebsd.org/D22650

On a context switch, handle the possibility that the old thread was
preempted after an "ic" or "tlbi" instruction but before it performed a
"dsb" instruction. The "ic" and "tlbi" instructions have unusual
synchronization requirements. If the old thread migrates to a new
processor, its completion of a "dsb" instruction on that new processor does
not guarantee that the "ic" or "tlbi" instructions performed on the old
processor have completed.

This issue is not restricted to the kernel. Since locore.S sets the UCI bit
in SCTLR, user-space programs can perform "ic ivau" instructions (as well as
some forms of the "dc" instruction).

Reviewed by: andrew, kib, markj, mmel
X-MFC with: r355145
Differential Revision: https://reviews.freebsd.org/D22622

Use the VM_MEMATTR macros to describe the MAIR offsets.

Remove the duplicate macros that defined a subset of the VM_MEMATTR values.
While here use VM_MEMATTR macros when filling in the MAIR register.

Reviewed by: alc, markj
Sponsored by: DARPA, AFRL
Differential Revision: https://reviews.freebsd.org/D22241

There is no reason why we need to pin the underlying thread to its current
processor in pmap_invalidate_{all,page,range}(). These functions are using
an instruction that broadcasts the TLB invalidation to every processor, so
even if a thread migrates in the middle of one of these functions every
processor will still perform the required TLB invalidations.

Reviewed by: andrew, markj
MFC after: 10 days
Differential Revision: https://reviews.freebsd.org/D22502

Until every possible root cause for an "invalid ASID" assertion failure is
resolved, assign every pmap a valid ASID when it is first initialized.

Achieve two goals at once: (1) Avoid an unnecessary broadcast TLB
invalidation in reclaim_pv_chunk(). (2) Prevent an "invalid ASID" assertion
failure in reclaim_pv_chunk(). The detailed explanation for this change is
provided by r354792.

X-MFC with: r354792

Implement vm.pmap.kernel_maps for arm64.

Reviewed by: alc
MFC after: 2 weeks
Differential Revision: https://reviews.freebsd.org/D22142

Let arm64 pmap_qenter() and pmap_kenter() unconditionally set NX.

As on amd64, there is no need for mappings created by these functions to
be executable.

Reviewed by: alc, andrew
MFC after: 2 weeks
Differential Revision: https://reviews.freebsd.org/D22141

Achieve two goals at once: (1) Avoid an unnecessary broadcast TLB
invalidation in pmap_remove_all(). (2) Prevent an "invalid ASID" assertion
failure in pmap_remove_all().

The architecture definition specifies that the TLB will not cache mappings
that don't have the "AF" bit set, so pmap_remove_all() needn't issue a TLB
invalidation for mappings that don't have the "AF" bit set.

We allocate ASIDs lazily. Specifically, we don't allocate an ASID for a
pmap until we are activating it. Now, consider what happens on a fork().
Before we activate the child's pmap, we use pmap_copy() to copy mappings
from the parent's pmap to the child's. These new mappings have their "AF"
bits cleared. Suppose that the page daemon decides to reclaim a page that
underlies one of these new mappings. Previously, the pmap_invalidate_page()
performed by pmap_remove_all() on a mapping in the child's pmap would fail
an assertion because that pmap hasn't yet been assigned an ASID. However,
we don't need to issue a TLB invalidation for such mappings because they
can't possibly be cached in the TLB.

Reported by: bob prohaska <fbsd@www.zefox.net>
Reviewed by: markj
MFC after: 1 week
X-MFC-before: r354286
Differential Revision: https://reviews.freebsd.org/D22388

Eliminate a redundant pmap_load() from pmap_remove_pages().

There is no reason why the pmap_invalidate_all() in pmap_remove_pages()
must be performed before the final PV list lock release. Move it past
the lock release.

Eliminate a stale comment from pmap_page_test_mappings(). We implemented
a modified bit in r350004.

MFC after: 1 week

Utilize ASIDs to reduce both the direct and indirect costs of context
switching. The indirect costs being unnecessary TLB misses that are
incurred when ASIDs are not used. In fact, currently, when we perform a
context switch on one processor, we issue a broadcast TLB invalidation that
flushes the TLB contents on every processor.

Mark all user-space ("ttbr0") page table entries with the non-global flag so
that they are cached in the TLB under their ASID.

Correct an error in pmap_pinit0(). The pointer to the root of the page
table was being initialized to the root of the kernel-space page table
rather than a user-space page table. However, the root of the page table
that was being cached in process 0's md_l0addr field correctly pointed to a
user-space page table. As long as ASIDs weren't being used, this was
harmless, except that it led to some unnecessary page table switches in
pmap_switch(). Specifically, other kernel processes besides process 0 would
have their md_l0addr field set to the root of the kernel-space page table,
and so pmap_switch() would actually change page tables when switching
between process 0 and other kernel processes.

Implement a workaround for Cavium erratum 27456 affecting ThunderX machines.
(I would like to thank andrew@ for providing the code to detect the affected
machines.)

Address integer overflow in the definition of TCR_ASID_16.

Setup TCR according to the PARange and ASIDBits fields from
ID_AA64MMFR0_EL1. Previously, TCR_ASID_16 was unconditionally set.

Modify build_l1_block_pagetable so that lower attributes, such as ATTR_nG,
can be specified as a parameter.

Eliminate some unused code.

Earlier versions were tested to varying degrees by: andrew, emaste, markj

MFC after: 3 weeks
Differential Revision: https://reviews.freebsd.org/D21922

Fix pmap_change_attr() on arm64 to allow KV addresses

Altough in the comment above the pmap_change_attr() it was mentioned
that VA could be in KV or DMAP memory space. However,
pmap_change_attr_locked() was accepting only the values inside the DMAP
memory range.

To fix that, the condition check was changed so also the va inside the
KV memory range would be accepted.

The sample use case that wasn't supported is the PCI Device that has the
BAR which should me mapped with the Write Combine attribute - for
example BAR2 of the ENA network controller on the A1 instances on AWS.

Tested on A1 AWS instance and changed ENA BAR2 mapped resource to be
write-combined memory region.

Differential Revision: https://reviews.freebsd.org/D22055
MFC after: 2 weeks
Submitted by: Michal Krawczyk <mk@semihalf.com>
Obtained from: Semihalf
Sponsored by: Amazon, Inc.

Set the userspace execute never bit on kernel mappings.

Arm64 allows us to create execute only mappings. To make sure userspace is
unable to accidentally execute kernel code set the user execute never
bit in the kernel page tables.

MFC after: 1 week
Sponsored by: DARPA, AFRL

Remove page locking from pmap_mincore().

After r352110 the page lock no longer protects a page's identity, so
there is no purpose in locking the page in pmap_mincore(). Instead,
if vm.mincore_mapped is set to the non-default value of 0, re-lookup
the page after acquiring its object lock, which holds the page's
identity stable.

The change removes the last callers of vm_page_pa_tryrelock(), so
remove it.

Reviewed by: kib
Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D21823

(6/6) Convert pmap to expect busy in write related operations now that all
callers hold it.

This simplifies pmap code and removes a dependency on the object lock.

Reviewed by: kib, markj
Tested by: pho
Sponsored by: Netflix, Intel
Differential Revision: https://reviews.freebsd.org/D21596

(3/6) Add a shared object busy synchronization mechanism that blocks new page
busy acquires while held.

This allows code that would need to acquire and release a very large number
of page busy locks to use the old mechanism where busy is only checked and
not held. This comes at the cost of false positives but never false
negatives which the single consumer, vm_fault_soft_fast(), handles.

Reviewed by: kib
Tested by: pho
Sponsored by: Netflix, Intel
Differential Revision: https://reviews.freebsd.org/D21592

In short, pmap_enter_quick_locked("user space", ..., VM_PROT_READ) doesn't
work. More precisely, it doesn't set ATTR_AP(ATTR_AP_USER) in the page
table entry, so any attempt to read from the mapped page by user space
generates a page fault. This problem has gone unnoticed because the page
fault handler, vm_fault(), will ultimately call pmap_enter(), which
replaces the non-working page table entry with one that has
ATTR_AP(ATTR_AP_USER) set.

This change reduces the number of page faults during a "buildworld" by
about 19.4%.

Reviewed by: andrew, markj
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D21841

Eliminate redundant calls to critical_enter() and critical_exit() from
pmap_update_entry(). It suffices that interrupts are blocked.

Reviewed by: andrew, markj
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D21753

Implement pmap_page_is_mapped() correctly on arm64 and riscv.

We must also check for large mappings. pmap_page_is_mapped() is
mostly used in assertions, so the problem was not very noticeable.

Reviewed by: alc
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D21824

Complete the removal of the "wire_count" field from struct vm_page.

Convert all remaining references to that field to "ref_count" and update
comments accordingly. No functional change intended.

Reviewed by: alc, kib
Sponsored by: Intel, Netflix
Differential Revision: https://reviews.freebsd.org/D21768

In case a translation fault on the kernel address space occurs from
within a critical section, we must perform a lock-free check on the
faulting address.

Reported by: andrew
Reviewed by: andrew, markj
X-MFC with: r350579
Differential Revision: https://reviews.freebsd.org/D21685

Revert r352406, which contained changes I didn't intend to commit.

Fix a couple of nits in r352110.

- Remove a dead variable from the amd64 pmap_extract_and_hold().
- Fix grammar in the vm_page_wire man page.

Reported by: alc
Reviewed by: alc, kib
Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D21639

Change synchonization rules for vm_page reference counting.

There are several mechanisms by which a vm_page reference is held,
preventing the page from being freed back to the page allocator. In
particular, holding the page's object lock is sufficient to prevent the
page from being freed; holding the busy lock or a wiring is sufficent as
well. These references are protected by the page lock, which must
therefore be acquired for many per-page operations. This results in
false sharing since the page locks are external to the vm_page
structures themselves and each lock protects multiple structures.

Transition to using an atomically updated per-page reference counter.
The object's reference is counted using a flag bit in the counter. A
second flag bit is used to atomically block new references via
pmap_extract_and_hold() while removing managed mappings of a page.
Thus, the reference count of a page is guaranteed not to increase if the
page is unbusied, unmapped, and the object's write lock is held. As
a consequence of this, the page lock no longer protects a page's
identity; operations which move pages between objects are now
synchronized solely by the objects' locks.

The vm_page_wire() and vm_page_unwire() KPIs are changed. The former
requires that either the object lock or the busy lock is held. The
latter no longer has a return value and may free the page if it releases
the last reference to that page. vm_page_unwire_noq() behaves the same
as before; the caller is responsible for checking its return value and
freeing or enqueuing the page as appropriate. vm_page_wire_mapped() is
introduced for use in pmap_extract_and_hold(). It fails if the page is
concurrently being unmapped, typically triggering a fallback to the
fault handler. vm_page_wire() no longer requires the page lock and
vm_page_unwire() now internally acquires the page lock when releasing
the last wiring of a page (since the page lock still protects a page's
queue state). In particular, synchronization details are no longer
leaked into the caller.

The change excises the page lock from several frequently executed code
paths. In particular, vm_object_terminate() no longer bounces between
page locks as it releases an object's pages, and direct I/O and
sendfile(SF_NOCACHE) completions no longer require the page lock. In
these latter cases we now get linear scalability in the common scenario
where different threads are operating on different files.

__FreeBSD_version is bumped. The DRM ports have been updated to
accomodate the KPI changes.

Reviewed by: jeff (earlier version)
Tested by: gallatin (earlier version), pho
Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D20486

aarch64: make pmap_change_attr public like on other platforms

Submitted by: Greg V <greg@unrelenting.technology>
Reviewed by: markj
Differential Revision: https://reviews.freebsd.org/D20787

Ordinarily, during a superpage promotion or demotion within a pmap, the
pmap's lock ensures that other operations on the pmap don't observe the
old mapping being broken before the new mapping is established. However,
pmap_kextract() doesn't acquire the kernel pmap's lock, so it may observe
the broken mapping. And, if it does, it returns an incorrect result.

This revision implements a lock-free solution to this problem in
pmap_update_entry() and pmap_kextract() because pmap_kextract() can't
acquire the kernel pmap's lock.

Reported by: andrew, greg_unrelenting.technology
Reviewed by: andrew, markj
Tested by: greg_unrelenting.technology
X-MFC with: r350579
Differential Revision: https://reviews.freebsd.org/D21169

Enable superpage promotion within the kernel pmap.

Reviewed by: markj
X-MFC after: r350004
Differential Revision: https://reviews.freebsd.org/D21149

Because of AArch64's weak memory consistency model, we need to include a
memory barrier between the stores for initializing a page table page and
the store for adding that page to the page table. Otherwise, a page table
walk by another processor's MMU could see the page table page before it
sees the initialized entries.

Simplify pmap_growkernel(). In particular, eliminate an unnecessary TLB
invalidation.

Reviewed by: andrew, markj
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D21126

Use ATTR_DBM even when hardware dirty bit management is not enabled.

The ARMv8 reference manual only states that the bit is reserved in
this case; following Linux's example, use it instead of a
software-defined bit for the purpose of indicating that a managed
mapping is writable.

Reviewed by: alc, andrew
MFC after: r350004
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D21121

In pmap_advise(), when we encounter a superpage mapping, we first demote the
mapping and then destroy one of the 4 KB page mappings so that there is a
potential trigger for repromotion. Currently, we destroy the first 4 KB
page mapping that falls within the (current) superpage mapping or the
virtual address range [sva, eva). However, I have found empirically that
destroying the last 4 KB mapping produces slightly better results,
specifically, more promotions and fewer failed promotion attempts.
Accordingly, this revision changes pmap_advise() to destroy the last 4 KB
page mapping. It also replaces some nearby uses of boolean_t with bool.

Reviewed by: kib, markj
Differential Revision: https://reviews.freebsd.org/D21115

Have arm64's pmap_fault() handle WnR faults on dirty PTEs.

If we take a WnR permission fault on a managed, writeable and dirty
PTE, simply return success without calling the main fault handler. This
situation can occur if multiple threads simultaneously access a clean
writeable mapping and trigger WnR faults; losers of the race to mark the
PTE dirty would end up calling the main fault handler, which had no work
to do.

Reported by: alc
Reviewed by: alc
MFC with: r350004
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D21097

Remove an unneeded trunc_page() in pmap_fault().

Reported by: alc
MFC with: r350004
Sponsored by: The FreeBSD Foundation

Implement pmap_advise(). (Without a working pmap_advise() implementation
madvise(MADV_DONTNEED) and madvise(MADV_FREE) are NOPs.)

Reviewed by: markj
X-MFC after: r350004
Differential Revision: https://reviews.freebsd.org/D21062

Simplify the handling of superpages in pmap_clear_modify(). Specifically,
if a demotion succeeds, then all of the 4KB page mappings within the
superpage-sized region must be valid, so there is no point in testing the
validity of the 4KB page mapping that is going to be write protected.

Deindent the nearby code.

Reviewed by: kib, markj
Tested by: pho (amd64, i386)
X-MFC after: r350004 (this change depends on arm64 dirty bit emulation)
Differential Revision: https://reviews.freebsd.org/D21027

With the introduction of software dirty bit emulation for managed mappings,
we should test ATTR_SW_DBM, not ATTR_AP_RW, to determine whether to set
PGA_WRITEABLE. In effect, we are currently setting PGA_WRITEABLE based on
whether the dirty bit is preset, not whether the mapping is writeable.
Correct this mistake.

Reviewed by: markj
X-MFC with: r350004
Differential Revision: https://reviews.freebsd.org/D21013

Introduce pmap_store(), and use it to replace pmap_load_store() in places
where the page table entry was previously invalid. (Note that I did not
replace pmap_load_store() when it was followed by a TLB invalidation, even
if we are not using the return value from pmap_load_store().)

Correct an error in pmap_enter(). A test for determining when to set
PGA_WRITEABLE was always true, even if the mapping was read only.

In pmap_enter_l2(), when replacing an empty kernel page table page by a
superpage mapping, clear the old l2 entry and issue a TLB invalidation. My
reading of the ARM architecture manual leads me to believe that the TLB
could hold an intermediate entry referencing the empty kernel page table
page even though it contains no valid mappings.

Replace a couple direct uses of atomic_clear_64() by the new
pmap_clear_bits().

In a couple comments, replace the term "paging-structure caches", which is
an Intel-specific term for the caches that hold intermediate entries in the
page table, with wording that is more consistent with the ARM architecture
manual.

Reviewed by: markj
X-MFC after: r350004
Differential Revision: https://reviews.freebsd.org/D20998

Always use the software DBM bit for now.

r350004 added most of the machinery needed to support hardware DBM
management, but it did not intend to actually enable use of the hardware
DBM bit.

Reviewed by: andrew
MFC with: r350004
Sponsored by: The FreeBSD Foundation

Propagate attribute changes during demotion.

After r349117 and r349122, some mapping attribute changes do not trigger
superpage demotion. However, pmap_demote_l2() was not updated to ensure
that the replacement L3 entries carry any attribute changes that
occurred since promotion.

Reported and tested by: manu
Reviewed by: alc
MFC after: 1 week
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D20965

Implement software access and dirty bit management for arm64.

Previously the arm64 pmap did no reference or modification tracking;
all mappings were treated as referenced and all read-write mappings
were treated as dirty. This change implements software management
of these attributes.

Dirty bit management is implemented to emulate ARMv8.1's optional
hardware dirty bit modifier management, following a suggestion from alc.
In particular, a mapping with ATTR_SW_DBM set is logically writeable and
is dirty if the ATTR_AP_RW_BIT bit is clear. Mappings with
ATTR_AP_RW_BIT set are write-protected, and a write access will trigger
a permission fault. pmap_fault() handles permission faults for such
mappings and marks the page dirty by clearing ATTR_AP_RW_BIT, thus
mapping the page read-write.

Reviewed by: alc
MFC after: 1 month
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D20907

Revert r349442, which was a workaround for bus errors caused by an errant
TLB entry. Specifically, at the start of pmap_enter_quick_locked(), we
would sometimes have a TLB entry for an invalid PTE, and we would need to
issue a TLB invalidation before exiting pmap_enter_quick_locked(). However,
we should never have a TLB entry for an invalid PTE. r349905 has addressed
the root cause of the problem, and so we no longer need this workaround.

X-MFC after: r349905

Apply some light cleanup to uses of pmap_pte_dirty().

- Check for ATTR_SW_MANAGED before anything else.
- Use pmap_pte_dirty() in pmap_remove_pages().

No functional change intended.

Reviewed by: alc
MFC after: 1 week
Sponsored by: The FreeBSD Foundation

According to Section D5.10.3 "Maintenance requirements on changing System
register values" of the architecture manual, an isb instruction should be
executed after updating ttbr0_el1 and before invalidating the TLB. The
lack of this instruction in pmap_activate() appears to be the reason why
andrew@ and I have observed an unexpected TLB entry for an invalid PTE on
entry to pmap_enter_quick_locked(). Thus, we should now be able to revert
the workaround committed in r349442.

Reviewed by: markj
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D20904

Rename pmap_page_dirty() to pmap_pte_dirty().

This is a precursor to implementing dirty bit management.

Discussed with: alc
MFC after: 1 week
Sponsored by: The FreeBSD Foundation

Introduce pmap_clear(), which zeroes a page table entry, and use it, instead
of pmap_load_clear(), in places where we don't care about the page table
entry's prior contents.

Eliminate an unnecessary pmap_load() from pmap_remove_all(). Instead, use
the value returned by the pmap_load_clear() on the very next line. (In the
future, when we support "hardware dirty bit management", using the value
from the pmap_load() rather than the pmap_load_clear() would have actually
been an error because the dirty bit could potentially change between the
pmap_load() and the pmap_load_clear().)

A KASSERT() in pmap_enter(), which originated in the amd64 pmap, was meant
to check the value returned by the pmap_load_clear() on the previous line.
However, we were ignoring the value returned by the pmap_load_clear(), and
so the KASSERT() was not serving its intended purpose. Use the value
returned by the pmap_load_clear() in the KASSERT().

MFC after: 2 weeks

Merge the vm_page hold and wire mechanisms.

The hold_count and wire_count fields of struct vm_page are separate
reference counters with similar semantics. The remaining essential
differences are that holds are not counted as a reference with respect
to LRU, and holds have an implicit free-on-last unhold semantic whereas
vm_page_unwire() callers must explicitly determine whether to free the
page once the last reference to the page is released.

This change removes the KPIs which directly manipulate hold_count.
Functions such as vm_fault_quick_hold_pages() now return wired pages
instead. Since r328977 the overhead of maintaining LRU for wired pages
is lower, and in many cases vm_fault_quick_hold_pages() callers would
swap holds for wirings on the returned pages anyway, so with this change
we remove a number of page lock acquisitions.

No functional change is intended. __FreeBSD_version is bumped.

Reviewed by: alc, kib
Discussed with: jeff
Discussed with: jhb, np (cxgbe)
Tested by: pho (previous version)
Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D19247

Three changes to pmap_enter():

1. Use _pmap_alloc_l3() instead of pmap_alloc_l3() in order to handle the
possibility that a superpage mapping for "va" was created while we slept.
(This is derived from the amd64 version.)

2. Eliminate code for allocating kernel page table pages. Kernel page
table pages are preallocated by pmap_growkernel().

3. Eliminate duplicated unlock operations when KERN_RESOURCE_SHORTAGE is
returned.

MFC after: 2 weeks

Implement pmap_copy(). (This includes the changes applied to the amd64
pmap_copy() in r349585.)

Reviewed by: kib, markj
Differential Revision: https://reviews.freebsd.org/D20790

Revert one of the changes from r349323. Specifically, undo the change
that replaced a pmap_invalidate_page() with a dsb(ishst) in
pmap_enter_quick_locked(). Even though this change is in principle
correct, I am seeing occasional, spurious bus errors that are only
reproducible without this pmap_invalidate_page(). (None of adding an
isb, "upgrading" the dsb to wait on loads as well as stores, or
disabling superpage mappings eliminates the bus errors.) Add an XXX
comment explaining why the pmap_invalidate_page() is being performed.

Discussed with: andrew, markj

pmap_enter_quick_locked() never replaces a valid mapping, so it need not
perform a TLB invalidation. A barrier suffices. (See r343876.)

Add a comment to pmap_enter_quick_locked() in order to highlight the
fact that it does not replace valid mappings.

Correct a typo in one of pmap_enter()'s comments.

MFC after: 1 week

Introduce pmap_remove_l3_range() and use it in two places:
(1) pmap_remove(), where it eliminates redundant TLB invalidations by
pmap_remove() and pmap_remove_l3(), and (2) pmap_enter_l2(), where it may
optimize the TLB invalidations by batching them.

Reviewed by: markj
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D12725

Correct an error in r349122. pmap_unwire() should update the pmap's wired
count, not its resident count.

X-MFC with: r349122

Eliminate a redundant call to pmap_invalidate_page() from
pmap_ts_referenced().

MFC after: 14 days
Differential Revision: https://reviews.freebsd.org/D12725

Three changes to arm64's pmap_unwire():

Implement wiring changes on superpage mappings. Previously, a superpage
mapping was unconditionally demoted by pmap_unwire(), even if the wiring
change applied to the entire superpage mapping.

Rewrite a comment to use the arm64 names for bits in a page table entry.
Previously, the bits were referred to by their x86 names.

Use atomic_"op"_64() instead of atomic_"op"_long() to update a page table
entry in order to match the prevailing style in this file.

MFC after: 10 days

Three enhancements to arm64's pmap_protect():

Implement protection changes on superpage mappings. Previously, a superpage
mapping was unconditionally demoted by pmap_protect(), even if the
protection change applied to the entire superpage mapping.

Precompute the bit mask describing the protection changes rather than
recomputing it for every page table entry that is changed.

Skip page table entries that already have the requested protection changes
in place.

Reviewed by: andrew, kib
MFC after: 10 days
Differential Revision: https://reviews.freebsd.org/D20657

Previously, when pmap_remove_pages() destroyed a dirty superpage mapping,
it only called vm_page_dirty() on the first of the superpage's constituent
4KB pages. This revision corrects that error, calling vm_page_dirty() on
all of superpage's constituent 4KB pages.

MFC after: 3 days

Batch the TLB invalidations that are performed by pmap_protect() rather
than performing them one at a time.

MFC after: 10 days

Change the arm64 pmap so that updates to the global count of wired pages are
not performed directly by the pmap. Instead, they are performed by
vm_page_free_pages_toq(). (This is the same approach that we use on x86.)

Reviewed by: kib, markj
MFC after: 10 days
Differential Revision: https://reviews.freebsd.org/D20627

Change pmap_demote_l2_locked() so that it removes the superpage mapping on a
demotion failure. Otherwise, some callers to pmap_demote_l2_locked(), such
as pmap_protect(), may leave an incorrect mapping in place on a demotion
failure.

Change pmap_demote_l2_locked() so that it handles addresses that are not
superpage aligned. Some callers to pmap_demote_l2_locked(), such as
pmap_protect(), may not pass a superpage aligned address.

Change pmap_enter_l2() so that it correctly calls vm_page_free_pages_toq().
The arm64 pmap is updating the count of wired pages when freeing page table
pages, so pmap_enter_l2() should pass false to vm_page_free_pages_toq().

Optimize TLB invalidation in pmap_remove_l2().

Reviewed by: kib, markj (an earlier version)
Discussed with: andrew
MFC after: 3 weeks
Differential Revision: https://reviews.freebsd.org/D20585

Implement an alternative solution to the amd64 and i386 pmap problem that we
previously addressed in r348246.

This pmap problem also exists on arm64 and riscv. However, the original
solution developed for amd64 and i386 cannot be used on arm64 and riscv. In
particular, arm64 and riscv do not define a PG_PROMOTED flag in their level
2 PTEs. (A PG_PROMOTED flag makes no sense on arm64, where unlike x86 or
riscv we are required to break the old 4KB mappings before making the 2MB
mapping; and on riscv there are no unused bits in the PTE to define a
PG_PROMOTED flag.)

This commit implements an alternative solution that can be used on all four
architectures. Moreover, this solution has two other advantages. First, on
older AMD processors that required the Erratum 383 workaround, it is less
costly. Specifically, it avoids unnecessary calls to pmap_fill_ptp() on a
superpage demotion. Second, it enables the elimination of some calls to
pagezero() in pmap_kernel_remove_{l2,pde}().

In addition, remove a related stale comment from pmap_enter_{l2,pde}().

Reviewed by: kib, markj (an earlier version)
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D20538

Add missing data barriers after storeing a new valid pagetable entry.

When moving from an invalid to a valid entry we don't need to invalidate
the tlb, however we do need to ensure the store is ordered before later
memory accesses. This is because this later access may be to a virtual
address within the newly mapped region.

Add the needed barriers to places where we don't later invalidate the
tlb. When we do invalidate the tlb there will be a barrier to correctly
order this.

This fixes a panic on boot on ThunderX2 when INVARIANTS is turned off:
panic: vm_fault_hold: fault on nofault entry, addr: 0xffff000040c11000

Reported by: jchandra
Tested by: jchandra
Sponsored by: DARPA, AFRL
Differential Revision: https://reviews.freebsd.org/D19097

Handle kernel superpage mappings in pmap_remove_l2().

PR: 233088
Reviewed by: alc, andrew, kib
Tested by: sbruno
MFC after: 3 days
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D17981

Use pmap_invalidate_all rather than invalidating 512 level 2 entries in
the early pmap_mapbios/unmapbios code. It is even worse when there are
multiple L2 entries to handle as we would need to iterate over all pages.

Sponsored by: DARPA, AFRL

Fix some style(9) issues in the arm64 pmap_mapbios/unmapbios. Split lines
when they are too long.

Sponsored by: DARPA, AFRL

Remove {max/min}_offset() macros, use vm_map_{max/min}() inlines.

Exposing max_offset and min_offset defines in public headers is
causing clashes with variable names, for example when building QEMU.

Based on the submission by: royger
Reviewed by: alc, markj (previous version)
Sponsored by: The FreeBSD Foundation (kib)
MFC after: 1 week
Approved by: re (marius)
Differential revision: https://reviews.freebsd.org/D16881

Eliminate kmem_malloc()'s unused arena parameter. (The arena parameter
became unused in FreeBSD 12.x as a side-effect of the NUMA-related
changes.)

Reviewed by: kib, markj
Discussed with: jeff, re@
Differential Revision: https://reviews.freebsd.org/D16825

Use the correct PTE when changing the attribute of multiple pages.

Submitted by: andrew (long time ago)
Sponsored by: Rubicon Communications, LLC (Netgate)

Add pmap_is_valid_memattr(9).

Discussed with: alc
Sponsored by: The FreeBSD Foundation, Mellanox Technologies
MFC after: 1 week
Differential revision: https://reviews.freebsd.org/D15583

Simplify the arm64 implementation of pmap_mincore().

No functional change intended.

Reviewed by: alc
MFC after: 2 weeks
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D16427

Eliminate a comment that doesn't apply to this pmap implementation.

Coalesce the variable definitions for PV entry management.

MFC after: 3 weeks

Add support for pmap_enter(psind = 1) to the arm64 pmap.

See the commit log messages for r321378 and r336288 for descriptions of
this functionality.

Reviewed by: alc
Differential Revision: https://reviews.freebsd.org/D16303

Initialize the L3 page's wire count correctly after a L2 entry demotion.

Reviewed by: alc
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D16303

Raise the size of L3 table for early devmap on arm64

Some driver (like efifb) needs to map more than the current L2_SIZE
Raise the size so we can map the framebuffer setup by the bootloader.

Reviewed by: cognet

Port r324665 and r325285 to arm64.

These changes ensure that reclaim_pv_chunk() can be safely be
executed concurrently by multiple threads.

Reviewed by: alc
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D16304

Reuse the PV entry when updating a mapping in pmap_enter().

This addresses a problem described in r335784, where memory
pressure forces reclamation of a PV chunk and in rare cases leads to a
use-after-free of a page table page.

Reviewed by: alc, kib
MFC after: 3 weeks
Differential Revision: https://reviews.freebsd.org/D16181

Re-count available PV entries after reclaiming a PV chunk.

The call to reclaim_pv_chunk() in reserve_pv_entries() may free a
PV chunk with free entries belonging to the current pmap. In this
case we must account for the free entries that were reclaimed, or
reserve_pv_entries() may return without having reserved the requested
number of entries.

Reviewed by: alc, kib
Tested by: pho (previous version)
MFC after: 2 weeks
Differential Revision: https://reviews.freebsd.org/D15911

Add pmap_mapdev_attr for arm64

This is needed for efifb.
arm and ricv pmap (the two arch with arm64 that uses subr_devmap) have very
different implementation so for now only add this for arm64.

Tested with efifb on Pine64 with a few other patches.

Reviewed by: cognet
Differential Revision: https://reviews.freebsd.org/D15294

Remove max_pa, it's unused.

Sponsored by: DARPA, AFRL

Push down the locking in pmap_fault to just be around the calls to
arm64_address_translate_*. There is no need to lock around the switch
statement as we only care about a few cases.

Sponsored by: DARPA, AFRL

On ThunderX2 we need to be careful to only map the memory the firmware
lists in the EFI memory map. As such we need to reduce the mappings to
restrict them to not be the full 1G block. For now reduce this to a 2M
block, however this may be further restricted to be 4k page aligned as
other SoCs may require.

This allows ThunderX2 to boot reliably to userspace without performing
any speculative memory accesses to invalid physical memory.

This is a recommit of r334035 now that we can access the EFI Runtime data
through the DMAP region.

Tested by: tuexen
Sponsored by: DARPA, AFRL

Allow us to read the physmap data into our own array and use this to build
the DMAP region on arm64.

We already have the needed information to build these tables, we just need
to extract it. This significantly simplifies the code.

Obtained from: ABT Systems Ltd
Sponsored by: Turing Robotic Industries

Revert r334035 for now. It breaks the boot on some boards as er expect to
be able to read UEFI RuntimeData memory via the DMAP region.

On ThunderX2 we need to be careful to only map the memory the firmware
lists in the EFI memory map. As such we need to reduce the mappings to
restrict them to not be the full 1G block. For now reduce this to a 2M
block, however this may be further restricted to be 4k page aligned as
other SoCs may require.

This allows ThunderX2 to boot reliably to userspace without performing
any speculative memory accesses to invalid physical memory.

Sponsored by: DARPA, AFRL

Stop using the DMAP region to map ACPI memory.

On some arm64 boards we need to access memory in ACPI tables that is not
mapped in the DMAP region. To handle this create the needed mappings in
pmap_mapbios in the KVA space.

Submitted by: Michal Stanek (mst@semihalf.com)
Sponsored by: Cavium
Differential Revision: https://reviews.freebsd.org/D15059

Switch arm64 to use the same physmem code as 32-bit arm.

The main advantage of this is to allow us to exclude memory from being
used by the kernel. This may be from the memreserve property, or ranges
marked as no-map under the reserved-memory node.

More work is still needed to remove the physmap array. This is still used
for creating the DMAP region, however other patches need to be committed
before we can remove this.

Obtained from: ABT Systems Ltd
Sponsored by: Turing Robotic Industries

In pmap_get_tables(), check that the L2 is indeed a table before attempting
to get the l3.

Restrict the arm64 DMAP region to the 1G blocks where we have at least
one physical page. This is in preparation for limiting it further as this
is needed on some hardware, however testing has shown issues with further
restricting the DMAP and ACPI.

Sponsored by: DARPA, AFRL
Sponsored by: Cavium (Hardware)

Unify bulk free operations in several pmaps.

Submitted by: Yoshihiro Ota
Reviewed by: markj
MFC after: 2 weeks
Differential revision: https://reviews.freebsd.org/D13485

vm_wait() rework.

Make vm_wait() take the vm_object argument which specifies the domain
set to wait for the min condition pass. If there is no object
associated with the wait, use curthread' policy domainset. The
mechanics of the wait in vm_wait() and vm_wait_domain() is supplied by
the new helper vm_wait_doms(), which directly takes the bitmask of the
domains to wait for passing min condition.

Eliminate pagedaemon_wait(). vm_domain_clear() handles the same
operations.

Eliminate VM_WAIT and VM_WAITPFAULT macros, the direct functions calls
are enough.

Eliminate several control state variables from vm_domain, unneeded
after the vm_wait() conversion.

Scetched and reviewed by: jeff
Tested by: pho
Sponsored by: The FreeBSD Foundation, Mellanox Technologies
Differential revision: https://reviews.freebsd.org/D14384

Make v_wire_count a per-cpu counter(9) counter. This eliminates a
significant source of cache line contention from vm_page_alloc(). Use
accessors and vm_page_unwire_noq() so that the mechanism can be easily
changed in the future.

Reviewed by: markj
Discussed with: kib, glebius
Tested by: pho (earlier version)
Sponsored by: Netflix, Dell/EMC Isilon
Differential Revision: https://reviews.freebsd.org/D14273

Use vm_page_unwire_noq() instead of directly modifying page wire counts.

No functional change intended.

Reviewed by: alc, kib (previous revision)
MFC after: 2 weeks
Differential Revision: https://reviews.freebsd.org/D14266

Only promote userspace mappings to superpages. This was dropped in r328510,
however due to the break-before-make requirement on arm64 is is currently
unsafe to promote kernel pages.

Sponsored by: DARPA, AFRL

Remove #endif forgotten in r328510.

Pointy hat: mmel

Fix handling of I-cache sync operations

- pmap_enter_object() can be used for mapping of executable pages, so it's
necessary to handle I-cache synchronization within it.

- Fix race in I-cache synchronization in pmap_enter(). The current code firstly
maps given page to target VA and then do I-cache sync on it. This causes
race, because this mapping become visible to other threads, before I-cache
is synced.
Do sync I-cache firstly (by using DMAP VA) and then map it to target VA.

- ARM64 ARM permits implementation of aliased (AIVIVT, VIPT) I-cache, but we
can use different that final VA for flushing it. So we should use full
I-cache flush on affected platforms. For now, and as temporary solution,
use full flush always.

Fix pmap_fault().

- special fault handling for break-before-make mechanism should be also
applied for instruction translation faults, not only for data translation
faults.

- since arm64_address_translate_...() functions are not atomic,
use these with disabled interrupts.

Add a pmap invalidate that doesn't call sched_pin.

When demoting DMAP pages curthread may be pointing to data within the
page we are demoting. Create a new invalidate that doesn't pin and use
it in the demote case.

As the demote has both interrupts disabled, and is within a critical section
this is safe from having the scheduler from switching to another CPU.

Reported by: loos
Reviewed by: loos
Sponsored by: DARPA, AFRL
Differential Revision: https://reviews.freebsd.org/D13955

Workaround Spectre Variant 2 on arm64.

We need to handle two cases:

1. One process attacking another process.
2. A process attacking the kernel.

For the first case we clear the branch predictor state on context switch
between different processes. For the second we do this when taking an
instruction abort on a non-userspace address.

To clear the branch predictor state a per-CPU function pointer has been
added. This is set by the new cpu errata code based on if the CPU is
known to be affected.

On Cortex-A57, A72, A73, and A75 we call into the PSCI firmware as newer
versions of this will clear the branch predictor state for us.

It has been reported the ThunderX is unaffected, however the ThunderX2 is
vulnerable. The Qualcomm Falkor core is also affected. As FreeBSD doesn't
yet run on the ThunderX2 or Falkor no workaround is included for these CPUs.

MFC after: 3 days
Sponsored by: DARPA, AFRL
Differential Revision: https://reviews.freebsd.org/D13812

Only install the new pagetable pointer into ttbr0_el1 when it differs from
the existing value.

MFC after: 1 week
Sponsored by: DARPA, AFRL

Move some of the common thread switching code into C. This will help with
future optimisations, e.g. using Address Space IDs (asid).

MFC after: 1 week
Sponsored by: DARPA, AFRL

Do pass removing some write-only variables from the kernel.

This reduces noise when kernel is compiled by newer GCC versions,
such as one used by external toolchain ports.

Reviewed by: kib, andrew(sys/arm and sys/arm64), emaste(partial), erj(partial)
Reviewed by: jhb (sys/dev/pci/* sys/kern/vfs_aio.c and sys/kern/kern_synch.c)
Differential Revision: https://reviews.freebsd.org/D10385

Move instantiation of msgbufp from 9 MD files to subr_prf.c.

This variable should be pure MI except possibly for reading it in MD
dump routines. Its initialization was pure MD in 4.4BSD, but FreeBSD
changed this in r36441 in 1998. There were many imperfections in
r36441. This commit fixes only a small one, to simplify fixing the
others 1 arch at a time. (r47678 added support for
special/early/multiple message buffer initialization which I want in
a more general form, but this was too fragile to use because hacking
on the msgbufp global corrupted it, and was only used for 5 hours in
-current...)

Make the arm64 pmap_invalidate functions static inline. This fixes building
with DIAGNOSTIC.

PR: 223874
Reported by: emaste
MFC after: 1 week

Fix the VM_NRESERVLEVEL == 0 build.

Add VM_NRESERVLEVEL guards in the pmaps that implement transparent
superpage promotion using reservations.

Reviewed by: alc, kib
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D12764

Move the pmap_l0_index, etc. macros to pte.h. These will be used by the
EFI Runtime Services code.

Sponsored by: DARPA, AFRL

Remove an unneeded call to pmap_invalidate_all. This was never called as
the anyvalid variable is never set.

MFC after: 1 week
Sponsored by: DARPA, AFRL

Fix a bug in pmap_protect where we invalidate the wrong page. With this we
can now remove an unneeded call to invalidate all entries.

MFC after: 1 week
Sponsored by: DARPA, AFRL

Move the l0 pagetable address to struct mdproc. It is a property of the
whole process so should live there.

Sponsored by: DARPA, AFRL

In the arm64 pmap_remove, when removing a full superpage there is no need
to demote it to 512 pages, then remove each of these. We can just remove
the l2 map directly. This is what the intel pmaps already do.

Sponsored by: DARPA, AFRL

Remove all calls to cpu_dcache_wb_range from the arm64 pmap code. These
were unneeded as we tell the tlb the pagetables are in cached memory. This
gives us a small, but statistically significant improvement over just
removing the PTE_SYNC cases.

While here remove PTE_SYNC, it's now unneeded.

Sponsored by: DARPA, AFRL

Stop calling cpu_dcache_wb_range from PTE_SYNC.

We set the shareability attributes in TCR_EL1 on boot. These tell the
hardware the pagetables are in cached memory so there is no need to flush
the entries from the cache to memory.

This has about 4.2% improvement in system time and 2.7% improvement in
user time for a buildkernel -j48 on a ThunderX.

Keep the old code for now to allow for further comparisons.

- Don't bother flushing the data cache for pages we're about to unmap, there's
no need to.
- Remove pmap_is_current(), pmap_[pte|l3]_valid_cacheable as there were only
used to know if we had to write back pages.
- In pmap_remove_pages(), don't bother invalidating each page in the TLB,
we're about to flush the whole TLB anyway.

This makes make world 8-9% faster on my hardware.

Reviewed by: andrew

Add reclaim_pv_chunk on arm64. This is based on the amd64 code so should
operate similarly, other than not needing the delayed invalidation.

It has been tested with artificial injection of vm_page_alloc failures
while running 'sort /dev/zero'.

Reviewed by: alc, kib
MFC after: 1 week
Sponsored by: DARPA, AFRL
Differential Revision: https://reviews.freebsd.org/D10574

Rather than checking if the top bit in a virtual address is a 0 or 1
compare against VM_MAXUSER_ADDRESS as we should have been doing.

Sponsored by: DARPA, AFRL

Set the arm64 Execute-never bits in more places.

We need to set the Execute-never bits when mapping device memory as the
hardware may perform speculative instruction fetches.

Set the Privileged Execute-ever bit on userspace memory to stop the kernel
if it is tricked into executing it.

Reviewed by: kib
Sponsored by: DARPA, AFRL
Differential Revision: https://reviews.freebsd.org/D10382

Start to use the User and Privileged execute-never bits in the arm64
pagetables. This sets both bits when entering an address we know shouldn't
be executed.

I expect we could mark all userspace pages as Privileged execute-never to
ensure the kernel doesn't branch to one of these addresses.

While here add the ARMv8.1 upper attributes.

Reviewed by: alc, kib (previous version)
MFC after: 1 week
Sponsored by: DARPA, AFRL
Differential Revision: https://reviews.freebsd.org/D10360

Do not lose dirty bits for removing PROT_WRITE on arm64.

Arm64 pmap interprets accessed writable ptes as modified, since
ARMv8.0 does not track Dirty Bit Modifier in hardware. If writable bit
is removed, page must be marked as dirty for MI VM.

This change is most important for COW, where fork caused losing
content of the dirty pages which were not yet scanned by pagedaemon.

Reviewed by: alc, andrew
Reported and tested by: Mark Millard <markmi@dsl-only.net>
PR: 217138, 217239
Sponsored by: The FreeBSD Foundation
MFC after: 2 weeks

Previously, vm_radix_remove() would panic if the radix trie didn't
contain a vm_page_t at the specified index. However, with this
change, vm_radix_remove() no longer panics. Instead, it returns NULL
if there is no vm_page_t at the specified index. Otherwise, it
returns the vm_page_t. The motivation for this change is that it
simplifies the use of radix tries in the amd64, arm64, and i386 pmap
implementations. Instead of performing a lookup before every remove,
the pmap can simply perform the remove.

Reviewed by: kib, markj
Differential Revision: https://reviews.freebsd.org/D8708

Various changes to pmap_ts_referenced()

Move PMAP_TS_REFERENCED_MAX out of the various pmap implementations and
into vm/pmap.h, and describe what its purpose is. Eliminate the archaic
"XXX" comment about its value. I don't believe that its exact value, e.g.,
5 versus 6, matters.

Update the arm64 and riscv pmap implementations of pmap_ts_referenced()
to opportunistically update the page's dirty field.

On amd64, use the PDE value already cached in a local variable rather than
dereferencing a pointer again and again.

Reviewed by: kib, markj
MFC after: 2 weeks
Differential Revision: https://reviews.freebsd.org/D7836

Only call cpu_icache_sync_range when inserting an executable page. If the
page is non-executable the contents of the i-cache are unimportant so this
call is just adding unneeded overhead when inserting pages.

While doing research using gem5 with an O3 pipeline and 1k/32k/1M iTLB/L1
iCache/L2 Bjoern Zeeb (bz@) observed a fairly high rate of calls into
arm64_icache_sync_range() from pmap_enter() along with a high number of
instruction fetches and iTLB/iCache hits.

Limiting the calls to arm64_icache_sync_range() to only executable pages,
we observe the iTLB and iCache Hit going down by about 43%. These numbers
are quite misleading when looked at alone as at the same time instructions
retired were reduced by 19.2% and instruction fetches were reduced by 38.8%.
Overall this reduced the runtime of the test program by 22.4%.

On Juno hardware, in steady-state, running the same test, using the cycle
count to determine runtime, we do see a reduction of up to 28.9% in runtime.

While these numbers certainly depend on the program executed, we expect an
overall performance improvement.

Reported by: bz
Obtained from: ABT Systems Ltd
MFC after: 1 week
Sponsored by: The FreeBSD Foundation

Enable superpages on arm64 by default. These seem to be stable, having
survived multiple world and kernel builds, and of poudriere building full
package sets.

I have observed a 3% reduction in buildworld times with superpages enabled,
however further testing is needed to see if this is observed in other
workloads.

Obtained from: ABT Systems Ltd
MFC after: 1 month
Sponsored by: The FreeBSD Foundation

Remove support for idle page zeroing.

Idle page zeroing has been disabled by default on all architectures since
r170816 and has some bugs that make it seemingly unusable. Specifically,
the idle-priority pagezero thread exacerbates contention for the free page
lock, and yields the CPU without releasing it in non-preemptive kernels. The
pagezero thread also does not behave correctly when superpage reservations
are enabled: its target is a function of v_free_count, which includes
reserved-but-free pages, but it is only able to zero pages belonging to the
physical memory allocator.

Reviewed by: alc, imp, kib
Differential Revision: https://reviews.freebsd.org/D7714

Fix arm64 superpages bugs in pmap_enter:
* Pass the correct virtual address when demoting a superpage
* Use the correct l3 table after demoting a superpage
* Remove an invalid KASSERT hit demoting then promoting a superpage [1]

With this it is believed that superpages on arm64 is stable.

Reported by: [1] cognet
Obtained from: ABT Systems Ltd
MFC after: 1 week
Sponsored by: The FreeBSD Foundation

Because we need to use a break-before-make sequence when promoting pages
there is a short period where functions that walk the kernel page table
without locking them may see an invalid entry. One solution would be to add
locking to these functions, however some may be called from locations where
we are unable to sleep.

Until a better solution can be found stop promoting pages in the kernel
pmap so these functions work as expected.

Obtained from: ABT Systems Ltd
MFC after: 1 month
Sponsored by: The FreeBSD Foundation

Fix an assert, it should check if, when moving from 1 l1 to 512 l2 blocks,
the l2 entry is a block type and not an l3 page.

While here fix the string to correct the level name and add a missing ')'.

Obtained from: ABT Systems Ltd
MFC after: 1 month
Sponsored by: The FreeBSD Foundation

Allow superpages on arm64 to be enabled. It is disabled for now to allow
for more testing across more hardware.

Obtained from: ABT Systems Ltd
MFC after: 1 month
Sponsored by: The FreeBSD Foundation

Add support to promote and demote managed superpages. This is the last part
needed before enabling superpages on arm64. This code is based on the amd64
pmap with changes as needed to handle the differences between the two
architectures.

Obtained from: ABT Systems Ltd
MFC after: 1 month
Sponsored by: The FreeBSD Foundation

Teach the parts of the arm64 pmap that need to iterate over pages to also
iterate over superpages. We don't yet create these, but soon will.

Obtained from: ABT Systems Ltd
MFC after: 1 month
Sponsored by: The FreeBSD Foundation

Also adjust the virtual address passed to vm_page_pa_tryrelock.

Reported by: alc
Obtained from: ABT Systems Ltd
MFC after: 1 month
Sponsored by: The FreeBSD Foundation

Map memory as read-only in pmap_enter_quick_locked as is done in other
pmap implementations.

Obtained from: ABT Systems Ltd
MFC after: 1 month
Sponsored by: The FreeBSD Foundation

If we find we have a superpage in pmap_enter_quick_locked return without
trying to add a new level 3 page.

Obtained from: ABT Systems Ltd
MFC after: 1 month
Sponsored by: The FreeBSD Foundation

Include the offset the virtual address is within an L1 or L2 block when
finding the vm_page_t in pmap_extract_and_hold. Previously it would return
the vm_page_t of the first page in a block. This would cause issues when,
for example, fsck reads from a device into the middle of a superpage. In
this case the read call would write to the start of the block, and not to
the buffer passed in.

Obtained from: ABT Systems Ltd
MFC after: 1 month
Sponsored by: The FreeBSD Foundation

Fix pmap_update_entry, pmap_invalidate_range takes the end address, not
the size.

Obtained from: ABT Systems Ltd
MFC after: 1 month
Sponsored by: The FreeBSD Foundation

Use switch statements in pmap_remove_pages. While only one level of
pagetable is supported more will be added soon to support removing
superpages.

Obtained from: ABT Systems Ltd
MFC after: 1 month
Sponsored by: The FreeBSD Foundation

Use pmap_update_entry in pmap_enter when updating an entry with a new
physical address. This is required when either mapping is writeable.

While here remove an unneeded call to pmap_pde, we already have the pde
from earlier in the function.

Obtained from: ABT Systems Ltd
MFC after: 1 month
Sponsored by: The FreeBSD Foundation

Add sysctls to report on superpages statistics. While here add extra
logging to these paths.

Obtained from: ABT Systems Ltd
MFC after: 1 month
Sponsored by: The FreeBSD Foundation

Add a size argument to pmap_update_entry.
Make use of this in pmap_promote_l2.

Obtained from: ABT Systems Ltd
MFC after: 1 month
Sponsored by: The FreeBSD Foundation

Add KASSERTS in pmap_alloc_l3 to ensure we are not encountering superpages
when we don't yet expect them;

Obtained from: ABT Systems Ltd
MFC after: 1 month
Sponsored by: The FreeBSD Foundation

Implement promotions and demotions in the arm64 pmap code. For now we don't
promote memory as I am not sure all the demotion cases are handled, however
it is useful to implement pmap_page_set_memattr. This is used, for example,
when mapping uncached memory for bus_dma(9).

pmap_page_set_memattr needs to demote the DMAP region as on ARM we need to
ensure all mappings to the same physical address have the same attributes.

Reviewed by: kib
Obtained from: ABT Systems Ltd
MFC after: 1 month
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D6987

Uncomment the vm.kvm_size and vm.kvm_free sysctls. These work as expected so
there is no reason to leave them commented out.

Obtained from: ABT Systems Ltd
MFC after: 1 month
Sponsored by: The FreeBSD Foundation

Implement pmap_align_superpage on arm64 based on the amd64 implementation.
This will be needed when superpage support is added.

Obtained from: ABT Systems Ltd
MFC after: 1 month
Sponsored by: The FreeBSD Foundation

Remove the pvh_global_lock lock from the arm64 pmap. It is unneeded on arm64
as invalidation will have completed before the pmap_invalidate_* functions
have complete.

Discussed with: alc, kib
Obtained from: ABT Systems Ltd
MFC after: 1 month
Sponsored by: The FreeBSD Foundation

Remove trailing whitespace from the arm64 pmap

Obtained from: ABT Systems Ltd
MFC after: 3 weeks
Sponsored by: The FreeBSD Foundation

Extract the common parts of pmap_kenter_device to a new function. This will
be used when superpage support is added.

Obtained from: ABT Systems Ltd
MFC after: 1 month
Sponsored by: The FreeBSD Foundation

Fix the comment above pmap_invalidate_page. tlbi will invalidate the tlb
on all CPUs.

Obtained from: ABT Systems Ltd
MFC after: 1 month
Sponsored by: The FreeBSD Foundation

Relax the barriers around a TLB invalidation to only wait on
inner-shareable memory accesses. There is no need for full system barriers.

Obtained from: ABT Systems Ltd
MFC after: 1 month
Sponsored by: The FreeBSD Foundation

sys: use our roundup2/rounddown2() macros when param.h is available.

rounddown2 tends to produce longer lines than the original code
and when the code has a high indentation level it was not really
advantageous to do the replacement.

This tries to strike a balance between readability using the macros
and flexibility of having the expressions, so not everything is
converted.

Set the upper limit of the DMAP region to the limit of RAM as was found in
the physmap. This will reduce the likelihood of an issue where we have
device memory mapped in the DMAP. This can only happen if it is within the
same 1G block of normal memory.

Reviewed by: kib
Obtained from: ABT Systems Ltd
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D5938

Increase the arm64 kernel address space to 512GB, and the DMAP region to
2TB. The latter can be increased in 512GB chunks by adjusting the lower
address, however more work will be needed to increase the former.

There is still some work needed to only create a DMAP region for the RAM
address space as on ARM architectures all mappings should have the same
memory attributes, and these will be different for device and normal memory.

Reviewed by: kib
Obtained from: ABT Systems Ltd
Relnotes: yes
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D5859

Use PHYS_IN_DMAP to check if a physical address is within the DMAP region.

Approved by: ABT Systems Ltd
Sponsored by: The FreeBSD Foundation

arm64: pagezero improvement

This change has been provided to improve pagezero call performance.

Submitted by: Dominik Ermel <der@semihalf.com>
Obtained from: Semihalf
Sponsored by: Cavium
Reviewed by: kib
Differential Revision: https://reviews.freebsd.org/D5741

Add support for 4 level pagetables. The userland address space has been
increased to 256TiB. The kernel address space can also be increased to be
the same size, but this will be performed in a later change.

To help work with an extra level of page tables two new functions have
been added, one to file the lowest level table entry, and one to find the
block/page level. Both of these find the entry for a given pmap and virtual
address.

This has been tested with a combination of buildworld, stress2 tests, and
by using sort to consume a large amount of memory by sorting /dev/zero. No
new issues are known to be present from this change.

Reviewed by: kib
Obtained from: ABT Systems Ltd
Relnotes: yes
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D5720

pmap arm64: fixing pmap_invalidate_range

It seems that if range within one page is given this page will not be
invalidated at all. Clean it up.

Submitted by: Dominik Ermel <der@semihalf.com>
Obtained from: Semihalf
Sponsored by: Cavium
Reviewed by: wma, zbb
Approved by: cognet (mentor)
Differential Revision: https://reviews.freebsd.org/D5569

Ignore invalid page descriptors in ARM64 pmap_mincore

Prevent the function from null-pointer-dereference when unexisting
mapping is being processed.

Obtained from: Semihalf
Sponsored by: Cavium
Approved by: cognet (mentor)
Reviewed by: zbb, cognet
Differential revision: https://reviews.freebsd.org/D5228

Ensure we don't overflow the phys_avail array. Some firmware may provide
more memory locations than we have space to record.

Sponsored by: ABT Systems Ltd

Add support for moving the DMAP range. This is needed as some AMD SoCs
place physical memory at an address outside the old DMAP range. This is an
issue as we rely on being able to move from PA -> VA using this range.

Obtained from: Patrick Wildt <patrick@bitrig.org> (earlier version)
Sponsored by: ABT Systems Ltd
Differential Revision: https://reviews.freebsd.org/D3885

Use pmap_load more consistently. While here try to only load the data once
when we reuse the same data.

Obtained from: EuroBSDCon Devsummit
Sponsored by: ABT Systems Ltd

Exploit r288122 to address a cosmetic issue. Since PV chunk pages don't
belong to a vm object, they can't be paged out. Since they can't be paged
out, they are never enqueued in a paging queue. Nonetheless, passing
PQ_INACTIVE to vm_page_unwire() creates the appearance that these pages
are being enqueued in the inactive queue. As of r288122, we can avoid
this false impression by passing PQ_NONE.

Submitted by: kmacy (an earlier version)
Differential Revision: https://reviews.freebsd.org/D1674

Add support for pmap_mincore on arm64 by walking the page tables to find
the details for the requested address.

PR: 202307
Obtained from: ABT Systems Ltd
Sponsored by: The FreeBSD Foundation

Add support for pmap_sync_icache on arm64.

Reviewed by: emaste, imp (both earlier version)
Obtained from: ABT Systems Ltd
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D3438

Add pages used by the PV entries to minidumps.

Add pmap_get_tables to get the page tables for a given virtual address. This
will be used for minidump support.

Obtained from: ABT Systems Ltd
Sponsored by: The FreeBSD Foundation

Add two new pmap functions:
vm_offset_t pmap_quick_enter_page(vm_page_t m)
void pmap_quick_remove_page(vm_offset_t kva)

These will create and destroy a temporary, CPU-local KVA mapping of a specified page.

Guarantees:
--Will not sleep and will not fail.
--Safe to call under a non-sleepable lock or from an ithread

Restrictions:
--Not guaranteed to be safe to call from an interrupt filter or under a spin mutex on all platforms
--Current implementation does not guarantee more than one page of mapping space across all platforms. MI code should not make nested calls to pmap_quick_enter_page.
--MI code should not perform locking while holding onto a mapping created by pmap_quick_enter_page

The idea is to use this in busdma, for bounce buffer copies as well as virtually-indexed cache maintenance on mips and arm.

NOTE: the non-i386, non-amd64 implementations of these functions still need review and testing.

Reviewed by: kib
Approved by: kib (mentor)
Differential Revision: http://reviews.freebsd.org/D3013

Add enough of pmap_page_set_memattr to run gstat. It still needs to split
the DMAP 1G pages so we set the attributes only on the specified page.

Obtained from: ABT Systems Ltd
Sponsored by: The FreeBSD Foundation

Add ARM64TODO markers to unimplemented functionality

Reviewed by: andrew
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D2389

Set memory to be inner-sharable. This isn't needed on device memory as the
MMU will ignore the attribute there, howeverit simplifies to code to alwas
set it.

Obtained from: ABT Systems Ltd
Sponsored by: The FreeBSD Foundation

Add more tlb invalidations. We currently invalidate when we may not need
to, but with this I can boot on a simulator that models the tlb.

Obtained from: ABT Systems Ltd
Sponsored by: The FreeBSD Foundation

Remove an unneeded define and old comment referencing amd64.

Remove an old comment, the cache is enabled.

Obtained from: ABT Systems Ltd
Sponsored by: The FreeBSD Foundation

Use pmap_load to load table entries. This simplifies finding places where
we access the tables.

Obtained from: ABT Systems Ltd
Sponsored by: The fReeBSD Foundation

Add pmap_mapbios and pmap_unmapbios. These will be needed to support acpi.

Sponsored by: The FreeBSD Foundation

Remove the calls to panic from pmap_object_init_pt and pmap_clear_modify.

Sponsored by: The FreeBSD Foundation

Bring in the start of the arm64 kernel.

This is only the minimum set of files needed to boot in qemu. As such it is
missing a few things.

The bus_dma code is currently only stub functions with a full implementation
from the development tree to follow.

The gic driver has been copied as the interrupt framework is different. It
is expected the two drivers will be merged by the arm intrng project,
however this will need to be imported into the tree and support for arm64
would need to be added.

This includes code developed by myself, SemiHalf, Ed Maste, and Robin
Randhawa from ARM. This has been funded by the FreeBSD Foundation, with
early development by myself in my spare time with assistance from Robin.

Differential Revision: https://reviews.freebsd.org/D2199
Reviewed by: emaste, imp
Relnotes: yes
Sponsored by: The FreeBSD Foundation