pmap: Convert boolean_t to bool.

Reviewed by: kib (older version)
Differential Revision: https://reviews.freebsd.org/D39921

powerpc_mmu_radix: add leaf page for wired mappings when pmap_enter(psind=1)

This applies the fix to powerpc's pmap as was done in commit aa3bcaad51076ceb
and d0941ed9b5c39d92 for amd64 and riscv pmaps, respectively.

Reported by: Jenkins
Reviewed by: bojan.novkovic_fer.hr, markj
Fixes: e4078494f344bcba8709216bd601efa3dd05f6b3
Differential Revision: https://reviews.freebsd.org/D43339

sys: Automated cleanup of cdefs and other formatting

Apply the following automated changes to try to eliminate
no-longer-needed sys/cdefs.h includes as well as now-empty
blank lines in a row.

Remove /^#if.*\n#endif.*\n#include\s+<sys/cdefs.h>.*\n/
Remove /\n+#include\s+<sys/cdefs.h>.*\n+#if.*\n#endif.*\n+/
Remove /\n+#if.*\n#endif.*\n+/
Remove /^#if.*\n#endif.*\n/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/types.h>/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/param.h>/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/capsicum.h>/

Sponsored by: Netflix

powerpc pmap: initialize kernel pmap radix trie

Commit 2d2bcba7ba70141388729ed49674b36fd01146c5 changed radix trie
implementation and made it necessary that radix tries be initialized
with vm_radix_init. @dbaio reports that in some configurations, there
is a powerpc boot panic and that this commit introduced the
problem. In powerpc/aim/mmu_radix.c, the radix trie in kernel_pmap is
initialized by zeroing all its fields.

Add a call to vm_radix_init to properly initialize
kernel_pmap->pm_radix.

Reported by: dbaio
Reviewed by: alc, jhibbits
Fixes: 2d2bcba7ba drop NULL check from radix trie search
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D41846

powerpc_mmu_radix: include vm_radix.h file

Replacing _vm_radix.h with vm_radix.h in one header file leaves one
source file short of function definitions. Include vm_radix.h in that
source file.

Fixes a problem rooted in change 429c871ddddac4bbf6abf1eb9e2e6603f87c2ef5.

sys: Remove $FreeBSD$: one-line .c pattern

Remove /^[\s*]*__FBSDID\("\$FreeBSD\$"\);?\s*\n/

spdx: The BSD-2-Clause-FreeBSD identifier is obsolete, drop -FreeBSD

The SPDX folks have obsoleted the BSD-2-Clause-FreeBSD identifier. Catch
up to that fact and revert to their recommended match of BSD-2-Clause.

Discussed with: pfg
MFC After: 3 days
Sponsored by: Netflix

powerpc: fix a few pmap related functions to return correct types

While experimenting with changing boolean_t to another type, I noticed
that several powerpc pmap related functions returned the wrong type:
boolean_t instead of int.

Fix several declarations and definitions to match the actual pmap
function types: pmap_dev_direct_mapped_t and pmap_ts_referenced_t.

MFC after: 3 days

powerpc/pmap: Account for a potential NULL pmap in pmap_sync_icache

It's apparently possible for pcpu->pc_curpmap to be NULL at some point,
leading to a panic. Account for this as is done with the other 64-bit
AIM pmap.

Reported by: pkubaj
Tested by: pkubaj
Fixes: 6f0b2a235a1 ("Add pmap_sync_icache() for radix pmap")
MFC after: 3 days

powerpc/pmap: Add pmap_sync_icache() for radix pmap

DTrace pid provider writes to user space to set breakpoints. Failing to
sync the icache can lead to SIGTRAP. Radix pmap is the only one missing
a pmap_sync_icache() method, so the pid provider would only potentially
crash a process on a POWER9 or later system.

kmem_malloc/free: Use void * instead of vm_offset_t for kernel pointers.

Reviewed by: kib, markj
Sponsored by: DARPA
Differential Revision: https://reviews.freebsd.org/D36549

pmap_unmapdev/bios: Accept a pointer instead of a vm_offset_t.

This matches the return type of pmap_mapdev/bios.

Reviewed by: kib, markj
Sponsored by: DARPA
Differential Revision: https://reviews.freebsd.org/D36548

Define _NPCM and the last PC_FREEn constant in terms of _NPCPV.

This applies one of the changes from
5567d6b4419b02a2099527228b1a51cc55a5b47d to other architectures
besides arm64.

Reviewed by: kib
Sponsored by: DARPA
Differential Revision: https://reviews.freebsd.org/D36263

Fix unused variable warning in mmu_radix.c

With clang 15, the following -Werror warning is produced:

sys/powerpc/aim/mmu_radix.c:5409:22: error: variable 'freed' set but not used [-Werror,-Wunused-but-set-variable]
int allfree, field, freed, idx;
^

The 'freed' variable is only used when PV_STATS is defined. Ensure it is
only declared and set in that case.

MFC after: 3 days

Adjust function definitions in mmu_radix.c to avoid clang 15 warnings

With clang 15, the following -Werror warnings are produced:

sys/powerpc/aim/mmu_radix.c:786:20: error: a function declaration without a prototype is deprecated in all versions of C [-Werror,-Wstrict-prototypes]
mmu_radix_tlbie_all()
^
void
sys/powerpc/aim/mmu_radix.c:3615:15: error: a function declaration without a prototype is deprecated in all versions of C [-Werror,-Wstrict-prototypes]
mmu_radix_init()
^
void
sys/powerpc/aim/mmu_radix.c:6081:20: error: a function declaration without a prototype is deprecated in all versions of C [-Werror,-Wstrict-prototypes]
mmu_radix_scan_init()
^
void

This is because mmu_radix_tlbie_all(), mmu_radix_init(), and
mmu_radix_scan_init() are declared with (void) argument lists, but
defined with empty argument lists. Make the definitions match the
declarations.

MFC after: 3 days

powerpc/aim: ifdef DDB pmap debugging functions

INVARIANTS and DDB can be enabled independently.

Submitted by: Sterling Jensen
Pull Request: https://github.com/freebsd/freebsd-src/pull/566

powerpc64: fix the calculation of Maxmem

The calculation of Maxmem was skipping the last phys_avail segment,
because of a wrong stop condition.

This was detected when using QEMU/PowerNV with Radix MMU and low
memory (2G). In this case opal_pci would allocate a DMA window that
was too small to cover all physical memory, resulting in reading all
zeroes from disk when using memory that was not inside the allocated
window.

Reviewed by: jhibbits
Sponsored by: Instituto de Pesquisas Eldorado (eldorado.org.br)
Differential Revision: https://reviews.freebsd.org/D33449
MFC after: 2 weeks

Use the vm_radix_init() helper when initializing pmaps

No functional change intended.

Reviewed by: alc, kib
MFC after: 1 week
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D32527

Convert consumers to vm_page_alloc_noobj_contig()

Remove now-unneeded page zeroing. No functional change intended.

Reviewed by: alc, hselasky, kib
MFC after: 1 week
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D32006

Convert vm_page_alloc() callers to use vm_page_alloc_noobj().

Remove page zeroing code from consumers and stop specifying
VM_ALLOC_NOOBJ. In a few places, also convert an allocation loop to
simply use VM_ALLOC_WAITOK.

Similarly, convert vm_page_alloc_domain() callers.

Note that callers are now responsible for assigning the pindex.

Reviewed by: alc, hselasky, kib
MFC after: 1 week
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D31986

powerpc64: fix OFWFB with Radix MMU

Current implementation of Radix MMU doesn't support mapping
arbitrary virtual addresses, such as the ones generated by
"direct mapping" I/O addresses. This caused the system to hang, when
early I/O addresses, such as those used by OpenFirmware Frame Buffer,
were remapped after the MMU was up.

To avoid having to modify mmu_radix_kenter_attr just to support this
use case, this change makes early I/O map use virtual addresses from
KVA area instead (similar to what mmu_radix_mapdev_attr does), as
these can be safely remapped later.

Reviewed by: alfredo (earlier version), jhibbits (in irc)
MFC after: 2 weeks
Sponsored by: Instituto de Pesquisas Eldorado (eldorado.org.br)
Differential Revision: https://reviews.freebsd.org/D31232

powerpc64: fix radix on pseries TLB invalidation

When running in a virtualized environment, TLB invalidations can only
be performed on process scope, as only the hypervisor is allowed to
invalidate a global scope, or else a Program Interrupt is triggered.

Since we are here, also make sure that the register process table
hypercall returns success.

Reviewed by: jhibbits
MFC after: 2 weeks
Sponsored by: Instituto de Pesquisas Eldorado (eldorado.org.br)
Differential Revision: https://reviews.freebsd.org/D31775

powerpc/pseries: Allow radix pmap in pseries for ISA 3.0

ISA 3.0 allows for nested radix translations with minimal to no
involvement of the hypervisor. This should make pseries signficantly
faster on POWER9 pseries instances, as fewer hypercalls are needed to
manage pmap now.

MFC after: 2 weeks
Relnotes: yes

Assert that valid PTEs are not overwritten when installing a new PTP

amd64 and 32-bit ARM already had assertions to this effect. Add them to
other pmaps.

Reviewed by: alc, kib
MFC after: 2 weeks
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D31171

powerpc64le/pmap: Fix superpage promotions

The page table is always big endian. Without byte swapping on LE, the
promotion ability checks were invalid, and superpage promotions always
failed.

Apply r350463(43ded0a321a) to powerpc64 radix pmap

Invalidate the last page of a demoted superpage mapping, instead of the
first page, as it results in slightly more promotions and fewer
failures. While here, replace 'boolean_t's with 'bool's in
mmu_radix_advise().

Apply r350335(5d18382b728) to powerpc64 radix pmap

Simplify pmap_clear_modify() a bit, by assuming that since the superpage
demotion succeeded, all 4k mappings from it are valid. Deindent the
surrounding code, as there are no 'else' branches in the code anyway.

Apply r355991(50079417a5c28caeca3f423b981047db76e8f5cb) to powerpc64

Micro-optimize control flow in _pmap_unwire_ptp(), and eliminate
unnecessary parentheses.

powerpc64/radix mmu: Remove dead variable

Remove dead variable from mmu_radix_extract_and_hold(). Based on
r352408 for amd64.

powerpc/radix pmap: Convert stat counters from ulongs to counters

This should help performance a hair, for concurrent stat updates, by
reducing contention on cache lines.

powerpc/mmu: Actually use the Radix pmap_align_superpage function

This was missed in the conversion to ifuncs. It might help improve
promotion rates.

powerpc64: add missing TLB invalidations to radix

Radix MMU code was missing TLB invalidations when some Level 3 PDEs were
modified. This caused TLB multi-hit machine check interrupts when
superpages were enabled.

Reviewed by: jhibbits
MFC after: 2 weeks
Sponsored by: Eldorado Research Institute (eldorado.org.br)
Differential Revision: https://reviews.freebsd.org/D29511

Add a VA_IS_CLEANMAP() macro.

This macro returns true if a provided virtual address is contained
in the kernel's clean submap.

In CHERI kernels, the buffer cache and transient I/O map are allocated
as separate regions. Abstracting this check reduces the diff relative
to FreeBSD. It is perhaps slightly more readable as well.

Reviewed by: kib
Obtained from: CheriBSD
Sponsored by: DARPA
Differential Revision: https://reviews.freebsd.org/D28710

Radix dump updates

powerpc/radix_mmu: Apply errata fixes for POWER9 TLB invalidation bug

Found in Linux, the only apparent source of errata documentation.

vm_phys: Try to clean up NUMA KPIs

It can useful for code outside the VM system to look up the NUMA domain
of a page backing a virtual or physical address, specifically when
creating NUMA-aware data structures. We have _vm_phys_domain() for
this, but the leading underscore implies that it's an internal function,
and vm_phys.h has dependencies on a number of other headers.

Rename vm_phys_domain() to vm_page_domain(), and _vm_phys_domain() to
vm_phys_domain(). Make the latter an inline function.

Add _vm_phys.h and define struct vm_phys_seg there so that it's easier
to use in other headers. Include it from vm_page.h so that
vm_page_domain() can be defined there.

Include machine/vmparam.h from _vm_phys.h since it depends directly on
some constants defined there.

Reviewed by: alc
Reviewed by: dougm, kib (earlier versions)
Differential Revision: https://reviews.freebsd.org/D27207

[PowerPC] Don't overwrite vm.pmap sysctl node

After r367417, both mmu_oea64 and mmu_radix were defining the vm.pmap
sysctl node, resulting in the later definition hiding the properties of
the previous one. Avoid this issue by defining vm.pmap in a common
source file and declaring it where needed.

This change also standardizes the tunable name used to enable superpages
and change its default to disabled on radix MMU, because it still has some
issues with superpages.

Reviewed by: bdragon, jhibbits
Sponsored by: Eldorado Research Institute (eldorado.org.br)
Differential Revision: https://reviews.freebsd.org/D27156

[PowerPC64LE] Radix MMU fixes for LE.

There were many, many endianness fixes needed for Radix MMU. The Radix
pagetable is stored in BE (as it is read and written to by the MMU hw),
so we need to convert back and forth every time we interact with it when
running in LE.

With these changes, I can successfully boot with radix enabled on POWER9 hw.

Reviewed by: luporl, jhibbits
Sponsored by: Tag1 Consulting, Inc.
Differential Revision: https://reviews.freebsd.org/D27181

Avoid dump_avail[] redefinition.

Move dump_avail[] extern declaration and inlines into a new header
vm/vm_dumpset.h. This fixes default gcc build for mips.

Reviewed by: alc, scottph
Tested by: kevans (previous version)
Sponsored by: The FreeBSD Foundation
Differential revision: https://reviews.freebsd.org/D26741

Include the psind in data returned by mincore(2).

Currently we use a single bit to indicate whether the virtual page is
part of a superpage. To support a forthcoming implementation of
non-transparent 1GB superpages, it is useful to provide more detailed
information about large page sizes.

The change converts MINCORE_SUPER into a mask for MINCORE_PSIND(psind)
values, indicating a mapping of size psind, where psind is an index into
the pagesizes array returned by getpagesizes(3), which in turn comes
from the hw.pagesizes sysctl. MINCORE_PSIND(1) is equal to the old
value of MINCORE_SUPER.

For now, two bits are used to record the page size, permitting values
of MAXPAGESIZES up to 4.

Reviewed by: alc, kib
Sponsored by: Juniper Networks, Inc.
Sponsored by: Klara, Inc.
Differential Revision: https://reviews.freebsd.org/D26238

powerpc: clean up empty lines in .c and .h files

Fix clang 11 inline asm constraint error when building powerpc GENERIC64
kernels:

sys/powerpc/aim/mmu_radix.c:728:19: error: invalid operand for inline asm constraint 'i'
__asm __volatile(PPC_TLBIEL(%0, %1, %2, %3, 1)
^
sys/powerpc/aim/mmu_radix.c:149:3: note: expanded from macro 'PPC_TLBIEL'
__XSTRING(.long PPC_INST_TLBIEL | \
^
sys/sys/cdefs.h:161:22: note: expanded from macro '__XSTRING'
#define __XSTRING(x) __STRING(x) /* expand x, then stringify */
^
sys/sys/cdefs.h:160:21: note: expanded from macro '__STRING'
#define __STRING(x) #x /* stringify without expanding x */
^
<scratch space>:112:1: note: expanded from here
".long 0x7c000224 | (((%0) & 0x1f) << 11) | (((%1) & 0x1f) << 21) | (((%2) & 0x3) << 18) | (((%3) & 0x1) << 17) | (((1) & 0x1) << 16)"
^

This is solved by making the affected inline functions __always_inline.

Suggested by: jhibbits
MFC after: 3 days

Revert r240317 to prevent leaking pmap entries

Subsequent to r240317, kmem_free() was replaced with kva_free() (r254025).
kva_free() releases the KVA allocation for the mapped region, but no longer
clears the pmap (pagetable) entries.

An affected pmap_unmapdev operation would leave the still-pmap'd VA space
free for allocation by other KVA consumers. However, this bug easily
avoided notice for ~7 years because most devices (1) never call
pmap_unmapdev and (2) on amd64, mostly fit within the DMAP and do not need
KVA allocations. Other affected arch are less popular: i386, MIPS, and
PowerPC. Arm64, arm32, and riscv are not affected.

Reported by: Don Morris <dgmorris AT earthlink.net>
Submitted by: Don Morris (amd64 part)
Reviewed by: kib, markj, Don (!amd64 parts)
MFC after: I don't intend to, but you might want to
Sponsored by: Dell Isilon
Differential Revision: https://reviews.freebsd.org/D25689

Revert r362998, r326999 while a better compatibility strategy is devised.

Migrate the feature of excluding RAM pages to use "excludelist"
as its nomenclature.

MFC after: 1 week

powerpc: Use IFUNCs for copyin/copyout/etc

Summary:
Radix on AIM, and all of Book-E (currently), can do direct addressing of
user space, instead of needing to map user addresses into kernel space.
Take advantage of this to optimize the copy(9) functions for this
behavior, and avoid effectively NOP translations.

Test Plan: Tested on powerpcspe, powerpc64/booke, powerpc64/AIM

Reviewed by: bdragon
Differential Revision: https://reviews.freebsd.org/D25129

powerpc: Add a (CPU/runtime features) flags set to pcpu struct

Summary:
The point of this addition is to cache CPU behavior 'features', to avoid
having to recompute based on CPU, etc.

The first such use case is to avoid the unnecessary manipulation of the
SLBs (Segment Lookaside Buffers) when using the Radix pmap on POWER9.
Since we already get the PCPU pointer wherever we swap the SLB entries,
we can use a cached flag to check if it's necessary to perform the
operation anyway, and skip it when not.

Reviewed by: bdragon
Differential Revision: https://reviews.freebsd.org/D24908

[PowerPC] Fix radix crash when passing -1 from userspace

Found by running libc tests with radix enabled.

Detect unsigned integer wrapping with a postcondition.

Note: Radix MMU is not enabled by default yet.

Sponsored by: Tag1 Consulting, Inc.

powerpc/mmu: Convert PowerPC pmap drivers to ifunc from kobj

With IFUNC support in the kernel, we can finally get rid of our poor-man's
ifunc for pmap, utilizing kobj. Since moea64 uses a second tier kobj as
well, for its own private methods, this adds a second pmap install function
(pmap_mmu_init()) to perform pmap 'post-install pre-bootstrap'
initialization, before the IFUNCs get initialized.

Reviewed by: bdragon

powerpc: Handle machine checks caused by D-ERAT multihit

Instead of crashing the user process when a D-ERAT multihit is detected, try
to flush the ERAT, and continue. This machine check indicates a likely PMAP
invalidation shortcoming that will need to be addressed, but it's
recoverable, so just recover. The recovery is pmap-specific to flush the
ERAT, so add a pmap function to do so, currently only implemented by the
POWER9 radix pmap.

powerpc/radix mmu: No need for delayed TLB invalidation

x86 needs delayed TLB invalidation because invalidation requires an
expensive IPI. PowerPC has had a TLB invalidation instruction since the
POWER1 in 1990, so there's no need to delay anything.

powerpc/mmu: Don't use the cache instructions to zero pages

A page (even physmem) can be marked as cache-inhibited. Attempting to use
'dcbz' to zero a page mapped cache-inhibited triggers an alignment
exception, which is fatal in kernel. This was seen when testing hardware
acceleration with X on POWER9.

At some point in the future, this should be changed to a more straight
forward zero loop instead of bzero(), and a similar change be made to the
other pmaps.

Reported by: pkubaj@

powerpc64: Implement Radix MMU for POWER9 CPUs

Summary:
POWER9 supports two MMU formats: traditional hashed page tables, and Radix
page tables, similar to what's presesnt on most other architectures. The
PowerISA also specifies a process table -- a table of page table pointers--
which on the POWER9 is only available with the Radix MMU, so we can take
advantage of it with the Radix MMU driver.

Written by Matt Macy.

Differential Revision: https://reviews.freebsd.org/D19516