devmap: eliminate unused arguments

The optional 'table' pointer is a legacy part of the interface, which
has been replaced by devmap_register_table()/devmap_add_entry(). The few
in-tree callers have already adapted to this, so it can be removed.

The 'l1pt' argument is already entirely unused within the function.

Reviewed by: andrew, markj
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D45319

arm64: Check DMAP address is valid in PHYS_IN_DMAP

When checking if a physical address is in the DMAP region we assume
all physical addresses between DMAP_MIN_PHYSADDR and DMAP_MAX_PHYSADDR
are able to be accesses through the DMAP. It may be the case that
there is device memory in this range that shouldn't be accessed through
the DMAP mappings.

Add a check to PHYS_IN_DMAP that the translated virtual address is a
valid kernel address. To support code that already checks the address
is valid add PHYS_IN_DMAP_RANGE.

PR: 278233
Reviewed by: alc, markj
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D44677

arm64: Move curthread setup earlier

In 469cfa3c30ee cperciva added TSLOG profiling to link_elf_ireloc. This
requires curthread to be read when the kernel linker is invoked, but it
hadn't yet been initialized. On amd64 this was harmless since [gs:0] was
readable; but on arm64 this broke since [x18] was not readable.

Move the curthread (and associated PCPU) setup earlier on arm64 in order
to allow TSLOG to work there.

Fixes: 469cfa3c30ee ("tslog: Annotate some early boot functions")
Differential Revision: https://reviews.freebsd.org/D44317

arm64: Use void pointers for arguments to arm64_get_writable_addr

No functional change, but this reduces diffs with CheriBSD downstream.

Reviewed by: andrew
Sponsored by: University of Cambridge, Google, Inc.
Differential Revision: https://reviews.freebsd.org/D44344

arm64: Fix the has_hyp check

boot_el is now the CurrentEL.EL field value. Use the new macros to
implement the has_hyp() check.

Sponsored by: Arm Ltd

arm64: Add in_vhe() to find if the kernel is in VHE

Add a function to support devices that may need to know if the kernel
has enabled the Armv8.1 Virtulization Host Extensions (FEAT_VHE).
Some devices, e.g. the generic timer, will need to know, e.g. use a
different interrupt.

Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D43973

arm: Clean up socdev_va

Support socdev_va on arm and ensure the variable is available on arm64.

Reviewed by: imp
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D43359

arm64: Add pmap integration for KMSAN

- In pmap_bootstrap_san(), allocate the root PTPs for the shadow maps.
(For KASAN, this is done earlier since we need to do some special
bootstrapping for the kernel stack.)
- Adjust ifdefs to include KMSAN.
- Expand the shadow maps when pmap_growkernel() is called.

MFC after: 2 weeks
Sponsored by: Klara, Inc.
Sponsored by: Juniper Networks, Inc.
Differential Revision: https://reviews.freebsd.org/D43405

arm64: Enable KMSAN when configured to do so

MFC after: 2 weeks
Sponsored by: Klara, Inc.
Sponsored by: Juniper Networks, Inc.

sys: Automated cleanup of cdefs and other formatting

Apply the following automated changes to try to eliminate
no-longer-needed sys/cdefs.h includes as well as now-empty
blank lines in a row.

Remove /^#if.*\n#endif.*\n#include\s+<sys/cdefs.h>.*\n/
Remove /\n+#include\s+<sys/cdefs.h>.*\n+#if.*\n#endif.*\n+/
Remove /\n+#if.*\n#endif.*\n+/
Remove /^#if.*\n#endif.*\n/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/types.h>/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/param.h>/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/capsicum.h>/

Sponsored by: Netflix

arm64: Make kern_delta unneeded in the boot params

Use pmap_early_vtophys to translate from a virtual to physical where
we were previously using the calculated delta. This means that, while
we still calculate it, we don't need to pass it to initarm or either
pmap bootstrap functions.

While here remove an unneeded printf that indirectly used it or was
related to the previous printf.

Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D42567

arm64: Use pmap_early_vtophys in pmap_bootstrap_san

Use pmap_early_vtophys to find the physical address of the kernel base
rather than using the calculated offset as it will be removed in a
latter commit.

Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D42566

arm64, risvc: warn about ignored kstack_pages for thread0

Suggested by: brooks
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
Differential revision: https://reviews.freebsd.org/D42143

arm64, riscv: Use KSTACK_PAGES for the thread0 kstack size designator

instead of kstack_pages. Although it is correct right now to use
kstack_pages on amd64 since the kern.kstack_pages tunable is not
functional on arm64, this is too fragile and wrong on riscv.

Include opt_kstack_pages.h into machdep.c to get the right definition
for KSTACK_PAGES.

Reviewed by: jhb, markj
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
Differential revision: https://reviews.freebsd.org/D42143

arm64 makectx: Fix overflow of tf_x array

PCB_LR isn't stored in tf_x, so trying to store it as pcb_x[PCB_LR] =
tf->tf_x[PCB_LR + PCB_X_START] overflowed the tf_x array.

Reported by: Morello (bounds check crash)
Reviewed by: jrtc27, andrew, markj
Sponsored by: DARPA
Differential Revision: https://reviews.freebsd.org/D41485

sys: Remove $FreeBSD$: one-line .c pattern

Remove /^[\s*]*__FBSDID\("\$FreeBSD\$"\);?\s*\n/

Add padding for future use on arm64

Allow new features to be supported without changing the size of
existing structures.

Reviewed by: kib
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D39777

Use a uint64_t to store the arm64 mpidr

Use a single uint64_t to hole the mpidr register as we can break the
KBI on 14. Keep the macro so code can still be MFCd to 13.

Sponsored by: Arm Ltd

Make arm64 pcb padding explicit

There is padding between some fields. Mark those I have found so they
can be reused later if needed.

Sponsored by: Arm Ltd

arm64: Ensure that thread0's PCB flags are initialized

On arm64, the PCB is stored at the top of the thread stack. For thread0
this comes from the static "initstack" region, which is placed in the
.init_pagetable section, which is not part of the BSS and thus doesn't
get zeroed by locore. (See the comment in ldscript.arm64.) It is thus
possible for the pcb_flags field to be uninitialized, which can result
in PCB_SINGLE_STEP being set.

Fix this by simply initializing the field. A separate commit will move
initstack out of the .init_pagetable section, since it has no reason to
be there, but it is preferable to explicitly initialize PCB fields
anyway. In particular, regular kernel stacks are not zeroed upon
allocation, so we should be consistent here.

Reviewed by: andrew
MFC after: 1 week
Sponsored by: Klara, Inc.
Sponsored by: Juniper Networks
Differential Revision: https://reviews.freebsd.org/D39343

arm64: add KASAN support

This entails:
- Marking some obvious candidates for __nosanitizeaddress
- Similar trap frame markings as amd64, for similar reasons
- Shadow map implementation

The shadow map implementation is roughly similar to what was done on
amd64, with some exceptions. Attempting to use available space at
preinit_map_va + PMAP_PREINIT_MAPPING_SIZE (up to the end of that range,
as depicted in the physmap) results in odd failures, so we instead
search the physmap for free regions that we can carve out, fragmenting
the shadow map as necessary to try and fit as much as we need for the
initial kernel map. pmap_bootstrap_san() is thus after
pmap_bootstrap(), which still included some technically reserved areas
of the memory map that needed to be included in the DMAP.

The odd failure noted above may be a bug, but I haven't investigated it
all that much.

Initial work by mhorne with additional fixes from kevans and markj.

Reviewed by: andrew, markj
Sponsored by: Juniper Networks, Inc.
Sponsored by: Klara, Inc.
Differential Revision: https://reviews.freebsd.org/D36701

Remove unused registes from the arm pcb

These were kept for ABI reasons. Remove them and bump __FreeBSD_version
so debuggers can be updated to use the new layout.

Reviewed by: jhb
Sponsored by: Arm Ltd
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D35378

arm64: limit EFI excluded regions to physical memory types

Consolidate add_efi_map_entry() and exclude_efi_map_entry() into a
single function, handle_efi_map_entry(), so that the exact set of entry
types handled is the same in the addition or exclusion cases. Before,
exclude_efi_map_entry() had a 'default' case that would exclude all
entry types that were not listed explicitly in the switch statement.

Logically, we do not need to exclude a range that could not possibly be
added to physmem, and we do not need to exclude bus ranges that are not
physical memory, for example EFI_MD_TYPE_IOMEM.

Since physmem's ram0 device will reserve bus memory resources for its
owned ranges, this was preventing attachment of the watchdog device on
the RPI4B. For some reason its region of memory-mapped I/O appeared in
the EFI memory map (with the aforementioned EFI_MD_TYPE_IOMEM type).
This change fixes the attachment issue, as we prevent the physmem API
from messing with this range of bus space.

PR: 270044
Reported by: karels, Mark Millard
Reviewed by: andrew, karels, imp
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D39003

arm64: disable the physical timer for now if HCR_EL2.E2H is set

On some hardware, we can't clear HCR_EL2.E2H so accesses to the physical
timer hopelessly trap to EL2. Stash off the value of HCR_EL2 and use it
in has_hyp() to avoid this.

Reviewed by: andrew
Differential Revision: https://reviews.freebsd.org/D38884

abstract out the vm detection via smbios..

This makes the detection of VMs common between platforms that
have SMBios.

Reviewed by: imp, kib
Differential Revision: https://reviews.freebsd.org/D38800

arm64: Fix thread0.td_kstack_pages init

Commit 86a994d6537d7b5e1efb1019e466d86a688fd570 initialized
thread0.td_kstack_pages to KSTACK_PAGES. Due to the lack of an
include of opt_kstack_pages.h it used the fallback value of 4 from
machine/param.h. This meant that increasing KSTACK_PAGES in the kernel
config resulted in a panic in _epoch_enter_preempt as the following
assertion was false during network stack setup:

MPASS((vm_offset_t)et >= td->td_kstack &&
(vm_offset_t)et + sizeof(struct epoch_tracker) <=
td->td_kstack + td->td_kstack_pages * PAGE_SIZE);

Switch to initializing with kstack_pages following other architectures.

Reviewed by: imp, markj
Sponsored by: DARPA, AFRL
Differential Revision: https://reviews.freebsd.org/D38048

arm64/machdep: Reserve memory when we find Linux EFI reserved memory table

When Linux loads a new kernel via kexec, somtiems it must reserve memory
for devices that are still active (and typically can't be reset or
shutdown). When present, this table is a linked list of ranges that are
still in use that the OS must avoid using.

Mark these areas as reserved.

This is part of the GICv3 workaround code where we must use the PA
addresses already programmed into the GICv3 when we take over. This part
ensure we don't allocate the mmeory for anything else.

Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D37440

arm64/machdep: Add parameter to the EFI table walking code

It would be nice to be able to pass an arbitrary pointer to the callback
code. Add one, and pass NULL in all the places that we do that today.
As noted by andrew@, we should likely refactor this into MI code and use
it here and amd64, but for the future.

Sponsored by: Netflix
Reviewed by: rpokala
Differential Revision: https://reviews.freebsd.org/D37439

arm64/machdep: Delete trailing whitespace

No functional change.

Sponsored by: Netflix

arm64: fix build without FDT

static_kenv is only used under `FDT`, and `try_load_dtb` is only defined
with `FDT`.

Reviewed by: andrew, imp, manu
Differential Revision: https://reviews.freebsd.org/D36791

Remove unneeded variables in the arm64 pmap bootstrap

These are now unneeded after cleaning up the pmap bootstrap process.
Remove them and the variables that set them.

Sponsored by: The FreeBSD Foundation

arm64: don't loop forever if first option in kern.cfg.order not available

strchr returns a pointer to the ',', so if the first option in the list
isn't available, we need to step over the , to look at the next
option. So if kern.cfg.order="acpi,fdt" and we have no acpi, we'd loop
forever with order=',fdt'.

Sponsored by: Netflix
Reviewed by: andrew, jhb
Differential Revision: https://reviews.freebsd.org/D36682

arm64: machdep.c fix two typos

Sponsored by: Netflix

Store mpidr as a 64-bit value on arm64

The mpidr register is 64 bit on arm64 and 32 bit on arm. Fix this by
extending the arm64 definition to include the top 32 bits.

To preserve KBI when MFCing split the value into two 32 bit values.
This will be cleaned up later only on main.

Reviewed by: bz
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D36346

arm64: Remove unused typedef

We don't use EFI_MEMORY_DESCRIPTOR that's typedef'd here. We use the one
from sys/efi.h instead. Remove the clutter here as these two are subtly
different (though wind up with the same layout due to alignment rules).

Sponsored by: Netflix

Map the ACPI tables into the DMAP

When we try to load these tables via acpidump(8) we need them to be in
the DMAP for /dev/mem to access. Add the EFI ACPI reclaim memory type
to the list of memory we map into DMAP but not used by the kernel as
this is the recommended place to put these.

Sponsored by: The FreeBSD Foundation

Include the EFI Runtime Code in the DMAP

Some UEFI implementations place the system table in a runtime code
memory region. Include it in the DMAP so we can read it later.

Reviewed by: kevans
Differential Revision: https://reviews.freebsd.org/D34861

Fix the spelling of EFI_PAGE_SIZE

We assume EFI_PAGE_SIZE is the same as PAGE_SIZE, however this may not
be the case. Use the former when working with a list of pages from the
UEFI firmware so the correct size is used.

This will be needed on arm64 where PAGE_SIZE could be 16k or 64k in the
future. The other architectures have been updated to be consistent.

Reviewed by: imp
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D34510

arm64, riscv: handle RB_KDB

This allows entering the debugger at the earliest possible time, if
the '-d' argument is passed to the kernel.

Reviewed by: jhb
Sponsored by: The FreeBSD Foundation
MFC after: 3 days
Differential Revision: https://reviews.freebsd.org/D34120

Add arm64 pointer authentication support

Pointer authentication allows userspace to add instructions to insert
a Pointer Authentication Code (PAC) into a register based on an address
and modifier and check if the PAC is correct. If the check fails it will
either return an invalid address or fault to the kernel.

As many of these instructions are a NOP when disabled and in earlier
revisions of the architecture this can be used, for example, to sign
the return address before pushing it to the stack making Return-oriented
programming (ROP) attack more difficult on hardware that supports them.

The kernel manages five 128 bit signing keys: 2 instruction keys, 2 data
keys, and a generic key. The instructions then use one of these when
signing the registers. Instructions that use the first four store the
PAC in the register being signed, however the instructions that use the
generic key store the PAC in a separate register.

Currently all userspace threads share all the keys within a process
with a new set of userspace keys being generated when executing a new
process. This means a forked child will share its keys with its parent
until it calls an appropriate exec system call.

In the kernel we allow the use of one of the instruction keys, the ia
key. This will be used to sign return addresses in function calls.
Unlike userspace each kernel thread has its own randomly generated.

Thread0 has a static key as does the early code on secondary CPUs.
This should be safe as there is minimal user interaction with these
threads, however we could generate random keys when the Armv8.5
Random number generation instructions are present.

Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D31261

Per-thread stack canary on arm64

With the update to llvm 13 we are able to tell the compiler it can find
the SSP canary relative to the register that holds the userspace stack
pointer. As this is unused in most of the kernel it can be used here
to point to a per-thread SSP canary.

As the kernel could be built with an old toolchain, e.g. when upgrading
from 13, add a warning that the options was enabled but the compiler
doesn't support it to both the build and kernel boot.

Discussed with: emaste
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D33079

Allocate arm64 per-CPU data in the correct domain

To minimise NUMA traffic allocate the pcpu, dpcpu, and boot stacks in
the correct domain when possible.

Submitted by: markj
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D32338

arm64: move signal delivery code to exec_machdep.c

also move ptrace-related helpers to ptrace_machdep.c.

Reviewed by: emaste, imp
Discussed with: jrtc27
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
Differential revision: https://reviews.freebsd.org/D32310

Allow ddb and dtrace use the DMAP region on arm64

When writing to memory on arm64 we may be trying to be accessing a
read-only page. In this case try to access via the DMAP region to
get a writable location.

While here simplify writing data in DDB and stop trashing the size as
it is passed into the cache handling functions.

Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D32053

Restrict spsr updated in the arm64 set_regs*

When using ptrace(2) on arm64 to set registers in a 32-bit program we
need to take care to only set some of the fields. Follow the existing
arm64 path and only let the user set the flags fields. This is also the
case in the arm kernel so fixes a change in behaviour between the two.

While here update set_regs to only set spsr and elr once.

Sponsored by: The FreeBSD Foundation

Create sys/reg.h for the common code previously in machine/reg.h

Move the common kernel function signatures from machine/reg.h to a new
sys/reg.h. This is in preperation for adding PT_GETREGSET to ptrace(2).

Reviewed by: imp, markj
Sponsored by: DARPA, AFRL (original work)
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D19830

Enable arm64 SError exceptions in the kernel

These are needed to signal to the kernel when a Reliability,
Availability, and Serviceability (RAS) exception has triggered.

Reviewed by: mhorne
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D31477

Start to clean up arm64 address space selection

On arm64 we should use bit 55 of the address to decide if aan address
is a user or kernel address. Add a new macro with this check and a
second to ensure the address is in teh canonical form, i.e.
the top bits are all zero or all one.

This will help with supporting future cpu features, including Top
Byte Ignore, Pointer Authentication, and Memory Tagging.

Reviewed by: kib
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D31179

arm64: Instrument initarm with TSLOG

The "function entry" needs to be recorded with TSRAW() rather than
TSENTER() since curthread is not yet defined. (The same approach
is taken in amd64's hammer_time.)

arm64: Disable early printf if TSLOG

The warning message "ERROR loading DTB" (for systems without a device
tree blob) is printed extremely early in the boot process -- among
other things, before curthread or other pcpu data has been set up.
Unfortunately, printf is instrumented with TSLOG, which cannot run
quite this early.

Wrap the printf in #ifndef TSLOG; the situations where the printf
will be useful are not ones where TSLOG would be in use.

Use the correct length when copying arm64 vfp registers

We passed the wrong length into memcpy in the arm64 get_fpcontext and
set_fpcontext. This caused us to copy two status registers we didn't
expect to copy.

These are safe as they exist in both the source and destination, although
in a different order, and we copy the correct values after the memcpy.

Obtained from: CheriBSD
Sponsored by: Innovate UK

Switch the EFI virtual address to a uint64_t

It is defined as a uint64_t in the UEFI spec. As it's not used as a
pointer by the kernel follow this and define it as the same in the
kernel.

Reviewed by: kib, manu, imp
Sponsored by: Innovate UK
Differential Revision: https://reviews.freebsd.org/D29759

arm64: clear debug registers after execve(2)

This is both intuitive and required, as any previous breakpoint settings
may not be applicable to the new process.

Reported by: arichardson
Reviewed by: kib
MFC after: 3 days
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D29672

arm64: Only check for freebsd,dts-version if we are booted in FDT mode.

Reported by: andrew

arm64: Check dtb version against the one we're expecting to find

Do for arm64 what was done for armv7 in e63faa9ba832b6

arm64: implement COMPAT_FREEBSD32 fill/set dbregs/fpregs

On FreeBSD/arm fill_fpregs, fill_dbregs are stubs that zero the reg
struct and return success. set_fpregs and set_dbregs do nothing and
return success.

Provide the same implementation for arm64 COMPAT_FREEBSD32.

Reviewed by: andrew
MFC after: 3 days
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D29314

aarch64: Clear TLS registers during exec().

These are not stored in the trapframe so must be cleared explicitly.

This is similar to one of the MIPS changes in 822d2d6ac94f.

Reviewed by: andrew
Obtained from: CheriBSD
MFC after: 1 week
Sponsored by: DARPA
Differential Revision: https://reviews.freebsd.org/D28711

Arch64: Clear VFP state on execve()

I noticed that many of the math-related tests were failing on AArch64.
After a lot of debugging, I noticed that the floating point exception flags
were not being reset when starting a new process. This change resets the
VFP inside exec_setregs() to ensure no VFP register state is leaked from
parent processes to children.

This commit also moves the clearing of fpcr that was added in 65618fdda0f27
from fork() to execve() since that makes more sense: fork() can retain
current register values, but execve() should result in a well-defined
clean state.

Reviewed By: andrew
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D29060

arm64: extend struct db_reg to include watchpoint registers

The motivation is to provide access to these registers from userspace
via ptrace(2) requests PT_GETDBREGS and PT_SETDBREGS.

This change breaks the ABI of these particular requests, but is
justified by the fact that the intended consumers (debuggers) have not
been taught to use them yet. Making this change now enables active
upstream work on lldb to begin using this interface, and take advantage
of the hardware debugging registers available on the platform.

PR: 252860
Reported by: Michał Górny (mgorny@gentoo.org)
Reviewed by: andrew, markj (earlier version)
Tested by: Michał Górny (mgorny@gentoo.org)
MFC after: 1 week
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D28415

arm64: validate breakpoint registers

In particular, we want to disallow setting breakpoints on kernel
addresses from userspace. The control register fields are validated or
ignored as appropriate.

Reviewed by: markj
MFC after: 1 week
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D28560

arm64: remove pcb_pc

The program counter field in the PCB is written in exactly one place,
makectx(), upon entry to the debugger. For threads other than curthread,
its value will be empty, or bogus. Rather than writing to this field in
more places, it can be removed in favor of using the value in the link
register.

To make this clearer, pcb->pcb_x[30] is renamed to pcb->pcb_lr, similar
to what already exists in struct trapframe. Also, prefer lr to x30 in
assembly, as it better conveys intention.

This improves PC_REGS() for kdb_thread != curthread. It is required for
a functional gdb(4) stub, fixing the output of `info threads`, in
particular.

The space occupied by pcb_pc is retained, for compatibility with kgdb.

Reviewed by: markj, jhb
MFC after: 2 weeks
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D27720

Simplify startup of secondary cores and store MPIDR register to pcpu.

- record MPIDR for all started cores in pcpu, they will be used as link
between physical locality of given core, ID in external description
(FDT or ACPI) and cupid.
- because of above, cpuid can (and should) be freely assigned, only boot
CPU must have cpuid 0. Simplify startup code according this.

Please note that pure cpuid is not sufficient instrument to hold any
information about core or cluster topology, nor to determistically iterate
over subpart of cores in CPU (iterate over all cores in single cluster for
example). Situation is more complicated by fact that PSCI can reject start
of core without reporting error (because power budget for example), or by
fact that is possible that we booted on non-first core in cluster (thus with
cpuid 0 assigned to random core).

Given cores topology should be exhibited to other parts of system
(for example to scheduler for big.little or multicluster systems) by using
smp_topo interface.

Differential Revision: https://reviews.freebsd.org/D13863

arm64: Do not rely on SPCR table to detect acpi

Since EDK2 commit d8e36289cef7bde628b023219cd65fa8e8d4562a, the Graphical console may
completely hide SPCR, causing panics later when locating timers.
As such simply rely on the ACPI Root pointer presence.

Submitted by: dan.kotowski@a9development.com
Reviewed by: andrew, mw
Differential Revision: https://reviews.freebsd.org/D27306

Simplify preload_dump() condition

Hiding this feature behind RB_VERBOSE is gratuitous. The tunable is enough
to limit its use to only those who explicitly request it.

Suggested by: kevans

Add a routine to dump boot metadata

The boot metadata (also referred to as modinfo, or preload metadata)
provides information about the size and location of the kernel,
pre-loaded modules, and other metadata (e.g. the EFI framebuffer) to be
consumed during by the kernel during early boot. It is encoded as a
series of type-length-value entries and is usually constructed by
loader(8) and passed to the kernel. It is also faked on some
architectures when booted by other means.

Although much of the module information is available via kldstat(8),
there is no easy way to debug the metadata in its entirety. Add some
routines to parse this data and allow it to be printed to the console
during early boot or output via a sysctl.

Since the output can be lengthly, printing to the console is gated
behind the debug.dump_modinfo_at_boot kenv variable as well as the
BOOTVERBOSE flag. The sysctl to print the metadata is named
debug.dump_modinfo.

Reviewed by: tsoome
Sponsored by: NetApp, Inc.
Sponsored by: Klara, Inc.
Differential Revision: https://reviews.freebsd.org/D26687

Don't try to print EFI memeory map if it doesn't exist.

MFC after: 1 week

Switch to an empty ttbr0 pagetable when the MMU is enabled

We don't need these pagetables after the early boot. Remove the chance we
write to memory we didn't expect to and remove architectural undefined
behaviour.

Reviewed by: alc (earlier version), mmel
Sponsored by: Innovate UK
Differential Revision: https://reviews.freebsd.org/D22606

arm64: clean up empty lines in .c and .h files

Add initial driver for ACPI Platform Error Interfaces.

APEI allows platform to report different kinds of errors to OS in several
ways. We've found that Supermicro X10/X11 motherboards report PCIe errors
appearing on hot-unplug via this interface using NMI. Without respective
driver it ended up in kernel panic without any additional information.

This driver introduces support for the APEI Generic Hardware Error Source
reporting via NMI, SCI or polling. It decodes the reported errors and
either pass them to pci(4) for processing or just logs otherwise. Errors
marked as fatal still end up in kernel panic, but some more informative.

When somebody get to native PCIe AER support implementation both of the
reporting mechanisms should get common error recovery code. Since in our
case errors happen when the device is already gone, there is nothing to
recover, so the code just clears the error statuses, practically ignoring
the otherwise destructive NMIs in nicer way.

MFC after: 2 weeks
Relnotes: yes
Sponsored by: iXsystems, Inc.

Print arm64 physmem info during boot.

PR: 243682
Reviewed by: andrew, emaste
MFC after: 1 week
Event: July 2020 Bugathon
Differential Revision: https://reviews.freebsd.org/D25625

Rerun kernel ifunc resolvers after all CPUs have started

On architectures that use RELA relocations it is safe to rerun the ifunc
resolvers on after all CPUs have started, but while they are sill parked.

On arm64 with big.LITTLE this is needed as some SoCs have shipped with
different ID register values the big and little clusters meaning we were
unable to rely on the register values from the boot CPU.

Add support for rerunning the resolvers on arm64 and amd64 as these are
both RELA using architectures.

Reviewed by: kib
Sponsored by: Innovate UK
Differential Revision: https://reviews.freebsd.org/D25455

Read the CPU 0 arm64 ID registers early in initarm

We also update the kernel view early in the boot. This will allow the
use of the common kernel view in ifunc resolvers.

Sponsored by: Innovate UK

Use EFI memory map to determine attributes for Acpi mappings on arm64.

AcpiOsMapMemory is used for device memory when e.g. an _INI method wants
to access physical memory, however, aarch64 pmap_mapbios is hardcoded to
writeback. Search for the correct memory type to use in pmap_mapbios.

Submitted by: Greg V <greg_unrelenting.technology>
Differential Revision: https://reviews.freebsd.org/D25201

Convert arm's physmem interface to MI code

The arm_physmem interface found in arm's MD code provides a convenient
set of routines for adding/excluding physical memory regions and
initializing important kernel globals such as Maxmem, realmem,
phys_avail[], and dump_avail[]. It is especially convenient for FDT
systems, since we can use FDT parsing functions and pass the result
directly to one of these physmem routines. This interface is already in
use on arm and arm64, and can be used to simplify this early
initialization on RISC-V as well.

This requires only a couple trivial changes:
- Move arm_physmem_kernel_addr to arm/machdep.c. It is unused on arm64,
and manipulated entirely in arm MD code.
- Convert arm32_btop/arm64_btop to atop. This is equivalently defined
on all architectures.
- Drop the "arm" prefix.

Reviewed by: manu, emaste ("looks reasonable")
MFC after: 2 weeks
Differential Revision: https://reviews.freebsd.org/D24153

Move the arm64 cache identification to identcpu.c

This allows us to call it on a per-CPU basis and to warn if the details
are different across CPUs.

While here read the L1 I-Cache type and store this for use later by pmap.

Sponsored by: Innovate UK

Store the boot exception level on arm64 so it can be queried later

A hypervisor, e.g. bhyve, will need to know what exception levelthe kernel
was in when it started booting. If it was EL2 we can then enable said
hypervisor.

Store the boot exception level and allow the kernel to later query it.

Obtained from: https://github.com/FreeBSD-UPB/freebsd (earlier version)
Sponsored by: Innovate UK

Add more arm64 CTR_EL0 register fields

While here make the _SIZE macros return the size in bytes, not the log2
of the size

Sponsored by: Innovate UK

Save and restore floating point registers in get/set_mcontext().

arm64 and riscv were only saving and restoring floating point
registers for sendsig() and sys_sigreturn(), but not for getcontext(),
setcontext(), and swapcontext().

While here, remove an always-false check for uap being NULL from
sys_sigreturn().

Reviewed by: br, mhorne
Sponsored by: DARPA
Differential Revision: https://reviews.freebsd.org/D23218

Repeat the spinlock_enter/exit pattern from amd64 on other architectures to
fix an assert violation introduced in r355784. Without this spinlock_exit()
may see owepreempt and switch before reducing the spinlock count. amd64
had been optimized to do a single critical enter/exit regardless of the
number of spinlocks which avoided the problem and this optimization had
not been applied elsewhere.

Reported by: emaste
Suggested by: rlibby
Discussed with: jhb, rlibby
Tested by: manu (arm64)

Add support for booting kernel directly from U-Boot using booti command.

In some cases, like is locked bootstrap or device's inability to boot from
removable media, we cannot use standard boot sequence and is necessary to
boot kernel directly from U-Boot.

Discussed with: jhibbits
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D13861

Use uintptr_t instead of register_t * for the stack base.

- Use ustringp for the location of the argv and environment strings
and allow destp to travel further down the stack for the stackgap
and auxv regions.
- Update the Linux copyout_strings variants to move destp down the
stack as was done for the native ABIs in r263349.
- Stop allocating a space for a stack gap in the Linux ABIs. This
used to hold translated system call arguments, but hasn't been used
since r159992.

Reviewed by: kib
Tested on: md64 (amd64, i386, linux64), i386 (i386, linux)
Sponsored by: DARPA
Differential Revision: https://reviews.freebsd.org/D22501

revert r354714 "Boot arm64 kernel using booti command from U-boot."

After discussing with mmel@, it was clear this is insufficient to address
all the needs. mmel@ will commit his original patch, from
https://reviews.freebsd.org/D13861, and the additions needed from r354714
will be made afterward.

Requested by: mmel
Sponsored by: Juniper Networks, Inc.

Port the NetBSD KCSAN runtime to FreeBSD.

Update the NetBSD Kernel Concurrency Sanitizer (KCSAN) runtime to work in
the FreeBSD kernel. It is a useful tool for finding data races between
threads executing on different CPUs.

This can be enabled by enabling KCSAN in the kernel config, or by using the
GENERIC-KCSAN amd64 kernel. It works on amd64 and arm64, however the later
needs a compiler change to allow -fsanitize=thread that KCSAN uses.

Sponsored by: DARPA, AFRL
Differential Revision: https://reviews.freebsd.org/D22315

Return 0 from ptrace_set_pc as it now completes successfully.

Sponsored by: DARPA, AFRL

Allow ptrace to set the probram counter on arm64.

Sponsored by: DARPA, AFRL

Boot arm64 kernel using booti command from U-boot.

Summary:
Boot arm64 kernel using booti command from U-boot. booti can relocate initrd
image into higher ram addresses, therefore align the initrd load address to 1GiB
and create VA = PA map for it. Create L2 pagetable entries to copy the initrd
image into KVA.
(parts of the code in https://reviews.freebsd.org/D13861 was referred and used
as appropriate)

Submitted by: Siddharth Tuli <siddharthtuli_gmail.com>
Reviewed by: manu
Sponsored by: Juniper Networks, Inc
Differential Revision: https://reviews.freebsd.org/D22255

Utilize ASIDs to reduce both the direct and indirect costs of context
switching. The indirect costs being unnecessary TLB misses that are
incurred when ASIDs are not used. In fact, currently, when we perform a
context switch on one processor, we issue a broadcast TLB invalidation that
flushes the TLB contents on every processor.

Mark all user-space ("ttbr0") page table entries with the non-global flag so
that they are cached in the TLB under their ASID.

Correct an error in pmap_pinit0(). The pointer to the root of the page
table was being initialized to the root of the kernel-space page table
rather than a user-space page table. However, the root of the page table
that was being cached in process 0's md_l0addr field correctly pointed to a
user-space page table. As long as ASIDs weren't being used, this was
harmless, except that it led to some unnecessary page table switches in
pmap_switch(). Specifically, other kernel processes besides process 0 would
have their md_l0addr field set to the root of the kernel-space page table,
and so pmap_switch() would actually change page tables when switching
between process 0 and other kernel processes.

Implement a workaround for Cavium erratum 27456 affecting ThunderX machines.
(I would like to thank andrew@ for providing the code to detect the affected
machines.)

Address integer overflow in the definition of TCR_ASID_16.

Setup TCR according to the PARange and ASIDBits fields from
ID_AA64MMFR0_EL1. Previously, TCR_ASID_16 was unconditionally set.

Modify build_l1_block_pagetable so that lower attributes, such as ATTR_nG,
can be specified as a parameter.

Eliminate some unused code.

Earlier versions were tested to varying degrees by: andrew, emaste, markj

MFC after: 3 weeks
Differential Revision: https://reviews.freebsd.org/D21922

Add support for setting hardware breakpoints from ptrace on arm64.

Implement get/fill_dbregs on arm64. This is used by ptrace with the
PT_GETDBREGS and PT_SETDBREGS requests. It allows userspace to set hardware
breakpoints.

The struct dbreg is based on Linux to ease adding hardware breakpoint
support to debuggers.

Reviewed by: jhb
Sponsored by: DARPA, AFRL
Differential Revision: https://reviews.freebsd.org/D22195

Fix a typo in r353895.

Reported by: andrew
MFC after: 3 days
Sponsored by: The FreeBSD Foundation

Allow exceptions to be masked when in userspace

We may want to mask exceptions when in userspace. This was previously
impossible as threads are created with all exceptions unmasked and
signals expected userspace to mask any. Fix these by copying the
mask state on thread creation and allow exceptions to be masked on
signal return, as long as they don't change.

Sponsored by: DARPA, AFRL

Use a lowercase name for arm64 special registers so they don't conflict
with macros of the same name.

Sponsored by: DARPA, AFRL

Add two files missed in r354170

Sponsored by: DARPA, AFRL

Apply r353893 to arm64.

Reported by: Jenkins (hardware CI lab)
MFC after: 1 week
Sponsored by: The FreeBSD Foundation

Fix debugging of 32bits arm binaries on arm64.

In set_regs32()/fill_regs32(), we have to get/set SP and LR from/to
tf_x[13] and tf_x[14].
set_regs() and fill_regs() may be called for a 32bits process, if the process
is ptrace'd from a 64bits debugger. So, in set_regs() and fill_regs(), get
or set PC and SPSR from where the debugger expects it, from tf_x[15] and
tf_x[16].

Extract eventfilter declarations to sys/_eventfilter.h

This allows replacing "sys/eventfilter.h" includes with "sys/_eventfilter.h"
in other header files (e.g., sys/{bus,conf,cpu}.h) and reduces header
pollution substantially.

EVENTHANDLER_DECLARE and EVENTHANDLER_LIST_DECLAREs were moved out of .c
files into appropriate headers (e.g., sys/proc.h, powernv/opal.h).

As a side effect of reduced header pollution, many .c files and headers no
longer contain needed definitions. The remainder of the patch addresses
adding appropriate includes to fix those files.

LOCK_DEBUG and LOCK_FILE_LINE_ARG are moved to sys/_lock.h, as required by
sys/mutex.h since r326106 (but silently protected by header pollution prior
to this change).

No functional change (intended). Of course, any out of tree modules that
relied on header pollution for sys/eventhandler.h, sys/lock.h, or
sys/mutex.h inclusion need to be fixed. __FreeBSD_version has been bumped.

Impleent COMPAT_FREEBSD32 for arm64.
This is based on early work by andrew@.

Fix reporting of SS_ONSTACK

Fix reporting of SS_ONSTACK in nested signal delivery when sigaltstack()
is used on some architectures.

Add a unit test for this. I tested the test by introducing the bug
on amd64. I did not test it on other architectures.

Reviewed by: kib
MFC after: 2 weeks
Sponsored by: Dell EMC Isilon
Differential Revision: https://reviews.freebsd.org/D18347

Prevent kernel stack disclosure in signal delivery

On arm64 and riscv platforms, sendsig() failed to zero the signal
frame before copying it out to userspace. Zero it.

On arm, I believe all the contents of the frame were initialized,
so there was no disclosure. However, explicitly zero the whole frame
because that fact could inadvertently change in the future,
it's more clear to the reader, and I could be wrong in the first place.

MFC after: 2 days
Security: similar to FreeBSD-EN-18:12.mem and CVE-2018-17155
Sponsored by: Dell EMC Isilon

Add kernel ifunc support on arm64.

Tested with ifunc resolvers in the kernel and module with calls from
kernel to kernel, module to kernel, and module to module.

Reviewed by: kib (previous version)
Approved by: re (gjb)
Differential Revision: https://reviews.freebsd.org/D17370

Clear all of the VFP state in fill_fpregs().

Zero the entire FP register set structure returned for ptrace() if a
thread hasn't used FP registers rather than leaking garbage in the
fp_sr and fp_cr fields.

Reviewed by: emaste, andrew
Approved by: re (rgrimes)
MFC after: 2 weeks
Sponsored by: DARPA, AFRL
Differential Revision: https://reviews.freebsd.org/D17140

Exclude the EFI framebuffer from phys_avail[] on arm64.

On the ThunderX the region occupied by the framebuffer is included in
the EFI map, so explicitly add it to the set of regions that aren't
managed by the physical memory allocator.

PR: 231064
Reviewed by: andrew
Approved by: re (gjb)
MFC after: 1 week
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D17073

Move the code to print the EFI memory table to a new function and call it
in teh bootverbose path after cninit().

This allows users to see these tables when booting with boot -v.

Sponsored by: DARPA, AFRL

Fix the early spelling of bootverbose.

Sponsored by: DARPA, AFRL

Create a new function to walk the EFI memory table & run a callback for
each entry. We can then use this to ensure the RunTime data is mapped in
the DMAP, but not in phys_avail.

Sponsored by: DARPA, AFRL

Allow us to read the physmap data into our own array and use this to build
the DMAP region on arm64.

We already have the needed information to build these tables, we just need
to extract it. This significantly simplifies the code.

Obtained from: ABT Systems Ltd
Sponsored by: Turing Robotic Industries

Print the physmem tables under a verbose boot.

Obtained from: ABT Systems Ltd
Sponsored by: Turing Robotic Industries

Exclude memory from the /reserved-memory mappings with the no-map property
set. This memory must not be mapped by the operating system other than
under control of the device driver.

Obtained from: ABT Systems Ltd
Sponsored by: Turing Robotic Industries

Switch arm64 to use the same physmem code as 32-bit arm.

The main advantage of this is to allow us to exclude memory from being
used by the kernel. This may be from the memreserve property, or ranges
marked as no-map under the reserved-memory node.

More work is still needed to remove the physmap array. This is still used
for creating the DMAP region, however other patches need to be committed
before we can remove this.

Obtained from: ABT Systems Ltd
Sponsored by: Turing Robotic Industries

Move most of the contents of opt_compat.h to opt_global.h.

opt_compat.h is mentioned in nearly 180 files. In-progress network
driver compabibility improvements may add over 100 more so this is
closer to "just about everywhere" than "only some files" per the
guidance in sys/conf/options.

Keep COMPAT_LINUX32 in opt_compat.h as it is confined to a subset of
sys/compat/linux/*.c. A fake _COMPAT_LINUX option ensure opt_compat.h
is created on all architectures.

Move COMPAT_LINUXKPI to opt_dontuse.h as it is only used to control the
set of compiled files.

Reviewed by: kib, cem, jhb, jtl
Sponsored by: DARPA, AFRL
Differential Revision: https://reviews.freebsd.org/D14941

Set the base address of translation table 0.

This fixes operation on Qualcomm Snapdragon and some other platforms.

During boot time on subsystems initialization we have some amount of
kernel threads created, then scheduler gives CPU time to each thread.
Eventually scheduler returns CPU execution back to thread 0. In this
case writing zero to ttbr0 in cpu_switch leads Qualcomm board to
reboot (asynchronously, CPU continues execution).

Similar to other kernel threads install a valid physical address
(kernel pmap) to user page table base register ttbr0.

Reviewed by: andrew
Sponsored by: DARPA, AFRL
Differential Revision: https://reviews.freebsd.org/D13536

Add a framework to install CPU errata on arm64. Each erratum can encode
a mask and value to compare with the Main ID Register. If these match then a
function is called to handle the installation of the erratum workaround.

No errata are currently handled, however this will change soon in a future
commit.

MFC after: 1 week
Sponsored by: DARPA, AFRL

Do pass removing some write-only variables from the kernel.

This reduces noise when kernel is compiled by newer GCC versions,
such as one used by external toolchain ports.

Reviewed by: kib, andrew(sys/arm and sys/arm64), emaste(partial), erj(partial)
Reviewed by: jhb (sys/dev/pci/* sys/kern/vfs_aio.c and sys/kern/kern_synch.c)
Differential Revision: https://reviews.freebsd.org/D10385

Set the kernel file name so the kern.bootfile sysctl is set. This allows
for the creation of kernel.old when running installkernel.

MFC after: 1 week
Sponsored by: DARPA, AFRL

Add a Saved Process Status Register bit for AArch32 execution mode.

The documentation on the Saved Process Status Register (SPSR) is a bit
weird; the M[4] bit is documented separately from M[3:0]. The M[4] bit
can be toggled to switch to 32-bit execution mode. This functionality is
orthogonal to M[3:0].

Change the definition of PSR_M_MASK to no longer include M[4]. Add a new
definition, PSR_AARCH32 that can be used to toggle 32-bit independently.
This bit will be used by the cloudabi32 code to force execution of
userspace code in 32-bit mode.

Reviewed by: andrew
Differential Revision: https://reviews.freebsd.org/D13148

Add rudimentary support for building FreeBSD/arm64 with COMPAT_FREEBSD32.

Right now I'm using two Raspberry Pi's (2 and 3) to test CloudABI
support for armv6, armv7 and aarch64. It would be nice if I could
restrict this to just a single instance when testing smaller changes.
This is why I'd like to get COMPAT_CLOUDABI32 to work on arm64.

As COMPAT_CLOUDABI32 depends on COMPAT_FREEBSD32, at least for the ELF
loading, this change adds all of the bits necessary to at least build a
kernel with COMPAT_FREEBSD32. All of the machine dependent system calls
are still stubbed out, for the reason that implementations for these are
only useful if actual support for running FreeBSD binaries is added.
This is outside the scope of this work.

Reviewed by: andrew
Differential Revision: https://reviews.freebsd.org/D13144

Don't let cpu_set_syscall_retval() clobber exec_setregs().

Upon successful completion, the execve() system call invokes
exec_setregs() to initialize the registers of the initial thread of the
newly executed process. What is weird is that when execve() returns, it
still goes through the normal system call return path, clobbering the
registers with the system call's return value (td->td_retval).

Though this doesn't seem to be problematic for x86 most of the times (as
the value of eax/rax doesn't matter upon startup), this can be pretty
frustrating for architectures where function argument and return
registers overlap (e.g., ARM). On these systems, exec_setregs() also
needs to initialize td_retval.

Even worse are architectures where cpu_set_syscall_retval() sets
registers to values not derived from td_retval. On these architectures,
there is no way cpu_set_syscall_retval() can set registers to the way it
wants them to be upon the start of execution.

To get rid of this madness, let sys_execve() return EJUSTRETURN. This
will cause cpu_set_syscall_retval() to leave registers intact. This
makes process execution easier to understand. It also eliminates the
difference between execution of the initial process and successive ones.
The initial call to sys_execve() is not performed through a system call
context.

Reviewed by: kib, jhibbits
Differential Revision: https://reviews.freebsd.org/D13180

Ensure we check the program state set in the trap frame on arm and arm64.
This value may be set by userspace so we need to check it before using it.
If this is not done correctly on exception return the kernel may continue
in kernel mode with all registers set to a userspace controlled value. Fix
this by moving the check into set_mcontext, and also add the missing
sanitisation from the arm64 set_regs.

Discussed with: security-officer@
MFC after: 3 days
Sponsored by: DARPA, AFRL

Support the EFI Runtime Services on arm64. As with amd64 we use the 1:1
mapping. This uses the new common code shared with amd64.

The RTC should only be accessed via EFI. There is no locking around it as
the spec only has this as a requirement for the PC-AT CMOS device.

Reviewed by: kib, imp
Sponsored by: DARPA, AFRL
Differential Revision: https://reviews.freebsd.org/D12595

Add support for handling undefined instructions in userspace and the
kernel. We can register callbacks to perform the required operation on the
saved registers before returning.

This is initially used to work around a bug in old versions of QEMU that
trigger such an exception when reading from an ID register when it should
load z zero value.

I expect this could be used with other exception types, e.g. to emulate
special register access from userland.

Sponsored by: DARPA, AFRL

arm64: return error instead of panic in unimplemented ptrace ops

We don't need a panic as a reminder that these need to be implemented.

Reported by: Shawn Webb
MFC after: 3 week
Sponsored by: The FreeBSD Foundation

Reliably enable debug exceptions on all CPUs.

Previously, debug exceptions were only enabled on the boot CPU if
DDB was enabled in the dbg_monitor_init() function. APs also called
this function, but since mp_machdep.c doesn't include opt_ddb.h, the
APs ended up calling an empty stub defined in <machine/debug_monitor.h>
instead of the real function. Also, if DDB was not enabled in the kernel,
the boot CPU would not enable debug exceptions.

Fix this by adding a new dbg_init() function that always clears the OS
lock to enable debug exceptions which the boot CPU and the APs call.
This function also calls dbg_monitor_init() to enable hardware breakpoints
from DDB on all CPUs if DDB is enabled. Eventually base support for
hardware breakpoints/watchpoints will need to move out of the DDB-only
debug_monitor.c for use by userland debuggers.

Reviewed by: andrew
Differential Revision: https://reviews.freebsd.org/D12001

Don't panic for PT_GETFPREGS.

Only fetch the VFP state from the CPU if the thread whose registers are
being requested is the current thread. If a stopped thread's registers
are being fetched by a debugger, the saved state in the PCB is already
valid.

Reviewed by: andrew
MFC after: 1 week

Print all virtual addresses in the show vtop ddb command. The results may
be different with PAN enabled.

MFC after: 1 week
Sponsored by: DARPA, AFRL

In ARMv8.1 ARM has added a process state bit to disable access to userspace
from the kernel. Make use of this to restrict accessing userspace to just
the functions that explicitly handle crossing the user kernel boundary.

Reported by: kib
Sponsored by: DARPA, AFRL
Differential Revision: https://reviews.freebsd.org/D10371

Add PCB_FP_USERMASK so we can mask off floating point flags that should
not be sent to userspace, for example the future flag to tell when we are
using floating point in the kernel.

Obtained from: ABT Systems Ltd
Sponsored by: The FreeBSD Foundation

Rework how we store the VFP registers in the pcb. This will be used when
creating a floating-point context within the kernel without having to move
the stored values in memory.

Sponsored by: The FreeBSD Foundation

Consolidate four efi_next_descriptor() definitions.

Sponsored by: The FreeBSD Foundation
MFC after: 1 week

Add a pc_clock pcpu field and use it to implement cpu_est_clockrate. This
will allow drivers that manage the clock frequency to communicate this with
the reset of the kernel.

Reported by: jmcneill
MFC after: 1 week
Sponsored by: ABT Systems Ltd

Print both the kernel read and write translation in DDB when asking for
a virtual to physical translation. These may be different, e.g. when a
page is mapped as read-only.

MFC after: 1 month
Sponsored by: ABT Systems Ltd

Implement userspace gettimeofday(2) with HPET timecounter.

Right now, userspace (fast) gettimeofday(2) on x86 only works for
RDTSC. For older machines, like Core2, where RDTSC is not C2/C3
invariant, and which fall to HPET hardware, this means that the call
has both the penalty of the syscall and of the uncached hw behind the
QPI or PCIe connection to the sought bridge. Nothing can me done
against the access latency, but the syscall overhead can be removed.
System already provides mappable /dev/hpetX devices, which gives
straight access to the HPET registers page.

Add yet another algorithm to the x86 'vdso' timehands. Libc is updated
to handle both RDTSC and HPET. For HPET, the index of the hpet device
to mmap is passed from kernel to userspace, index might be changed and
libc invalidates its mapping as needed.

Remove cpu_fill_vdso_timehands() KPI, instead require that
timecounters which can be used from userspace, to provide
tc_fill_vdso_timehands{,32}() methods. Merge i386 and amd64
libc/<arch>/sys/__vdso_gettc.c into one source file in the new
libc/x86/sys location. __vdso_gettc() internal interface is changed
to move timecounter algorithm detection into the MD code.

Measurements show that RDTSC even with the syscall overhead is faster
than userspace HPET access. But still, userspace HPET is three-four
times faster than syscall HPET on several Core2 and SandyBridge
machines.

Tested by: Howard Su <howard0su@gmail.com>
Sponsored by: The FreeBSD Foundation
MFC after: 1 month
Differential revision: https://reviews.freebsd.org/D7473

Add a kernel variable to let the user to select their preferred order
between ACPI and FDT. This will be needed on machines with both, e.g. the
SoftIron Overdrive 3000. The kernel will accept one or more comma separated
values of either 'acpi' or 'fdt'. Any other values are skipped.

To set it the user can either set it on the loader command line, or
in loader.conf e.g. in loader.conf:
kern.cfg.order=acpi,fdt

This will try using ACPI then FDT. If none of the selected options work the
kernel tries to use one to get the serial console, then panics.

Reviewed by: emaste (earlier version)
Obtained from: ABT Systems Ltd
MFC after: 1 month
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D7274

Add more UEFI/e820 memory types from latest specifications.

This is only cosmetics.

MFC after: 2 weeks

Move arm's devmap to some generic place, so it can be used
by other architectures.

Reviewed by: imp
Differential Revision: https://reviews.freebsd.org/D6091
Sponsored by: DARPA, AFRL
Sponsored by: HEIF5

arm64: pagezero improvement

This change has been provided to improve pagezero call performance.

Submitted by: Dominik Ermel <der@semihalf.com>
Obtained from: Semihalf
Sponsored by: Cavium
Reviewed by: kib
Differential Revision: https://reviews.freebsd.org/D5741

arm64: bzero optimization

This optimization attempts to utylize as wide as possible register store instructions to zero large buffers.
The implementation, if possible, will use 'dc zva' to zero buffer by cache lines.

Speedup: 60x faster memory zeroing

Submitted by: Dominik Ermel <der@semihalf.com>
Obtained from: Semihalf
Sponsored by: Cavium
Reviewed by: kib
Differential Revision: https://reviews.freebsd.org/D5726

Add support for 4 level pagetables. The userland address space has been
increased to 256TiB. The kernel address space can also be increased to be
the same size, but this will be performed in a later change.

To help work with an extra level of page tables two new functions have
been added, one to file the lowest level table entry, and one to find the
block/page level. Both of these find the entry for a given pmap and virtual
address.

This has been tested with a combination of buildworld, stress2 tests, and
by using sort to consume a large amount of memory by sorting /dev/zero. No
new issues are known to be present from this change.

Reviewed by: kib
Obtained from: ABT Systems Ltd
Relnotes: yes
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D5720

Get memory ranges from FDT if no EFI API is available on ARM64

Obtained from: Semihalf
Submitted by: Michal Stanek <mst@semihalf.com>
Sponsored by: Annapurna Labs
Approved by: cognet (mentor)
Reviewed by: andrew, wma
Differential revision: https://reviews.freebsd.org/D5408

Implement single stepping on arm64. We need to set the single step bits in
the processor and debug state registers. A flag has been added to the pcb
to tell us when to enable single stepping for a given thread.

Reviewed by: kib
Sponsored by: ABT Systems Ltd
Differential Revision: https://reviews.freebsd.org/D4730

Convert ss_sp in stack_t and sigstack to void *.

POSIX requires these members to be of type void * rather than the
char * inherited from 4BSD. NetBSD and OpenBSD both changed their
fields to void * back in 1998. No new build failures were reported
via an exp-run.

PR: 206503 (exp-run)
Reviewed by: kib
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D5092

Stop including fdt_common.h in the arm64 code. We don't use anything from
it, however may have relied on header pollution to pull in the needed
headers through it

Sponsored by: ABT Systems Ltd

Make the 'env' directive described in config(5) work on all architectures,
providing compiled-in static environment data that is used instead of any
data passed in from a boot loader.

Previously 'env' worked only on i386 and arm xscale systems, because it
required the MD startup code to examine the global envmode variable and
decide whether to use static_env or an environment obtained from the boot
loader, and set the global kern_envp accordingly. Most startup code wasn't
doing so. Making things even more complex, some mips startup code uses an
alternate scheme that involves calling init_static_kenv() to pass an empty
buffer and its size, then uses a series of kern_setenv() calls to populate
that buffer.

Now all MD startup code calls init_static_kenv(), and that routine provides
a single point where envmode is checked and the decision is made whether to
use the compiled-in static_kenv or the values provided by the MD code.

The routine also continues to serve its original purpose for mips; if a
non-zero buffer size is passed the routine installs the empty buffer ready
to accept kern_setenv() values. Now if the size is zero, the provided buffer
full of existing env data is installed. A NULL pointer can be passed if the
boot loader provides no env data; this allows the static env to be installed
if envmode is set to do so.

Most of the work here is a near-mechanical change to call the init function
instead of directly setting kern_envp. A notable exception is in xen/pv.c;
that code was originally installing a buffer full of preformatted env data
along with its non-zero size (like mips code does), which would have allowed
kern_setenv() calls to wipe out the preformatted data. Now it passes a zero
for the size so that the buffer of data it installs is treated as
non-writeable.

Add support for usermode (vdso-like) gettimeofday(2) and
clock_gettime(2) on ARMv7 and ARMv8 systems which have architectural
generic timer hardware. It is similar how the RDTSC timer is used in
userspace on x86.

Fix a permission problem where generic timer access from EL0 (or
userspace on v7) was not properly initialized on APs.

For ARMv7, mark the stack non-executable. The shared page is added for
all arms (including ARMv8 64bit), and the signal trampoline code is
moved to the page.

Reviewed by: andrew
Discussed with: emaste, mmel
Sponsored by: The FreeBSD Foundation
Differential revision: https://reviews.freebsd.org/D4209

Correctly align the stack. The early csu assumed we passed the aux vector
in through the stack pointer, however this may have been misaligned
causing some userland applications to crash. A workaround was committed in
r284707 where userland would check if the aux vector was passed using the
old or new ABI and adjust the stack if needed. As 4 months have passed it
is time to move to the new ABI, with the expectation the compat code in csu
and the runtime linker to be removed in the future.

Sponsored by: ABT Systems Ltd

Don't read the floating-point registers for now. We will need to enable the
VFP around the read of these instructions as they may raise an exception.

Sponsored by: ABT Systems Ltd

Add ddb show commands to print the special registers and to ask the
hardware to perform address translation for us. These are useful to help
track down what caused us to enter the debugger.

Sponsored by: ABT Systems Ltd

Fill in dump_avail based on the physical memory from EFI.

Obtained from: ABT Systems Ltd
Sponsored by: The FreeBSD Foundation

Pass the pcb to store the vfp state in to vfp_save_state. This fixes a bug
in savectx where it will be used to store the current state however will
pass in a pcb when vfp_save_state expected a thread pointer.

Obtained from: ABT Systems Ltd
Sponsored by: The FreeBSD Foundation

Try to put the CPU into a low power state if we failed to otherwise halt
the system.

Obtained from: ABT Systems Ltd
Sponsored by: The FreeBSD Foundation

Add ARM64TODO markers to unimplemented functionality

Reviewed by: andrew
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D2389

Clear the carry bit on the saved program state register when asked to
clear the return value, it's used to indicate an error.

Obtained from: ABT Systems Ltd
Sponsored by: The FreeBSD Foundation

Add support for SMP. This uses the FDT data to find the CPUs to start on,
and psci to start them. I expect ACPI support to be added later.

This has been tested on qemu with 2 cpus as that is the current value of
MAXCPUS. This is expected to be increased in the future as FreeBSD has
been tested on 48 cores on the Cavium ThunderX hardware.

Partially based on a patch from Robin Randhawa from ARM.

Approved by: ABT Systems Ltd
Relnotes: yes
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D3024

Remove sv_sigtbl handling from the arm64 sendsig(). There is no ABI
emulators on arm64.

Reviewed by: andrew
Review: https://reviews.freebsd.org/D2889
Sponsored by: The FreeBSD Foundation

Add basic support for ACPI. It splits out the nexus driver to two new
drivers, one for fdt, one for acpi. It then uses this to decide if it will
use fdt or acpi.

The GICv2 (interrupt controller) and Generic Timer drivers have been
updated to handle both cases.

As this is early code we still need FDT to find the kernel console, and
some parts are still missing, including PCI support.

Differential Revision: https://reviews.freebsd.org/D2463
Reviewed by: jhb, jkim, emaste
Obtained from: ABT Systems Ltd
Relnotes: Yes
Sponsored by: The FreeBSD Foundation

Bring in the start of the arm64 kernel.

This is only the minimum set of files needed to boot in qemu. As such it is
missing a few things.

The bus_dma code is currently only stub functions with a full implementation
from the development tree to follow.

The gic driver has been copied as the interrupt framework is different. It
is expected the two drivers will be merged by the arm intrng project,
however this will need to be imported into the tree and support for arm64
would need to be added.

This includes code developed by myself, SemiHalf, Ed Maste, and Robin
Randhawa from ARM. This has been funded by the FreeBSD Foundation, with
early development by myself in my spare time with assistance from Robin.

Differential Revision: https://reviews.freebsd.org/D2199
Reviewed by: emaste, imp
Relnotes: yes
Sponsored by: The FreeBSD Foundation