arm64: Flush the spintable release address

Ensure the spintable release address is written back from the cache
to memory. The other CPUs reading this may not be reading it with the
cache enabled so ensure it is written to the point of coherency
before issuing the wakeup sev.

As cpu_dcache_wbinv_range includes the needed barrier remove it from
the inline asm before the sev instruction.

Reviewed by: jhibbits, kevans
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D45083

arm64: Support a shared release for spin-table

When releasing multiple CPUs that share a release address we need them
to wait for their turn to boot. Add a mechanism to do this by booting
them until they enable the TLB before waiting their turn to enter
init_secondary.

Reviewed by: jhibbits, kevans
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D45082

intrng: Extract arm/arm64 IPI->PIC glue code

The arm and arm64 implementations of dispatching IPIs via PIC_IPI_SEND
are almost identical, and entirely MI with the lone exception of a
single store barrier on arm64 (that is likely either redundant or needed
on arm too). Thus, de-duplicate this code by moving it to INTRNG as a
generic IPI glue framework. The ipi_* functions remain declared in MD
smp.h headers and implemented in MD code, but are trivial wrappers
around intr_ipi_send that could be made MI, at least for INTRNG ports,
at a later date.

Note that, whilst both arm and arm64 had an ii_send member in intr_ipi
to abstract over how to send interrupts,, they were always ultimately
using PIC_IPI_SEND, and so this complexity has been removed. A follow-up
commit will re-introduce the same flexibility by instead allowing a
device other than the root PIC to be registered as the IPI sender.

As part of this, strengthen a MAXCPU assertion that was missed in commit
2f0b059eeafc ("intrng: switch from MAXCPU to mp_ncpus") (which itself is
mis-titled).

Reviewed by: mmel, mhorne
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D35898

sys: Automated cleanup of cdefs and other formatting

Apply the following automated changes to try to eliminate
no-longer-needed sys/cdefs.h includes as well as now-empty
blank lines in a row.

Remove /^#if.*\n#endif.*\n#include\s+<sys/cdefs.h>.*\n/
Remove /\n+#include\s+<sys/cdefs.h>.*\n+#if.*\n#endif.*\n+/
Remove /\n+#if.*\n#endif.*\n+/
Remove /^#if.*\n#endif.*\n/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/types.h>/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/param.h>/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/capsicum.h>/

Sponsored by: Netflix

arm64: Initialize x18 for APs earlier during boot

When KMSAN is configured, the instrumentation inserts calls to
__msan_get_context_state() into all function prologues. The
implementation dereferences curthread and thus assumes that x18 points
to the PCPU area. This applies in particular to init_secondary(), which
currently is responsible for initializing x18 for APs.

Move initialization into locore to avoid this problem. No functional
change intended.

Reviewed by: kib, andrew
MFC after: 2 weeks
Sponsored by: Juniper Networks, Inc.
Sponsored by: Klara, Inc.
Differential Revision: https://reviews.freebsd.org/D42533

sys: Remove $FreeBSD$: one-line .c pattern

Remove /^[\s*]*__FBSDID\("\$FreeBSD\$"\);?\s*\n/

arm64: Make dpcpu static

We don't use this directly outside this file so it can be static.

Sponsored by: Arm Ltd

Add helpers to allocate an arm64 VFP state struct

This will be used by bhyve and will allow the size to change, e.g. for SVE.

Reviewed by: markj
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D40131

arm64: Malloc the cpu_desc array

We only need this during boot. Allocate the array before starting CPUs
to reduce the memory usage.

Reviewed by: Zach Leaf <zachary.leaf@arm.com>
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D40433

smp: Dynamically allocate the stoppcbs array

This avoids bloating the kernel image when MAXCPU is large.

A follow-up patch for kgdb and other kernel debuggers is needed since
the stoppcbs symbol is now a pointer. Bump __FreeBSD_version so that
debuggers can use osreldate to figure out how to handle stoppcbs.

PR: 269572
MFC after: never
Reviewed by: mjg, emaste
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D39806

ofw_cpu_early_foreach: Change callback to return bool instead of boolean_t.

Reviewed by: imp
Differential Revision: https://reviews.freebsd.org/D39926

arm: remove passing trapframe to intr_ipi_dispatch()

This was needed before INTRNG was in place and handling the push of
curthread->td_intr_frame. Since INTRNG now handles this, there is no
longer and need for playing around with the frame inside IPI interrupts.

arm: remove interrupt nesting by ipi_preempt()/ipi_hardclock()

This was needed when intr_ipi_dispatch() was called by hardware-specific
IPI interrupt routines which didn't save the trap frame. Now all ARM
interrupts pass through INTRNG which will have already saved the trap
frame and disabled preemption.

Remove the conditional trapframe/argument passing to the handlers.

Reviewed by: andrew
Differential Revision: https://reviews.freebsd.org/D37938

Use a uint64_t to store the arm64 mpidr

Use a single uint64_t to hole the mpidr register as we can break the
KBI on 14. Keep the macro so code can still be MFCd to 13.

Sponsored by: Arm Ltd

arm64: add a spin-table implementation for Apple Silicon

The M1 has no EL3, so we're limited to a spin-table implementation if we
want to eventually use bhyve on it. Implement spin-table now, but note
that we still prefer PSCI where possible.

Reviewed by: mmel
Differential Revision: https://reviews.freebsd.org/D34661

kmem_malloc/free: Use void * instead of vm_offset_t for kernel pointers.

Reviewed by: kib, markj
Sponsored by: DARPA
Differential Revision: https://reviews.freebsd.org/D36549

arm64, riscv: size boot stacks appropriately

In 8db2e8fd16c4 ("Remove the secondary_stacks array in arm64 [...]"),
bootstacks was setup to be allocated dynamically. While this is
generally how x86 does it, it inadvertently shrunk each boot stack from
KSTACK_PAGES pages to a single page.

Resize these back up to the expected size using the kstack_pages
tunable, as we'll need larger stacks with upcoming sanitizer work.

Reviewed by: andrew, imp, markj
Fixes: 8db2e8fd16c4 ("Remove the secondary_stacks array [...]")
Sponsored by: Juniper Networks, Inc.
Sponsored by: Klara, Inc.
Differential Revision: https://reviews.freebsd.org/D36475

Store mpidr as a 64-bit value on arm64

The mpidr register is 64 bit on arm64 and 32 bit on arm. Fix this by
extending the arm64 definition to include the top 32 bits.

To preserve KBI when MFCing split the value into two 32 bit values.
This will be cleaned up later only on main.

Reviewed by: bz
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D36346

Disable promotion on pcpu memory on arm64

We need to be careful to not promote or demote the memory containing
the per-CPU structures as the exception handlers will dereference it
so any time it's invalid may cause recursive exceptions.

Add a new pmap function to set a flag in the pte marking memory that
cannot be promoted or demoted and use it to mark pcpu memory.

Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D35434

Fix the test used to wait for AP startup on x86, arm64, riscv

On arm64, testing pc_curpcb != NULL is not correct since pc_curpcb is
set in pmap_switch() while the bootstrap stack is still in use. As a
result, smp_after_idle_runnable() can free the boot stack prematurely.

Take a different approach: use smp_rendezvous() to wait for all APs to
acknowledge an interrupt. Since APs must not enable interrupts until
they've entered the scheduler, i.e., switched off the boot stack, this
provides the right guarantee without depending as much on the
implementation of cpu_throw(). And, this approach applies to all
platforms, so convert x86 and riscv as well.

Reported by: mmel
Tested by: mmel
Reviewed by: kib
Fixes: 8db2e8fd16c4 ("Remove the secondary_stacks array in arm64 and riscv kernels.")
MFC after: 2 weeks
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D35435

Add arm64 pointer authentication support

Pointer authentication allows userspace to add instructions to insert
a Pointer Authentication Code (PAC) into a register based on an address
and modifier and check if the PAC is correct. If the check fails it will
either return an invalid address or fault to the kernel.

As many of these instructions are a NOP when disabled and in earlier
revisions of the architecture this can be used, for example, to sign
the return address before pushing it to the stack making Return-oriented
programming (ROP) attack more difficult on hardware that supports them.

The kernel manages five 128 bit signing keys: 2 instruction keys, 2 data
keys, and a generic key. The instructions then use one of these when
signing the registers. Instructions that use the first four store the
PAC in the register being signed, however the instructions that use the
generic key store the PAC in a separate register.

Currently all userspace threads share all the keys within a process
with a new set of userspace keys being generated when executing a new
process. This means a forked child will share its keys with its parent
until it calls an appropriate exec system call.

In the kernel we allow the use of one of the instruction keys, the ia
key. This will be used to sign return addresses in function calls.
Unlike userspace each kernel thread has its own randomly generated.

Thread0 has a static key as does the early code on secondary CPUs.
This should be safe as there is minimal user interaction with these
threads, however we could generate random keys when the Armv8.5
Random number generation instructions are present.

Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D31261

Fix set but not used warnings in arm64 core code

sched: split sched_ap_entry() out of sched_throw()

sched_throw() can no longer take a NULL thread, APs enter through
sched_ap_entry() instead. This completely removes branching in the
common case and cleans up both paths. No functional change intended.

Reviewed by: kib, markj
Differential Revision: https://reviews.freebsd.org/D32829

sched: separate out schedinit_ap()

schedinit_ap() sets up an AP for a later call to sched_throw(NULL).

Currently, ULE sets up some pcpu bits and fixes the idlethread lock with
a call to sched_throw(NULL); this results in a window where curthread is
setup in platforms' init_secondary(), but it has the wrong td_lock.
Typical platform AP startup procedure looks something like:

- Setup curthread
- ... other stuff, including cpu_initclocks_ap()
- Signal smp_started
- sched_throw(NULL) to enter the scheduler

cpu_initclocks_ap() may have callouts to process (e.g., nvme) and
attempt to sched_add() for this AP, but this attempt fails because
of the noted violated assumption leading to locking heartburn in
sched_setpreempt().

Interrupts are still disabled until cpu_throw() so we're not really at
risk of being preempted -- just let the scheduler in on it a little
earlier as part of setting up curthread.

Reviewed by: alfredo, kib, markj
Triage help from: andrew, markj
Smoke-tested by: alfredo (ppc), kevans (arm64, x86), mhorne (arm)
Differential Revision: https://reviews.freebsd.org/D32797

Pass the ACPI ID when reading the ACPI domain

The ACPI ID may not be the same as the FreeBSD CPU id. Use the former
when finding the CPU domain as there is no requirement for it to be
identical to the latter.

Reported by: dch, kevans
Reviewed by: kevans
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D32546

Stop reading the arm64 domain when it's known

There is no need to read the domain on arm64 when there is only one
in the ACPI tables. This can also happen when the table is missing
as it is unneeded.

Reported by: dch
Sponsored by: The FreeBSD Foundation

Allocate arm64 per-CPU data in the correct domain

To minimise NUMA traffic allocate the pcpu, dpcpu, and boot stacks in
the correct domain when possible.

Submitted by: markj
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D32338

Read the arm64 midr register earlier

We use the midr_el1 register to decode which CPU type we are booting
from. Read it on the secondary CPUs before waiting for the boot CPU
to release us as it will need to use it before the release.

Sponsored by: The FreeBSD Foundation

Remove "All Rights Reserved" from FreeBSD Foundation sys/ copyrights

These ones were unambiguous cases where the Foundation was the only
listed copyright holder (in the associated license block).

Sponsored by: The FreeBSD Foundation

Enable IPIs on CPU 0 on arm and arm64

Not all interrupt controllers enable IPIs by default as the Arm
GIC specs make it an implementation defined option. As at least two
hypervisors have also previously masked the IPIs on boot.

As we already enable these IPIs on the non-boot CPUs it is expected
this is a safe operation.

Differential Revision: https://reviews.freebsd.org/D26975

arm64: Include NUMA locality info in the CPU topology

The scheduler uses this topology to try and preserve locality when
migrating threads between CPUs and when performing work stealing.
Ensure that on NUMA systems it will at least take the NUMA topology into
account.

Reviewed by: mmel
Submitted by: Klara, Inc.
Sponsored by: Ampere Computing
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D28579

Verify (and fix) the context_id argument passed to the mpentry () by PSCI.

Some older PSCI implementations corrupt (or do not pass) the context_id
argument to newly started secondary cores. Although the ideal solution to this
problem is u-boot update, we can find the correct value for the argument (cpuid)
by comparing of real core mpidr register with the value stored in pcu->mpidr.

MFC after: 2 weeks

Ensure the boot CPU is CPU 0 on arm64

We assume the boot CPU is always CPU 0 on arm64. To allow for this reserve
cpuid 0 for the boot CPU in the ACPI and FDT cases but otherwise start the
CPU as normal. We then check for the boot CPU in start_cpu and return as if
it was started.

While here extract the FDT CPU init code into a new function to simplify
cpu_mp_start and return FALSE from start_cpu when the CPU fails to start.

Reviewed by: mmel
Sponsored by: Innovate UK
Differential Revision: https://reviews.freebsd.org/D27497

Simplify startup of secondary cores and store MPIDR register to pcpu.

- record MPIDR for all started cores in pcpu, they will be used as link
between physical locality of given core, ID in external description
(FDT or ACPI) and cupid.
- because of above, cpuid can (and should) be freely assigned, only boot
CPU must have cpuid 0. Simplify startup code according this.

Please note that pure cpuid is not sufficient instrument to hold any
information about core or cluster topology, nor to determistically iterate
over subpart of cores in CPU (iterate over all cores in single cluster for
example). Situation is more complicated by fact that PSCI can reject start
of core without reporting error (because power budget for example), or by
fact that is possible that we booted on non-first core in cluster (thus with
cpuid 0 assigned to random core).

Given cores topology should be exhibited to other parts of system
(for example to scheduler for big.little or multicluster systems) by using
smp_topo interface.

Differential Revision: https://reviews.freebsd.org/D13863

arm64: Make local stores observable before sending IPIs

Add a synchronizing instruction to flush and wait until the local
CPU's writes are observable to other CPUs before sending IPIs.

This fixes an issue where recipient CPUs doing a rendezvous could
enter the rendezvous handling code before the initiator's writes
to the smp_rv_* variables were visible. This manifested as a
system hang, where a single CPU's increment of smp_rv_waiters[0]
actually happened "before" the initiator's zeroing of that field,
so all CPUs were stuck with the field appearing to be at
ncpus - 1.

Reviewed by: andrew, markj
Approved by: scottl (implicit)
MFC after: 1 week
Sponsored by: Ampere Computing, Inc.
Differential Revision: https://reviews.freebsd.org/D25798

Read the CPU 0 arm64 ID registers early in initarm

We also update the kernel view early in the boot. This will allow the
use of the common kernel view in ifunc resolvers.

Sponsored by: Innovate UK

Read the arm64 ID registers earlier in the boot process.

Also move parsing the registers to just after the secondary CPUs have
started. This means the kernel register view from all CPUs is available
after the CPU SYSINITs have finished, e.g. for use by ifunc resolvers.

Sponsored by: Innovate UK
Differential Revision: https://reviews.freebsd.org/D25505

Call acpi_pxm_set_proximity_info() slightly earlier on x86.

This function is responsible for setting pc_domain in each pcpu
structure. Call it from the main function that starts APs, rather than
a separate SYSINIT. This makes it easier to close the window where
UMA's per-CPU slab allocator may be called while pc_domain is
uninitialized. In particular, the allocator uses pc_domain to allocate
domain-local pages, so allocations before this point end up using domain
0 for everything.

Reviewed by: kib
MFC after: 1 week
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D24757

Remove the secondary_stacks array in arm64 and riscv kernels.

Instead, dynamically allocate a page for the boot stack of each AP when
starting them up, like we do on x86. This shrinks the bss by
MAXCPU*KSTACK_PAGES pages, which corresponds to 4MB on arm64 and 256KB
on riscv.

Duplicate the logic used on x86 to free the bootstacks, by using a
sysinit to wait for each AP to switch to a thread before freeing its
stack.

While here, mark some static MD variables as such.

Reviewed by: kib
MFC after: 1 month
Sponsored by: Juniper Networks, Klara Inc.
Differential Revision: https://reviews.freebsd.org/D24158

arm64: Don't enable interrupts in init_secondary().

Doing so can cause deadlocks or panics during boot, if an interrupt
handler accesses uninitialized per-CPU scheduler structures. This seems
to occur frequently when running under QEMU or AWS. The idle threads
are set up to release a spinlock section and enable interrupts in
fork_exit(), so there is no need to enable interrupts earlier.

Reviewed by: kib
MFC after: 1 week
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D23328

Port the NetBSD KCSAN runtime to FreeBSD.

Update the NetBSD Kernel Concurrency Sanitizer (KCSAN) runtime to work in
the FreeBSD kernel. It is a useful tool for finding data races between
threads executing on different CPUs.

This can be enabled by enabling KCSAN in the kernel config, or by using the
GENERIC-KCSAN amd64 kernel. It works on amd64 and arm64, however the later
needs a compiler change to allow -fsanitize=thread that KCSAN uses.

Sponsored by: DARPA, AFRL
Differential Revision: https://reviews.freebsd.org/D22315

Utilize ASIDs to reduce both the direct and indirect costs of context
switching. The indirect costs being unnecessary TLB misses that are
incurred when ASIDs are not used. In fact, currently, when we perform a
context switch on one processor, we issue a broadcast TLB invalidation that
flushes the TLB contents on every processor.

Mark all user-space ("ttbr0") page table entries with the non-global flag so
that they are cached in the TLB under their ASID.

Correct an error in pmap_pinit0(). The pointer to the root of the page
table was being initialized to the root of the kernel-space page table
rather than a user-space page table. However, the root of the page table
that was being cached in process 0's md_l0addr field correctly pointed to a
user-space page table. As long as ASIDs weren't being used, this was
harmless, except that it led to some unnecessary page table switches in
pmap_switch(). Specifically, other kernel processes besides process 0 would
have their md_l0addr field set to the root of the kernel-space page table,
and so pmap_switch() would actually change page tables when switching
between process 0 and other kernel processes.

Implement a workaround for Cavium erratum 27456 affecting ThunderX machines.
(I would like to thank andrew@ for providing the code to detect the affected
machines.)

Address integer overflow in the definition of TCR_ASID_16.

Setup TCR according to the PARange and ASIDBits fields from
ID_AA64MMFR0_EL1. Previously, TCR_ASID_16 was unconditionally set.

Modify build_l1_block_pagetable so that lower attributes, such as ATTR_nG,
can be specified as a parameter.

Eliminate some unused code.

Earlier versions were tested to varying degrees by: andrew, emaste, markj

MFC after: 3 weeks
Differential Revision: https://reviews.freebsd.org/D21922

Update the debug monitor handling to work after userspace has started

The debug monitor register state is now stored in a struct and updated
when required. Currently there is only a kernel state, however a
per-process state will be added in a future change.

Sponsored by: DARPA, AFRL
Differential Revision: https://reviews.freebsd.org/D22128

Centralize __pcpu definitions.

Many extern struct pcpu <something>__pcpu declarations were
copied/pasted in sources. The issue is that the definition is MD, but
it cannot be provided by machine/pcpu.h due to actual struct pcpu
defined in sys/pcpu.h later than the inclusion of machine/pcpu.h.
This forced the copying when other code needed direct access to
__pcpu. There is no way around it, due to machine/pcpu.h supplying
part of struct pcpu fields.

To work around the problem, add a new machine/pcpu_aux.h header, which
should fill any needed MD definitions after struct pcpu definition is
completed. This allows to remove copies of __pcpu spread around the
source. Also on x86 it makes it possible to remove work arounds like
OFFSETOF_CURTHREAD or clang specific warnings supressions.

Reported and tested by: lwhsu, bcran
Reviewed by: imp, markj (previous version)
Discussed with: jhb
Sponsored by: The FreeBSD Foundation
Differential revision: https://reviews.freebsd.org/D21418

Include ktr.h in more compilation units

Similar to r348026, exhaustive search for uses of CTRn() and cross reference
ktr.h includes. Where it was obvious that an OS compat header of some kind
included ktr.h indirectly, .c files were left alone. Some of these files
clearly got ktr.h via header pollution in some scenarios, or tinderbox would
not be passing prior to this revision, but go ahead and explicitly include it
in files using it anyway.

Like r348026, these CUs did not show up in tinderbox as missing the include.

Reported by: peterj (arm64/mp_machdep.c)
X-MFC-With: r347984
Sponsored by: Dell EMC Isilon

Add the hw.ncpu tunable to arm64.

This allows us to limit the number of CPUs to use, e.g. to debug problems
seen when enabling multiple clusters.

Reviewed by: manu
Sponsored by: DARPA, AFRL
Differential Revision: https://reviews.freebsd.org/D19404

arm64: add ACPI based NUMA support

Use the newly defined SRAT/SLIT parsing APIs in arm64 to support
ACPI based NUMA.

Reviewed by: markj
Differential Revision: https://reviews.freebsd.org/D17943

Always set the MP_QUIRK_CPULIST quirk under ACPI. This needs a run time
check to only set it for emulators as the CPU list may be changed when
the emulator starts. Until this is working just always set it.

Sponsored by: DARPA, AFRL

Remove function prototypes for functions removed in r339943.

Sponsored by: DARPA, AFRL

Remove the unused arm64_cpu driver.

This was previously used for CPU initilisation, however this hasn't been
the case in a long time.

Sponsored by: DARPA, AFRL

Make it possible to disable NUMA support with a tunable.

This provides a chicken switch for anyone negatively impacted by
enabling NUMA in the amd64 GENERIC kernel configuration. With
NUMA disabled at boot-time, information about the NUMA topology
is not exposed to the rest of the kernel, and all of physical
memory is viewed as coming from a single domain.

This method still has some performance overhead relative to disabling
NUMA support at compile time.

PR: 231460
Reviewed by: alc, gallatin, kib
MFC after: 1 week
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D17439

Eliminate the arena parameter to kmem_free(). Implicitly this corrects an
error in the function hypercall_memfree(), where the wrong arena was being
passed to kmem_free().

Introduce a per-page flag, VPO_KMEM_EXEC, to mark physical pages that are
mapped in kmem with execute permissions. Use this flag to determine which
arena the kmem virtual addresses are returned to.

Eliminate UMA_SLAB_KRWX. The introduction of VPO_KMEM_EXEC makes it
redundant.

Update the nearby comment for UMA_SLAB_KERNEL.

Reviewed by: kib, markj
Discussed with: jeff
Approved by: re (marius)
Differential Revision: https://reviews.freebsd.org/D16845

Eliminate kmem_malloc()'s unused arena parameter. (The arena parameter
became unused in FreeBSD 12.x as a side-effect of the NUMA-related
changes.)

Reviewed by: kib, markj
Discussed with: jeff, re@
Differential Revision: https://reviews.freebsd.org/D16825

Allow releasing APs to take more time, as long as we are making progress.
On large core count machines this can be slow while all the CPUs update
the online counter.

Sponsored by: DARPA, AFRL
Sponsored by: Cavium (Hardware)

Add a framework to install CPU errata on arm64. Each erratum can encode
a mask and value to compare with the Main ID Register. If these match then a
function is called to handle the installation of the erratum workaround.

No errata are currently handled, however this will change soon in a future
commit.

MFC after: 1 week
Sponsored by: DARPA, AFRL

Add a memory barrier to ensure the atomic write is visible to the other
CPUs before waking them up.

Sponsored by: DARPA, AFRL

Add support for quirks while enabling secondary CPUs. This uses the fdt
compatible string to check if the board is compatible with a given quirk.
It's possible this will be moved later, however as it's currently only used
by the MP code put it there.

So far the only instance of a quirk is when the list of CPUs may be
incorrect. This can happen on virtual machines with a hard coded
devicetree, but where the user may then set the number of CPUs as an
argument. This is the case on the ARM models so include the model specific
compat strings for these, including the spelling mistake found in some of
the OpenplatformPkg dtb files.

Sponsored by: DARPA, AFRL

Reliably enable debug exceptions on all CPUs.

Previously, debug exceptions were only enabled on the boot CPU if
DDB was enabled in the dbg_monitor_init() function. APs also called
this function, but since mp_machdep.c doesn't include opt_ddb.h, the
APs ended up calling an empty stub defined in <machine/debug_monitor.h>
instead of the real function. Also, if DDB was not enabled in the kernel,
the boot CPU would not enable debug exceptions.

Fix this by adding a new dbg_init() function that always clears the OS
lock to enable debug exceptions which the boot CPU and the APs call.
This function also calls dbg_monitor_init() to enable hardware breakpoints
from DDB on all CPUs if DDB is enabled. Eventually base support for
hardware breakpoints/watchpoints will need to move out of the DDB-only
debug_monitor.c for use by userland debuggers.

Reviewed by: andrew
Differential Revision: https://reviews.freebsd.org/D12001

Mark each cpu in the appropriate cpuset_domain set. This allows devices to
handle cases where they can only run on a single domain.

To allow all devices access to this set we need to move reading the domain
earlier in the boot as it was previously handled in the CPU driver, however
this is too late for the GICv3 ITS driver.

Sponsored by: DARPA, AFRL

Read the numa-node-id property from each CPU node. This will initially be
used to support the dual package ThunderX where we need to send MSI/MSI-X
interrupts to the same package as the device the interrupt came from.

Sponsored by: DARPA, AFRL

In ARMv8.1 ARM has added a process state bit to disable access to userspace
from the kernel. Make use of this to restrict accessing userspace to just
the functions that explicitly handle crossing the user kernel boundary.

Reported by: kib
Sponsored by: DARPA, AFRL
Differential Revision: https://reviews.freebsd.org/D10371

Remove a static function declaration for a function not implemented.

Makes head code compile on 10.3 and cleanup is never wrong.

MFC after: 3 days

Add ACPI support to the arm64 mp code. We use the Multiple APIC Description
Table to find the CPUs to find the CPUs to start. Currently we assume PSCI,
however this assumption is shared with the FDT code.

Obtained from: ABT Systems Ltd
Sponsored by: The FreeBSD Foundation

We only use the cpu0 variable in the FDT code.

Obtained from: ABT Systems Ltd
Sponsored by: The FreeBSD Foundation

Mark cpu_find_cpu0_fdt as FDT only. It's only called when this is set, and
the kernel is using FDT.

Obtained from: ABT Systems Ltd
Sponsored by: The FreeBSD Foundation

Remove the unneeded ofw_cpu_reg function signature, it's not used in this
file.

Obtained from: ABT Systems Ltd
Sponsored by: The FreeBSD Foundation

Only release CPUs when they exist.

MFC after: 1 week
Sponsored by: DARPA, AFRL

Create a new PSCI error code and use it to signal that starting the CPU is
impossible as the PSCI firmware is missing.

Sponsored by: ABT Systmes Ltd

Move printing the AArch64 ID registers to a new SYSINIT, the previous
location only prints them when booting on SMP with multiple cores.

MFC after: 1 week
Sponsored by: DARPA, AFRL

Add a kernel variable to let the user to select their preferred order
between ACPI and FDT. This will be needed on machines with both, e.g. the
SoftIron Overdrive 3000. The kernel will accept one or more comma separated
values of either 'acpi' or 'fdt'. Any other values are skipped.

To set it the user can either set it on the loader command line, or
in loader.conf e.g. in loader.conf:
kern.cfg.order=acpi,fdt

This will try using ACPI then FDT. If none of the selected options work the
kernel tries to use one to get the serial console, then panics.

Reviewed by: emaste (earlier version)
Obtained from: ABT Systems Ltd
MFC after: 1 month
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D7274

Remove an unused variable.

Obtained from: ABT Systems Ltd
MFC after: 1 month
Sponsored by: The FreeBSD Foundation

Fix a typo in a string in a KASSERT to sanity check the CPU IDs.

Obtained from: ABT Systems Ltd
MFC after: 1 month
Sponsored by: The FreeBSD Foundation

Rework how we number CPUs on arm64 to try and keep clusters together.

Obtained from: ABT Systems Ltd
MFC after: 1 month
Sponsored by: The FreeBSD Foundation

Finish removing the non-INTRNG support from sys/arm64.

Obtained from: ABT Systems Ltd
MFC after: 1 month
Sponsored by: The FreeBSD Foundation

dpcpu_init should have also passed in the calculated cpuid, not the
devicetree ID.

Obtained from: ABT Systems Ltd
Sponsored by: The FreeBSD Foundation

Allow the kernel to boot on a CPU where the devicetree has numbered it with
a non-zero ID. To do this we increment the cpuid of any CPUs with a smaller
devicetree ID by one to stop them conflicting with the boot CPU.

Obtained from: ABT Systems Ltd
Sponsored by: The FreeBSD Foundation

Move the call to intr_pic_init_secondary to the same place as in the
non-intrng case.

Obtained from: ABT Systems Ltd
Sponsored by: The FreeBSD Foundation

Add support for intrng to arm64. As the GICv3 drivers will need to be
updated, and until further testing can be done, this is disabled for now.

It is expected arm64 will switch to this interface, and the old interface
will be removed before 11.0 is released.

Obtained from: ABT Systems Ltd
Relnotes: yes
Sponsored by: The FreeBSD Foundation

Reduce the diff for when we switch to intrng. The IPI interrupts will be
split out to multiple handlers.

Obtained from: ABT Systems Ltd
Sponsored by: The FreeBSD Foundation

Rename COUNT_IPI to INTR_IPI_COUNT to reduce the diff with intrng.

Obtained from: ABT Systems Ltd
Sponsored by: The FreeBSD Foundation

Implement single stepping on arm64. We need to set the single step bits in
the processor and debug state registers. A flag has been added to the pcb
to tell us when to enable single stepping for a given thread.

Reviewed by: kib
Sponsored by: ABT Systems Ltd
Differential Revision: https://reviews.freebsd.org/D4730

Decode and print the ID_AA64* registers on boot. These registers hold
information on what the core supports. In most cases these will be
identical across most CPUs in the SoC, however there may be the case where,
with a big.LITTLE setup they may differ. In this case we print the
decoded data on all CPUs.

Reviewed by: kib
Sponsored by: ABT Systems Ltd
Differential Revision: https://reviews.freebsd.org/D4725

Make the arm64_cpu driver quiet as it adds no new information.
Only report the register used to start each cpu in bootverbose.

Sponsored by: ABT Systems Ltd

Remove apostrophe from AP's for consistency (arm64 mp_machdep)

Release memory for CPUs that fail to init on ARM64

cpu_init_fdt will now release memory allocated for structures
serving CPUs that have failed to init.

Obtained from: Semihalf
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D3297

Remove redundant mp_naps from ARM64 secondary CPU start-up code

The global variable has been only used for CPU startup ordering
which is not needed anyway.

Obtained from: Semihalf
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D3296

Fix KSTACK_PAGES issue when the default value was changed in KERNCONF

If KSTACK_PAGES was changed to anything alse than the default,
the value from param.h was taken instead in some places and
the value from KENRCONF in some others. This resulted in
inconsistency which caused corruption in SMP envorinment.

Ensure all places where KSTACK_PAGES are used the opt_kstack_pages.h
is included.

The file opt_kstack_pages.h could not be included in param.h
because was breaking the toolchain compilation.

Reviewed by: kib
Obtained from: Semihalf
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D3094

Add identify_cpu() to ARM64 init_secondary routine

Identify current CPU. This is necessary to setup
affinity registers and to provide support for
runtime chip identification.

Reviewed by: andrew
Obtained from: Semihalf
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D3095

Fix secondary PIC initialization order

Call arm_init_secondary before any other PIC-related functions
are called. This is necessary for GICv3 where PIC_INIT_SECONDARY
allocates resources needed for all further operations.

Obtained from: Semihalf
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D3066

Add support for SMP. This uses the FDT data to find the CPUs to start on,
and psci to start them. I expect ACPI support to be added later.

This has been tested on qemu with 2 cpus as that is the current value of
MAXCPUS. This is expected to be increased in the future as FreeBSD has
been tested on 48 cores on the Cavium ThunderX hardware.

Partially based on a patch from Robin Randhawa from ARM.

Approved by: ABT Systems Ltd
Relnotes: yes
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D3024