arm64: Use the _REG macros to read ID registers

To build with old toolchains use the *_REG macros to access the ID
registers. These become a name in the form S?_?_C?_C?_? where the '?'
values encode the op and CR values needed to access the register.

For consistency use these macros for all ID registers, even if most
toolchains understand them.

Reviewed by: Zachary Leaf <zachary.leaf@arm.com>
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D45177

arm64: add CPU part identifiers for Apple M1 and M2

Part numbers are sourced from Linux (arch/arm64/include/asm/cputype.h).

MFC after: 2 weeks
Pull Request: https://github.com/freebsd/freebsd-src/pull/1144

arm64: Switch the address argument to cpu_*cache* to a pointer

No functional change, but this reduces diffs with CheriBSD downstream.

Reviewed by: andrew
Sponsored by: University of Cambridge, Google, Inc.
Differential Revision: https://reviews.freebsd.org/D44342

arm64: Tell userspace when BTI is supported

To allow userspace to decide when to try enabling BTI report if it's
available via the ID register and hwcaps.

Sponsored by: Arm Ltd

identcpu: Let the hw.model be readable in capability mode

On some platforms this static information can be derived directly from
the CPU, and there doesn't seem much use in restricting it.

Reviewed by: emaste, imp, kib
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D43281

sys: Automated cleanup of cdefs and other formatting

Apply the following automated changes to try to eliminate
no-longer-needed sys/cdefs.h includes as well as now-empty
blank lines in a row.

Remove /^#if.*\n#endif.*\n#include\s+<sys/cdefs.h>.*\n/
Remove /\n+#include\s+<sys/cdefs.h>.*\n+#if.*\n#endif.*\n+/
Remove /\n+#if.*\n#endif.*\n+/
Remove /^#if.*\n#endif.*\n/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/types.h>/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/param.h>/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/capsicum.h>/

Sponsored by: Netflix

arm64: Create a Linux view of the ID registers

When adding support for new hardware extensions we may not want to
enable support for the FreeBSD and Linux ABIs at the same time. To
support this split the Linux ID register and hwcaps so they can be
configured separately.

Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D42372

arm64: Use a hwcap ID rather than pointer

To allow for a different Linux hwcap value store the hwcap ID rather
than a pointer to elf{32,}_hwcap{2,}. This will be needed when creating
a different view of the ID registers for FreeBSD and Linux.

Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D42371

arm64: Add an is64bit flag to user readable ID regs

To tell if a register is for 32 or 64 bit userspace add a flag to the
user reg struct. This will be used later to rework how the hwcaps are
generated.

Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D42370

arm64: Add a missing MRS_HWCAP_END

Add a missing end marker to stop the arm64 CPU ID code reading past
the end of an array.

Sponsored by: Arm Ltd

arm64: Export HWCAP_CPUID

We have export the CPUID registers since 2018 but never set the
HWCAP_CPUID flag to tell userspace it could use them. Fix this by
setting it.

Sponsored by: Arm Ltd

arm64: Enable FEAT_E0PD when supported

FEAT_E0PD adds two fields to the tcr_el1 special register that, when
set, cause userspace access to either the top or bottom half of the
address spaces without a page walk.

This can be used to stop userspace probing the kernel address space
as the CPU will raise an exception in the same time if the probed
address is in the TLB or not.

Reviewed by: kevans
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D41760

sys: Remove $FreeBSD$: one-line .c pattern

Remove /^[\s*]*__FBSDID\("\$FreeBSD\$"\);?\s*\n/

arm64: Comment out some ID registers

Older toolchains don't support reading these registers. Until a fix is
added comment them out.

Reported by: jhb
Sponsored by: Arm Ltd

arm64: Decode the ID_AA64PFR2_EL1 register

No fields have been defined, but it has been documented in the
Architecture Reference Manual.

Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D40897

arm64: Update the ID_AA64PFR1_EL1 fields

While here move to decimal for the _op and _CR definitions to be used
by a future macro to define the register when the assembler doesn't
know about it.

Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D40896

arm64: Update the ID_AA64PFR0_EL1 fields

While here move to decimal for the _op and _CR definitions to be used
by a future macro to define the register when the assembler doesn't
know about it.

Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D40895

arm64: Decode the ID_AA64MMFR4_EL1 register

No fields have been defined, but it has been documented in the
Architecture Reference Manual.

Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D40894

arm64: Decode the ID_AA64MMFR3_EL1 register

Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D40893

arm64: Update the ID_AA64MMFR1_EL1 fields

Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D40891

arm64: Update the ID_AA64MMFR0_EL1 fields

While here move to decimal for the _op and _CR definitions to be used
by a future macro to define the register when the assembler doesn't
know about it.

Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D40890

arm64: Update the ID_AA64ISAR1_EL1 fields

While here move to decimal for the _op and _CR definitions to be used
by a future macro to define the register when the assembler doesn't
know about it.

Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D40889

arm64: Update the ID_AA64ISAR0_EL1 fields

While here move to decimal for the _op and _CR definitions to be used
by a future macro to define the register when the assembler doesn't
know about it.

Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D40888

arm64: Update the ID_AA64DFR0_EL1 fields

While here move to decimal for the _op and _CR definitions to be used
by a future macro to define the register when the assembler doesn't
know about it.

Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D40887

arm64: Fix a use of update_lower_register

In get_kernel_reg_masked we use update_lower_register to get the lower
value of two registers for a given field. It will return the entire
register value with just the single field updated.

Because of this get_kernel_reg_masked needs to use the returned value
directly rather than ORing each field together. Fix this by updating
the mask and returning that from get_kernel_reg_masked.

Reviewed by: markj
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D40846

arm64: Make register definitions const

No functional change intended.

Reviewed by: andrew
MFC after: 2 weeks
Sponsored by: The FreeBSD Foundation
Sponsored by: Klara, Inc. (hardware)
Differential Revision: https://reviews.freebsd.org/D40502

arm64: Add a masked get_kernel_reg()

This lets consumers fetch the value of a system register and apply a
mask over individual fields. That is, each field in the returned value
will be the "smaller" of the two provided by "mask" and the value saved
in kern_cpu_desc. This will be used by vmm to sanitize host system
register fields.

Reviewed by: andrew
MFC after: 2 weeks
Sponsored by: The FreeBSD Foundation
Sponsored by: Klara, Inc. (hardware)
Differential Revision: https://reviews.freebsd.org/D40500

Add more arm64 ID registers to the user_regs array

This is a mapping from ID register value to offset in struct cpu_desc.
These registers may be needed with future architecture revisions either
by userspace or by bhyve.

Sponsored by: Arm Ltd

arm64: Malloc the cpu_desc array

We only need this during boot. Allocate the array before starting CPUs
to reduce the memory usage.

Reviewed by: Zach Leaf <zachary.leaf@arm.com>
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D40433

arm64: Reduce the direct use of cpu_desc

To help moving to a dynamically allocated cpu_desc array reduce the
places we use it directly and create a pointer that is passed in to
functions that read it.

Reviewed by: Zach Leaf <zachary.leaf@arm.com>
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D40392

Allow sysctl hw.machine/hw.machine_arch in capability mode

There's no harm in reading strings like 'amd64'.

Reviewed by: emaste, manu
Sponsored by: https://www.patreon.com/valpackett
Differential Revision: https://reviews.freebsd.org/D28703

Add CPU Ident for Qualcomm Kryo 400 (used in MS Dev Kit)

Reviewed by: imp
Sponsored by: Klara, Inc.
Differential Revision: https://reviews.freebsd.org/D37767

Add more Arm CPUs to the arm64 cpu ident

These are based on CPUs found in https://github.com/ARM-software/data

Sponsored by: The FreeBSD Foundation

Reduce the arm64 ID registers we print

On systems with different CPUs we may print all the ID registers for
all CPUs. Reduce this to just print them when they change from the
previous CPU.

Sponsored by: The FreeBSD Foundation

Decode the arm64 ID_AA64ISAR1_EL1 register

Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D35627

Add an IDC only arm64 icache sync function

When the IDC flag is set in the cache type register we don't need to
clean the data cache to the point of unification. Previously we
supported this flag being set only when the DIC flags was also set.
Add a new handler for when this is not the case.

Reviewed by: kib
Sponsored by: The FreeBSD Foundation, Ampere (hardware)
Differential Revision: https://reviews.freebsd.org/D36296

Decode the arm64 SVE ID register

The field values are only valid when the ID_AA64PFR0_EL1.SVE or
ID_AA64PFR1_EL1.SME vields are non-zero. When this is not the case
the register is reserved as zero so is safe to read, but the SVEver
field will be incorrect so only print the decoded register when
the SVE or SME fields indicate it is valid.

Sponsored by: The FreeBSD Foundation

arm64: Print per-CPU cache summary

Summary:
It can be useful to see a summary of CPU caches on bootup. This is done
for most platforms already, so add this to arm64, in the form of (taken
from Apple M1 pro test):

L1 cache: 192KB (instruction), 128KB (data)
L2 cache: 12288KB (unified)

This is printed out per-CPU, only under bootverbose.

Future refinements could instead determine if a cache level is shared
with other cores (L2 is shared among cores on some SoCs, for instance),
and perform a better calculation to the full true cache sizes. For
instance, it's known that the M1 pro, on which this test was done, has 2
12MB L2 clusters, for a total of 24MB. Seeing each CPU with 12288KB L2
would make one think that there's 12MB * NCPUs, for possibly 120MB
cache, which is incorrect.

Sponsored by: Juniper Networks, Inc.
Reviewed by: #arm64, andrew
Differential Revision: https://reviews.freebsd.org/D35366

Decode all Arm GIC feature ID bits

The AWS Graviton3 CPU features a GIC 4.1 CPU Interface. Teach the CPU
identift code to decode it.

Sponsored by: The FreeBSD Foundation

Add more Arm CPU IDs

Add more CPU main ID register values for Arm Cortex and Neoverse CPUs

Sponsored by: The FreeBSD Foundation

Add the Ampere and Fujitsu arm64 implementer IDs

Reviewed By: andrew, emaste
MFC after: 3 days
Sponsored by: Ampere Computing
Differential Revision: https://reviews.freebsd.org/D34666

Add more arm64 PAC identification fields

Sponsored by: The FreeBSD Foundation

Add arm64 pointer authentication support

Pointer authentication allows userspace to add instructions to insert
a Pointer Authentication Code (PAC) into a register based on an address
and modifier and check if the PAC is correct. If the check fails it will
either return an invalid address or fault to the kernel.

As many of these instructions are a NOP when disabled and in earlier
revisions of the architecture this can be used, for example, to sign
the return address before pushing it to the stack making Return-oriented
programming (ROP) attack more difficult on hardware that supports them.

The kernel manages five 128 bit signing keys: 2 instruction keys, 2 data
keys, and a generic key. The instructions then use one of these when
signing the registers. Instructions that use the first four store the
PAC in the register being signed, however the instructions that use the
generic key store the PAC in a separate register.

Currently all userspace threads share all the keys within a process
with a new set of userspace keys being generated when executing a new
process. This means a forked child will share its keys with its parent
until it calls an appropriate exec system call.

In the kernel we allow the use of one of the instruction keys, the ia
key. This will be used to sign return addresses in function calls.
Unlike userspace each kernel thread has its own randomly generated.

Thread0 has a static key as does the early code on secondary CPUs.
This should be safe as there is minimal user interaction with these
threads, however we could generate random keys when the Armv8.5
Random number generation instructions are present.

Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D31261

Move the arm64 identify_cpu SYSINIT earlier

It is used by late ifunc resolvers so needs to be at an earlier stage
of the boot. Previously it was at the same stage so may not have run
before the ifunc resolvers.

Sponsored by: The FreeBSD Foundation

Add the Apple arm64 implementer ID

Sponsored by: The FreeBSD Foundation

Sort the arm64 cpu_implementers list by name

We perform a linear search, so make it easier to add new entries in the
correct place.

Sponsored by: The FreeBSD Foundation

Only store the arm64 ID registers in the cpu_desc

There is no need to store a pointer to the CPU implementer and part
strings. Switch to load them directly into the sbuf used to print them
on boot.

While here print the machine ID register when we fail to determine the
implementer or part we are booting on.

Reviewed by: markj, kib
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D31346

Move setting arm64 HWCAP values to the ID tables

The HWCAPS values are based on the ID registers. Move setting these
to the existing ID register parsing code.

Previously we would need to handle all possible ID field values where
a HWCAP is set, however as most ID fields follow a scheme where when
the field increments it will only add new features meaning we only
need to check if the field is greater than when the HWCAP feature
was added.

While here stop setting HWCAP value that need kernel support, but this
support is missing.

Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D31201

Add missing arm64 ID registers

These may contain values we export to userpsace.

Sponsored by: The FreeBSD Foundation

Sort the arm64 ID_AA64* user registers

Sponsored by: The FreeBSD Foundation

Add macros for arm64 special reg op and CR values

Use these to simplify the definition of the user_regs array.

Reviewed by: imp, markj
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D31333

Teach the arm64 kernel to identify the Arm AEM

The Arm Architecture Envelope Model is a simulator that models the
architecture rather than any specific implementation. Add its part ID
macro and add it to the list of Arm CPUs we can decode.

Sponsored by: The FreeBSD Foundation

arm64: HWCAP/HWCAP2 aux args support for 32-bit ARM binaries.

This fixes build/run of golang under COMPAT32 emulation.

PR: 256897
Reviewed by: andrew, mmel, manu, jhb, cognet, Robert Clausecker
Tested by: brd, andrew, Robert Clausecker
MFC after: 3 weeks
Relnotes: yes
Sponsored by: Rubicon Communications, LLC ("Netgate")
Differential Revision: https://reviews.freebsd.org/D31175

arm64: Print CPU features slightly earlier

In particular, print them before we release APs. Otherwise they tend to
get mixed with other kernel messages.

Reviewed by: andrew, manu
MFC after: 1 week
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D31242

Split out the arm64 ID field comparison function

This will be useful in an update for finding which HWCAPS to set.

Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D31200

Hide arm64 features that don't have a HWCAP

We should only export MSR fields if there is also a HWCAP so it doesn't
matter which software uses.

Sponsored by: The FreeBSD Foundation

Sync the arm64 special registers with the Armv8.5 XML

Add the missing macros and decode all the fields as described in the
Arm Architecture System Registers XML corresponding to Armv8.5.

Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D30983

Switch the order of the ID_AA64PFR1_EL1 fields

This makes them consistent with the fields in other registers.

Sponsored by: The FreeBSD Foundation

arm64: extend struct db_reg to include watchpoint registers

The motivation is to provide access to these registers from userspace
via ptrace(2) requests PT_GETDBREGS and PT_SETDBREGS.

This change breaks the ABI of these particular requests, but is
justified by the fact that the intended consumers (debuggers) have not
been taught to use them yet. Making this change now enables active
upstream work on lldb to begin using this interface, and take advantage
of the hardware debugging registers available on the platform.

PR: 252860
Reported by: Michał Górny (mgorny@gentoo.org)
Reviewed by: andrew, markj (earlier version)
Tested by: Michał Górny (mgorny@gentoo.org)
MFC after: 1 week
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D28415

arm64: export a few more HWCAPs

These were missed in the previous pass. The extensions (partially)
supported by this change are:
- ARMv8.2-FHM, Floating-point multiplication variant
- ARMv8.4-LSE, Large System Extensions
- ARMv8.4-DIT, Data Independent Timing instructions

Reviewed by: andrew, markj
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D26707

Update the ID_AA64MMFR2_EL1 register definitions

This brings these definitions in sync with the ARMv8.6 version of the
architecture reference manual.

Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D26706

arm64: set the correct HWCAP

This appears to be a typo. The AdvSIMD field encodes support for
half-precision floating point SIMD instructions, which corresponds to
HWCAP_ASIMDHP, not HWCAP_ASIMDDP.

MFC after: 3 days
Sponsored by: The FreeBSD Foundation

arm64: export new HWCAP features

Expose some of the new HWCAP features added in r65304. This includes the
addition of elf_hwcap2 into the sysvec, and a separate function to parse
for those features.

This only exposes features which require no further configuration, e.g.
indicating the presence of certain instructions. Larger features (SVE)
will not be advertised until we actually support them. The exact list of
features/extensions this patch exposes is:
- ARMv8.0-DGH
- ARMv8.0-SB
- ARMv8.2-BF16
- ARMv8.2-DCCVADP
- ARMv8.2-I8MM
- ARMv8.4-LRCPC
- ARMv8.5-CondM
- ARMv8.5-FRINT
- ARMv8.5-RNG
- PSTATE.SSBS

While here, annotate elf_hwcap and elf_hwcap2 as __read_frequently, and
move the declarations to the machine/md_var.h header.

Submitted by: mikael@ (D22314 portion)
MFC after: 2 weeks
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D26031
Differential Revision: https://reviews.freebsd.org/D22314

Partially revert r365069.

This whitespace was intentionally added to help differentiate the different
register groups within this file.

While here add missing whitespace from earlier in the file,

Sponsored by: Innovate UK

arm64: clean up empty lines in .c and .h files

arm64/acpi: Give the real PA limit to ACPI

Read PA bits from ID_AA64MMFR0_EL1.PARange.

Reviewed by: andrew, markj
Approved by: scottl (implicit)
MFC after: 1 week
Sponsored by: Ampere Computing, Inc.
Differential Revision: https://reviews.freebsd.org/D26133

arm64: parse HWCAP values using user_cpu_desc

The hard work of parsing fields per-CPU, handling heterogeneous
features, and excluding features from userspace is already done by
update_special_regs. We can build our set of HWCAPs from the result.

This exposed a small bug in update_special_regs, in which the
generated bitmask was not wide enough, and as a result some bits
weren't being exposed in user_cpu_desc. Fix this.

While here, adjust some formatting.

Reviewed by: andrew
MFC after: 2 weeks
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D26069

arm64: update instruction set attribute register definitions

This adds definitions for the latest additions to the AA64ISAR[01] ID
registers. This brings these registers in sync with ARMv8.6 initial spec
release.

An future change will parse many of these fields for HWCAP features.

Reviewed by: andrew, manu, markj (all previous versions)
MFC after: 2 weeks
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D26029

Read the CPU 0 arm64 ID registers early in initarm

We also update the kernel view early in the boot. This will allow the
use of the common kernel view in ifunc resolvers.

Sponsored by: Innovate UK

Read the arm64 ID registers earlier in the boot process.

Also move parsing the registers to just after the secondary CPUs have
started. This means the kernel register view from all CPUs is available
after the CPU SYSINITs have finished, e.g. for use by ifunc resolvers.

Sponsored by: Innovate UK
Differential Revision: https://reviews.freebsd.org/D25505

Fix the spelling of identify in the arm64 identcpu code

Sponsored by: Innovate UK

Create a kernel arm64 ID register view

In preparation for using ifuncs in the kernel is is useful to have a common
view of the arm64 ID registers across all CPUs. Add this and extract the
logic for finding the lower value of two fields to a new helper function.

Sponsored by: Innovate UK
Differential Revision: https://reviews.freebsd.org/D25463

Stop performing a full icache sync when the DIC and IDC flags are set

The DIC and IDC bits in the CTR_EL0 register signal to the kernel when it
can relax the instruction cache synchronisation operations. The IDC bit
means we can relax cleaning the data cache to the point of unification
while the DIC bit means we don't need to invalidate the instruction cache
for data coherence. In both cases an appropriate barrier is still needed.

For now only implement the case where both bits are set, as is the case
on the Neoverse-N1 as used in the Amazon AWS Graviton 2 CPU. Note that
this behaviour is a optional on the N1 so we may later need to implement
only one or the other bit being set.

There is a tunable to disable each flag on boot.

Testing on a 4 core Graviton 2 instance found a significant improvement
in sys and real time when running "make buildkernel -j4", with no
significant difference in user time.

Reviewed by: markj
Sponsored by: Innovate UK
Differential Revision: https://reviews.freebsd.org/D24853

Fix the name reported when the core supports a 64-bit CCIDX

Fix the spelling of aliasing.

Sponsored by: Innovate UK

Move the arm64 cache identification to identcpu.c

This allows us to call it on a per-CPU basis and to warn if the details
are different across CPUs.

While here read the L1 I-Cache type and store this for use later by pmap.

Sponsored by: Innovate UK

Fix the spelling of the VIPT cache type field

Sponsored by: Innovate UK

Fix the following -Werror warning from clang 10.0.0:

sys/arm64/arm64/identcpu.c:1170:5: error: misleading indentation; statement is not part of the previous 'if' [-Werror,-Wmisleading-indentation]
break;
^
sys/arm64/arm64/identcpu.c:1168:4: note: previous statement is here
if (fv[j].desc[0] != '\0')
^

The break should be after the if statement, indented one level less.

MFC after: 3 days
Differential Revision: https://reviews.freebsd.org/D23871

Fix the cache type identification

DIC and IDC are supported when the field bits are set.

Sponsored by: Innovate UK

Teach the arm64 ident CPU code to print non-ID registers

Add support for non-ID registers when printing CPU information. This is
used with the cache type register to print details of the cache on boot.

Sponsored by: Innovate UK

Dynamically select LSE-based atomic(9)s on arm64.

Once all CPUs are online, determine if they all support LSE atomics and
set lse_supported to indicate this. For now the atomic(9)
implementations are still always inlined, though it would be preferable
to create out-of-line functions to avoid text bloat. This was not done
here since big.little systems exist in which some CPUs implement LSE
while others do not, and ifunc resolution must occur well before this
scenario can be detected. It does seem unlikely that FreeBSD will
ever run on such platforms, however, so converting atomic(9) to use
ifuncs is probably a good next step.

Add a LSE_ATOMICS arm64 kernel configuration option to unconditionally
select LSE-based atomic(9) implementations when the target system is
known.

Reviewed by: andrew, kib
MFC after: 1 month
Sponsored by: The FreeBSD Foundation, Amazon (hardware)
Differential Revision: https://reviews.freebsd.org/D23325

Print missing ID_AA64PFR{0,1}_EL1 register fields.

MFC after: 1 week
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D23213

Add more Arm arm64 CPU identification values

- Add all the Cortex-A CPU ID register values I can find.
- Add the Neoverse-N1 ID regiser value [1]
- Sort macros by register value.

PR: 243065
Submitted by: Ali Saidi <alisaidi AT amazon.com> [1]
Sponsored by: DARPA, AFRL (other than [1])

Add support for setting hardware breakpoints from ptrace on arm64.

Implement get/fill_dbregs on arm64. This is used by ptrace with the
PT_GETDBREGS and PT_SETDBREGS requests. It allows userspace to set hardware
breakpoints.

The struct dbreg is based on Linux to ease adding hardware breakpoint
support to debuggers.

Reviewed by: jhb
Sponsored by: DARPA, AFRL
Differential Revision: https://reviews.freebsd.org/D22195

Allow the userspace ID register fields to be read from the kernel

To allow consistent values to be used in both the kernel and userspace
create a function for these to be read from the kernel. They use a newly
created macro with the name of the ID register to read. For now there is
redundant information in the user_regs array as it still holds the CRm and
Op2 values, however this will be fixed in a later change.

This will be used by ptrace to allow hardware breakpoints in userspace.

Sponsored by: DARPA, AFRL

Add two files missed in r354170

Sponsored by: DARPA, AFRL

Make special register names lowercase so they don't conflict with future
ID register macros.

MFC after: 2 weeks
Sponsored by: DARPA, AFRL

Use tables to store the information to decode the arm64 ID registers.

Arm updates these with each new architecture revision. To help keep them
updated use a collection of tables to hold the needed information to
decode these registers.

Sponsored by: DARPA, AFRL
Differential Revision: https://reviews.freebsd.org/D22020

Sort the id_aa64*_fields arrays to be in alphanumerical order.

Sponsored by: DARPA, AFRL

Identify eMAG CPU used in Ampere Computing systems.

Reviewed by: emaste@
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D21314

arm64: Implement HWCAP

Add HWCAP support for arm64.
defines are the same as in Linux and a userland program can use
elf_aux_info to get the data.
We only save the common denominator for all cores in case the
big and little cluster have different support (this is known to
exists even if we don't support those SoCs in FreeBSD)

Differential Revision: https://reviews.freebsd.org/D17137

Rename arm64 macros in preperation for a script to generate them.

I have a script to generate most of the ID_AA64* macros from the Arm
XML source [1]. In preperation for using this we need to clean up the
macros to be in line with what the script will generate. This is the
first step, rename the macros to follow the names in said XML.

[1] https://developer.arm.com/architectures/cpu-architecture/a-profile/exploration-tools

MFC after: 1 week
Sponsored by: DARPA, AFRL
Differential Revision: https://reviews.freebsd.org/D20976

Add a hw.model sysctl oid for arm64 which reports the CPU model similar to armv6/7.

Reviewed by: andrew, manu
Differential Revision: https://reviews.freebsd.org/D20123

Sort printing of the ID registers on arm64 to be identical to the
documentation. This will simplify checking new fields when they are added.

MFC after: 2 weeks
Sponsored by: DARPA, AFRL

Impleent COMPAT_FREEBSD32 for arm64.
This is based on early work by andrew@.

Stop advertising ARMv8.3 Pointer Authentication

This needs firmware and kernel support before userspace can use it. Until
then don't advertise it's available.

MFC after: 3 days

Export ID_AA64ISAR{0,1}_EL1 to userland.

As with r338962 also export the instruction set attribute register. This
will allow userland to identify optional instructions the hardware
supports, for example in a future ifunc handler to decide which
implementation of a function to return.

Approved by: re (kib)

Export ID_AA64PFR0_EL1 to userland

Create a user view of the ID_AA64PFR0_EL1 register with values common
across all CPUs.

Approved by: re (kib)
Sponsored by: ABT Systems Ltd
Differential Revision: https://reviews.freebsd.org/D17301

Move the undefined instruction handler to identcpu.c so we have access
to the registers from boot.

Approved by: re (kib)
Sponsored by: ABT Systems Ltd
Differential Revision: https://reviews.freebsd.org/D17301

use sbuf so that lines are printed together... As aarch64 often
has SMP enabled, lines can get intermixed with other console output
making these lines hard to read...

Reviewed by: manu
Differential Revision: https://reviews.freebsd.org/D16689

ARM64: Add ThunderX2 CPU revision macro. Add ThunderX2 name in identcpu.c

Submitted by: Patryk Duda <pdk@semihalf.com>
Obtained from: Semihalf
Sponsored by: Cavium

Rename the ThunderX CPU identification macros to include the X. This is the
name people know the product by, and is consistent with the later SoC ID
macros.

Sponsored by: DARPA, AFRL

Add the ARMv8.3 ID register fields. These were found in the A-Profile
exploration tools documentation:
https://developer.arm.com/products/architecture/a-profile/exploration-tools

Sponsored by: DARPA, AFRL

Add support for handling undefined instructions in userspace and the
kernel. We can register callbacks to perform the required operation on the
saved registers before returning.

This is initially used to work around a bug in old versions of QEMU that
trigger such an exception when reading from an ID register when it should
load z zero value.

I expect this could be used with other exception types, e.g. to emulate
special register access from userland.

Sponsored by: DARPA, AFRL

Not all CPUs handle reading ID_AA64MMFR2_EL1 (e.g. qemu), disable it for now.

Sponsored by: DARPA, AFRL

Add the ATS1E1 case to the ID_AA64MMFR1_EL1 decoding.

Sponsored by: DARPA, AFRL

Use the correct mask when printing undecoded fields from the
ID_AA64MMFR2_EL1 register.

Sponsored by: DARPA, AFRL

Fix a mismerge, make sure PRINT_ID_AA64_MMFR2 has a unique value.

Sponsored by: DARPA, AFRL

Add the ARMv8.2 ID register additions and use them to decode the register
values. As not all assemblers understand the new ID_AA64MMFR2_EL1 register
add a macro to access it. This seems to be safe for older CPUs to read this
new register, with them returning zero.

Sponsored by: DARPA, AFRL

Uppercase the special register names in identcpu to be more consistent with
the other source files.

Sponsored by: DARPA, AFRL

Make the bit mask of ARMv8 ID registers to print sparse to keep values
close, but without having to change all values when new registers are added.

Sponsored by: DARPA, AFRL

Add more ARM Ltd parts to the list of knows CPUs.

Submitted by: Jon Brawn <jon@brawn.org>

Add ARM Cortex A72 to CPU list

This change is required to properly detect CPUs
on Marvell Armada 80x0/70x0 SoC family.

Submitted by: Rafal Kozik <rk@semihalf.com>
Reviewed by: andrew, cognet (mentor)
Approved by: cognet (mentor)
Sponsored by: Semihalf
Differential Revision: https://reviews.freebsd.org/D12184

Fix improper use of "its".

Sponsored by: Dell EMC Isilon

Move printing the AArch64 ID registers to a new SYSINIT, the previous
location only prints them when booting on SMP with multiple cores.

MFC after: 1 week
Sponsored by: DARPA, AFRL

Add a warning about a known erratum we have observed on ThunderX pass 1.1.
As this is evaluation hardware with only a few users, and there is a lack
of information add a warning when booting on this hardware.

Reported by: cognet
Obtained from: ABT Systems Ltd
MFC after: Instant
Sponsored by: The FreeBSD Foundation

Add the ARMv8.1 identification registers to the list we print when booting.

MFC after: 1 week
Sponsored by: ABT Systems Ltd

Decode and print the ID_AA64* registers on boot. These registers hold
information on what the core supports. In most cases these will be
identical across most CPUs in the SoC, however there may be the case where,
with a big.LITTLE setup they may differ. In this case we print the
decoded data on all CPUs.

Reviewed by: kib
Sponsored by: ABT Systems Ltd
Differential Revision: https://reviews.freebsd.org/D4725

Rework CPU identification on ARM64

This commit reworks the code responsible for identification of
the CPUs during runtime.
It is necessary to provide a way for workarounds and erratums
to be applied only for certain HW versions.

The copy of MIDR is now stored in pcpu to provide a fast and
convenient way for assambly code to read it (pcpu is used quite often
so there is a chance it's inside the cache).
The MIDR is also better way of identification than using user-friendly
cpu_desc structure, because it can be compiled into comparision of
single u32 with only one access to the memory - this is crucial
for some erratums which are called from performance-critical
places.

Changes in cpu_identify makes this function safe to be called
on non-boot CPUs.

New function CPU_MATCH was implemented which returns boolean
value based on mathing masked MIDR with chip identification.
Example of usage:

printf("is thunder: %d\n", CPU_MATCH(CPU_IMPL_MASK | CPU_PART_MASK,
CPU_IMPL_CAVIUM, CPU_PART_THUNDER, 0, 0));
printf("is generic: %d\n", CPU_MATCH(CPU_IMPL_MASK | CPU_PART_MASK,
CPU_IMPL_ARM, CPU_PART_FOUNDATION, 0, 0));

Reviewed by: andrew
Obtained from: Semihalf
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D3030

Bring in the start of the arm64 kernel.

This is only the minimum set of files needed to boot in qemu. As such it is
missing a few things.

The bus_dma code is currently only stub functions with a full implementation
from the development tree to follow.

The gic driver has been copied as the interrupt framework is different. It
is expected the two drivers will be merged by the arm intrng project,
however this will need to be imported into the tree and support for arm64
would need to be added.

This includes code developed by myself, SemiHalf, Ed Maste, and Robin
Randhawa from ARM. This has been funded by the FreeBSD Foundation, with
early development by myself in my spare time with assistance from Robin.

Differential Revision: https://reviews.freebsd.org/D2199
Reviewed by: emaste, imp
Relnotes: yes
Sponsored by: The FreeBSD Foundation