arm64/gicv3: Check if the hardware supports LPIs

Some simulators have the ITS in the DTB passed to the kernel, however
it is a runtime configuration option to enable it.

Check the GICD_TYPER register to see if LPIs are enabled before
attaching the ITS driver.

Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D44914

arm64: Switch the address argument to cpu_*cache* to a pointer

No functional change, but this reduces diffs with CheriBSD downstream.

Reviewed by: andrew
Sponsored by: University of Cambridge, Google, Inc.
Differential Revision: https://reviews.freebsd.org/D44342

arm64 gicv3: Use void pointers instead of vm_offset_t

This removes the need for local diffs in CheriBSD where vm_offset_t
is an address and not the same as a uintptr_t. No functional change.

Reviewed by: imp, andrew
Sponsored by: University of Cambridge, Google, Inc.
Differential Revision: https://reviews.freebsd.org/D44341

gicv3: If the LPI is already allocated, remember it

If the LPI Configuration Tabel has been pre-allocated by the boot
loader, then we have to remember PROPBASER and use it rather than
allocating memory for it ourselves. Linux provides us with a reserved
table that contains all the gicv3 allocations, so make sure what we read
from PROPBASER matches something in that table. Normally, bare metal
boot loaders leave the gic in a reset state. However, Linux brings it up
fully so it can do I/O to boot the next kernel via kexec. Since the
gicv3 PENDBASER can't be reset while running due to undefined behavior,
we must reuse what's there for both PENDBASER and PROPBASER.

With this commit, the workaround is complete. Details are at
https://lkml.iu.edu/hypermail/linux/kernel/1809.2/06246.html
and pointers in the thread.

Sponsored by: Netflix
Reviewed by: andrew
Differential Revision: https://reviews.freebsd.org/D44038

gicv3: In its_init_cpu_lpi record each cpu's PENDBASER

When we're using the preallocated memory for gicv3, record each cpu's
PENDBASER where we'd normally allocate memory for it. Make sure that
memory is in the excluded list and map the PA to VA and store that, to
mimic what we do with the allocation case.

Sponsored by: Netflix
Reviewed by: andrew
Differential Revision: https://reviews.freebsd.org/D44037

gicv3: Report PENDBASE when bootverbose

Report some stats about PENDBASE when we're running under
bootverbose. We don't do this by default because experience has been
excess output in this routine hangs the system.

Sponsored by: Netflix
Reviewed by: andrew
Differential Revision: https://reviews.freebsd.org/D44036

gicv3: Only set the redistributor base if we're not prealloced

Only set the redistributor base if we're not reallocated. If we are
preallocated, leave it all alone.

Sponsored by: Netflix
Reviewed by: andrew
Differential Revision: https://reviews.freebsd.org/D44035

gicv3: Don't allocate pend_base if we're already started

If the gicv3 is already started, then don't allocate memory for the
pend_base tables.

Sponsored by: Netflix
Reviewed by: andrew
Differential Revision: https://reviews.freebsd.org/D44034

gicv3: Panic if the gicv3 already running

Due to undefined behavior, it's impossible to re-program a gicv3 ITS
table once it's programmed once. Memory corruption happens otherwise.
Panic if we detect the LPI is already enabled.

Sponsored by: Netflix
Reviewed by: andrew
Differential Revision: https://reviews.freebsd.org/D44033

gicv3: Change how we initialize its children.

The current code is written such that all the attach routines can do so
in parallel. However, newbus serializes children today, and is likely to
do so in the future. Only allocate memory for the first time. Add an
assertion that this memory is allocated for larger units.

Sponsored by: Netflix
Reviewed by: andrew
Differential Revision: https://reviews.freebsd.org/D44032

gicv3_its: Drop a couple of redundant vm_offset_t casts

These are already vm_offset_t, no need to cast.

sys: Automated cleanup of cdefs and other formatting

Apply the following automated changes to try to eliminate
no-longer-needed sys/cdefs.h includes as well as now-empty
blank lines in a row.

Remove /^#if.*\n#endif.*\n#include\s+<sys/cdefs.h>.*\n/
Remove /\n+#include\s+<sys/cdefs.h>.*\n+#if.*\n#endif.*\n+/
Remove /\n+#if.*\n#endif.*\n+/
Remove /^#if.*\n#endif.*\n/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/types.h>/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/param.h>/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/capsicum.h>/

Sponsored by: Netflix

gicv3: Add logging for when its_device_alloc fails

Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D41566

gicv3: Support indirect ITS tables

The GICv3 ITS device supports two options for device tables. Currently
we support a single table to hold all device IDs, however when the
device ID space grows large this can be too large for the GITS_BASER
register to describe.

To handle this case, and to reduce the memory needed when this space
is sparse support the second option, the indirect table. The indirect
table is a 2 level table where the first level contains the physical
address of the second with a valid bit. The second level is an ITS
page sized table where each entry is the original entry size.

As we don't need to allocate a second level table for devices IDs that
don't exist this can reduce the allocation size.

Reviewed by: gallatin
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D41555

gicv3: Add checks for the device ID

Add checks that the device ID is supported by the hardware and is
within the range allocated when the driver attaches.

Reviewed by: gallatin, imp
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D41554

gicv3: Add a verbose message for unknown tables

Add a message under bootverbose when we find a gicv3 its table type
that is unknown.

Reviewed by: gallatin, imp
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D41553

gicv3: Stop setting the esize field

The GITS_BASER esize field is read-only, there is no need to change it.

Reviewed by: gallatin, imp
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D41552

Revert "gicv3: Stop setting the esize field"

This reverts commit 47a4b8ca96e3e18262629c05e46228c99a618487.

It has the wrong differential review link

Revert "gicv3: Add a verbose message for unknown tables"

This reverts commit 7f9694ad7e7c271e341cba65ce482a552ea47230.

It has the wrong differential review link

Revert "gicv3: Add checks for the device ID"

This reverts commit 950421e231e5e5067dcb09b026e14f9810cc3cd4.

It has the wrong differential review link

gicv3: Add checks for the device ID

Add checks that the device ID is supported by the hardware and is
within the range allocated when the driver attaches.

Reviewed by: gallatin, imp
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D41551

gicv3: Add a verbose message for unknown tables

Add a message under bootverbose when we find a gicv3 its table type
that is unknown.

Reviewed by: gallatin, imp
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D41551

gicv3: Stop setting the esize field

The GITS_BASER esize field is read-only, there is no need to change it.

Reviewed by: gallatin, imp
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D41551

gicv3: Split out finding the page size

When adding indirect (2 level) tabled we will need to know the page
size to calculate the size of the level 1 table. To allow for this find
the page size before entering the loop to calculate the final register
value.

Reviewed by: gallatin, imp
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D41551

sys: Remove $FreeBSD$: one-line .c pattern

Remove /^[\s*]*__FBSDID\("\$FreeBSD\$"\);?\s*\n/

gicv3: Use an offset to find the redist registers

To find the redistributor registers use the resource we have already
found and add an offset. This removed the need to create a
per-redistributor resource as it can now be a pointer to the resource
found in attach.

While here check the offset is within the bounds of the resource. Some
ACPI tables list each redistributor as a separate memory range, even
if they are physically contiguous. In this case we may not have each
resource virtually contiguous with neighbouring resources. This can
lead to a data abort when reading past the resource range.

Reviewed by: kevans
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D40263

Clear the indirect flag in the GICv3 ITS driver

Summary:
The indirect flag tells the hardware to use a flat or two level table.
As we only support using the flat table ensure the flag that marks
which is in use is set correctly.

We can't rely on this being set correctly as some firmware may set the
indirect flag, e.g. booting from LinuxBoot.

Reviewed by: imp
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D36873

gicv3_its: Fix typo

Sponsored by: Netflix

iommu_gas: pass size to iommu_map without rounding

Let the caller to iommu_map pass the size parameter without rounding
it up to a multiple of page size. Let iommu_map round it up when
necessary, which is not all of the time, so that in some cases less
space is reserved.

Reviewed by: alc, kib (previous version)
Tested by: pho, br
Discussed with: andrew
MFC after: 3 weeks
Differential Revision: https://reviews.freebsd.org/D35424

arm64: Remove unused devclass arguments to DRIVER_MODULE.

Check if iommu ctx is valid before passing it to functions.

Differential Revision: https://reviews.freebsd.org/D35130
Sponsored by: UKRI

arm64: gic: disable the ITS if it's enabled prior to configuration

The ITS is defined to be disabled on a warm reset, but we may be coming
in via another kernel/hypervisor type setup where the ITS has been
previously configured then relinquished to the next kernel in the chain.

If it's enabled, the later configuration of GITS_BASER will almost
certainly fail -- clear it to prevent that.

Reviewed by: andrew
Sponsored by: Ampere Computing
Submitted by: Klara, Inc.
Differential Revision: https://reviews.freebsd.org/D34546

arm64: Check for intrng-reported errors in gicv3_its

Currently, any errors when adding a PIC child handler are ignored,
instead just continuing on to registering that PIC as an MSI, and
ignoring any errors that occur for that too.

Reviewed by: andrew
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D33342

Fix set but not used warnings in arm64 core code

arm64/gicv3: improve a panic message

Print the device/unit in the panic message for which we cannot get
the MSI device ID to have a clue where to start looking.
While here use __func__ instead of hardcoding the function name.

Reviewed by: emaste
Differential Revision: https://reviews.freebsd.org/D32917

Use a 64 bit read to access GICR_TYPER

The GICv3 ITS only needs to implement 32 bit access to the GICR_TYPER
when the CPU implements AArch32. As this may not always be the case
use a 64 bit read when checking if the ITS is enabled on the CPU.

PR: 258217
Reported by: Olivier Delande <olivier.delande@provenrun.com>
Sponsored by: The FreeBSD Foundation

Remove "All Rights Reserved" from FreeBSD Foundation sys/ copyrights

These ones were unambiguous cases where the Foundation was the only
listed copyright holder (in the associated license block).

Sponsored by: The FreeBSD Foundation

gicv3_its: Flush cache after allocating ITT memory

It has to be zeroed before committing it to device.
We do that by allocating it with M_ZERO, but there was no
memory barrier or cache flush to ensure its sees it zeroed.
This fixes MSIX on LS1028A SoC.

Submitted by: Kornel Duleba <mindal@semihalf.com>
Reviewed by: andrew
Obtained from: Semihalf
Sponsored by: Alstom Group
Differential Revision: https://reviews.freebsd.org/D30033

gicv3_its: Don't restrict target CPUs based on SRAT

ACPI Sec 5.2.16.5 (SRAT, GIC Interrupt Translation Service (ITS)
Affinity Structure) says:

> The GIC ITS Affinity Structure provides the association between
> a GIC ITS and a proximity domain. This enables the OSPM to
> discover the memory that is closest to the ITS, and use that in
> allocating its management tables and command queue.

Previously the ITS driver was using the proximity domain to
restrict which CPUs can be targeted by an LPI. We keep that logic
just for the original dual socket ThunderX which cannot forward
LPIs between sockets.

We also use the SRAT entry for its intended purpose of attempting
to allocate ITS table structures near the ITS.

Reviewed by: andrew
Sponsored by: Ampere Computing LLC
Differential Revision: https://reviews.freebsd.org/D28340

gicv3_its: Leave LPI interrupts enabled during handling

This follows the behavior on x86 where edge triggered interrupts are
not disabled when executing the handler. Because the ITS is a shared
resource, contention for the command queue lock can be substantial.

Suggested by: gallatin
Reviewed by: andrew
Tested by: gallatin
Sponsored by: Ampere Computing LLC
Differential Revision: https://reviews.freebsd.org/D28709

Use the new PAGE_SIZE_4K in the GICv3 driver

Stop assuming PAGE_SIZE is 4k in the GICv3 ITS driver. We could use
a 16k or 64k page in the future.

Sponsored by: Innovate UK

Take the ITS device lock around gicv3_its_release_irqsrc() since that
function checks that the mutex lock is owned.

This fixes 'devctl disable re0' operation.

Sponsored by: Innovate DSbD
Differential Revision: https://reviews.freebsd.org/D26904

Add IOMMU support to GICv3 Interrupt Translation Service (ITS) driver.

Submitted by: andrew
Sponsored by: Innovate DSbD
Differential Revision: https://reviews.freebsd.org/D26878

arm64: clean up empty lines in .c and .h files

Create MSI/MSI-X isrcs as needed in the GICv3 ITS driver

Previously we would create an isrc for each MSI/MSI-X interrupt. This
causes issues for other interrupt sources in the system, e.g. a GPIO
driver, as they may be unable to allocate interrupts. This works around
this by allocating the isrc only when needed.

Reported by: alisaidi@amazon.com
Reviewed by: mmel
Sponsored by: Innovaate UK
Differential Revision: https://reviews.freebsd.org/D24876

Export tracing facility of GIC500 ITS block.
Possibility of tracing of processing message based interrupts is very
useful for debugging of PCIe driver, mainly for its MSI part.

MFC after: 1 week

arm64: remove no longer needed atomic_load_ptr casts

Remove the GICv3 ITS irq and replace it with an ID

In r357324 most of the use of gi_irq was moved to gi_lpi. Complete this
with the last few places we need the IRQ value and create gi_id for the
per-device value we need.

MFC after: 2 weeks
Sponsored by: DARPA, AFRL

Use a unique name for the GICv3 ITS vmem

When there are multiple GICv3 ITS devices we don't know which vmem is for
which device. Use device_get_nameunit to get a per-device name.

MFC after: 2 weeks
Sponsored by: DARPA, AFRL

Disable the use of the quantum cache in the GICv3 ITS

This uses UMA to allocate space. It causes issues when there are multiple
ITS devices in the system where interrupts are not allocated from a low
address on some interrupt controllers. Disabling the quantum cache fixes
this on the Neoverse N1 SDP.

MFC after: 2 weeks
Sponsored by: DARPA, AFRL

Call the MAPTI command earlier in the ITS driver

The GICv3 Software Overview suggests when allocating a new MSI/MSI-X
interrupt we need to call MAPD followed by MAPTI. Unfortunately the code
would place a MOVI command between these. This is invalid as it needs
values set by the MAPTI to be present.

Re-order so we allocate a temporary CPU for the interrupt, then use the
MAPTI command to assign the MSI to it.

MFC after: 2 weeks
Sponsored by: DARPA, AFRL

Only create one ITS configuration table

When there are multiple ITS devices in the system we would allocate a
configuration table for each, however only one table is needed as all the
ITS devices share this.

Allocate a table only when the global table is unset.

While here fix the type of this to be a pointer to a uint8_t array as the
entries are all 8 bits wide.

MFC after: 2 weeks
Sponsored by: DARPA, AFRL

Shift the ITS processor ID after reading it.

When using the processor ID value we mask off the low and high bits that
should be zero. Unfortunatly we don't shift the ID value so it won't be
affected. Add the shift when reading the ID as this will need to align
with the address based target value.

MFC after: 2 weeks
Sponsored by: DARPA, AFRL

arm64 gicv3_its: enable all ITS blocks for a CPU

We now support multiple ITS blocks raising interrupts to a CPU.
Add all available CPUs to the ITS when no NUMA information is
available.

This reverts the check added in r340602, at that tim we did not
suppport multiple ITS blocks for a CPU.

Differential Revision: https://reviews.freebsd.org/D20417

arm64 gicv3_its: Fix a typo

Fix 'Cavium' spelling in errata description.

Reviewed by: andrew
Differential Revision: https://reviews.freebsd.org/D20418

gicv3_its: do LPI init only once per CPU

The initialization required for LPIs (setting up pending tables etc.)
has to be done just once per CPU, even in the case where there are
multiple ITS blocks associated with the CPU.

Add a flag lpi_enabled in the per-cpu distributor info for this and
use it to ensure that we call its_init_cpu_lpi() just once.

This enables us to support platforms where multiple GIC ITS blocks
can generate LPIs to a CPU.

Reviewed by: andrew
Differential Revision: https://reviews.freebsd.org/D19844

gicv3_its: refactor LPI init into a new function

Move the per-cpu LPI intialization to a separate function. This is
in preparation for a commit that does LPI init only once for a CPU,
even when there are multiple ITS blocks associated with the CPU.

No functional changes in this commit.

Reviewed by: andrew
Differential Revision: https://reviews.freebsd.org/D19843

gic_v3: consolidate per-cpu redistributor information

Update 'struct gic_redists' to consolidate all per-cpu redistributor
information into a new 'struct redist_pcpu'. Provide a new interface
(GICV3_IVAR_REDIST) for the GIC driver, which can be used to retrieve
the per-cpu data.

This per-cpu redistributor struct will be later used to improve the
GIC ITS setup.

While there, remove some unused fields in gic_v3_var.h interface.
No functional changes.

Reviewed by: andrew
Differential Revision: https://reviews.freebsd.org/D19842

Extract eventfilter declarations to sys/_eventfilter.h

This allows replacing "sys/eventfilter.h" includes with "sys/_eventfilter.h"
in other header files (e.g., sys/{bus,conf,cpu}.h) and reduces header
pollution substantially.

EVENTHANDLER_DECLARE and EVENTHANDLER_LIST_DECLAREs were moved out of .c
files into appropriate headers (e.g., sys/proc.h, powernv/opal.h).

As a side effect of reduced header pollution, many .c files and headers no
longer contain needed definitions. The remainder of the patch addresses
adding appropriate includes to fix those files.

LOCK_DEBUG and LOCK_FILE_LINE_ARG are moved to sys/_lock.h, as required by
sys/mutex.h since r326106 (but silently protected by header pollution prior
to this change).

No functional change (intended). Of course, any out of tree modules that
relied on header pollution for sys/eventhandler.h, sys/lock.h, or
sys/mutex.h inclusion need to be fixed. __FreeBSD_version has been bumped.

arm64 gicv3: add IORT and NUMA support

acpi_iort.c has added support to query GIC proximity and MSI XREF
ID for GIC ITS blocks. Use this when GIC ITS blocks are initialized
from ACPI.

Reviewed by: andrew
Differential Revision: https://reviews.freebsd.org/D18003

gitv3_its: fixes for multiple GIC ITS blocks

First pass of support for multiple GIC ITS blocks with ACPI.
Changes are to:
* register the correct subset of interrupts with pic_register
in case of ACPI.
* initialize just the cpu interface for the first ITS, when
domain information is not avialable. This has to be done
until we split the per-CPU init to do LPI setup just once.
* remove duplicate check for the GIC ITS domain, the sc_cpus
are setup from domain, so the check again in per-CPU init
seems unnecessary.

Reviewed by: andrew
Differential Revision: https://reviews.freebsd.org/D17841

Use the correct address to write back to memory in the GICv3 ITS driver.
This seems to no be needed on supported hardware as they are cache-coherent,
however this may not be the case on all platforms.

Sponsored by: DARPA, AFRL

Create macros for the ACPI interrupt cross references. This is considered a
band aid until a better solution to find the correct interrupt controller
can be found.

While here fix one place in the GICv3 ITS driver where the offset wasn't
correctly applied.

Sponsored by: DARPA, AFRL
Sponsored by: Cavium (Hardware)

Register each GICv3 ITS driver with a useful cross reference. We currently
only use the first driver, however this may change in the future and
hardware exists with multiple ITS devices.

Sponsored by: DARPA, AFRL
Sponsored by: Cavium (Hardware)

Move setting the IRQ base and length into the common GICv3 ITS attach
function. This is common across both ACPI and FDT.

Sponsored by: ABT Systems Ltd
Sponsored by: Cavium (Hardware)

Correctly set the 16kB page size field in the ITS BASER register. Some
new arm64 hardware, e.g. ThunderX2, seems to use this page size so was
failing to attach as the register value read back was incorrect.

While here fix the spelling on shareability.

Sponsored by: DARPA, AFRL

iAdd ACPI attachments the the GIC and GICv3 interrupt controller drivers.
For each we need to walk the MADT to find which we have, then add the
driver as needed. As each may have a child they will each walk the same
table to find these details.

Reviewed by: mmel
Obtained from: ABT Systems Ltd
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D8720

Do pass removing some write-only variables from the kernel.

This reduces noise when kernel is compiled by newer GCC versions,
such as one used by external toolchain ports.

Reviewed by: kib, andrew(sys/arm and sys/arm64), emaste(partial), erj(partial)
Reviewed by: jhb (sys/dev/pci/* sys/kern/vfs_aio.c and sys/kern/kern_synch.c)
Differential Revision: https://reviews.freebsd.org/D10385

Allocate the ITS translation table with a 64k page alignment. This is the
largest alignment the ITS can require.

This fixes a bug with the ARM Architecture Envelope Model (AEM) where it
only allows 64k pages so will fail to attach the ITS device when this table
is not sufficiently aligned.

Sponsored by: DARPA, AFRL

Use mp_maxid when iterating over CPUs as we may have sparse id allocations.

Sponsored by: DARPA, AFRL

Add support for multiple GICv3 ITS devices. For this we add sc_irq_base
and sc_irq_length to the softc to handle the base number of IRQs available,
make gicv3_get_nirqs return the number of available interrupt IDs, and
limit which CPUs we send interrupts to based on the numa domain.

The last point is only strictly needed on a dual socket ThunderX where we
are unable to send MSI/MSI-X interrupts between sockets.

Sponsored by: DARPA, AFRL

Start to teach the GICv3 driver about NUMA. On ThunderX we may have
multiple ITS devices, however we only want a single ITS device to be
configured on each CPU. To fix this only enable ITS when the node matches
the CPUs node.

Sponsored by: DARPA, AFRL

Move the common bit manipulation macros from the GICv3 header to the
common GIC header file.

Obtained from: ABT Systems Ltd
Sponsored by: The FreeBSD Foundation

Move structures only used by the GICv3 ITS driver from a shared header to
the ITS driver file. There is no need for other drivers to need to know
about these structures.

Obtained from: ABT Systems Ltd
MFC after: 1 month
Sponsored by: The FreeBSD Foundation

Add the GICv3 ITS intrng driver. As the interface to the interrupt
framework has significantly changed the driver has moved to a new file.
While it shares some code with the existing driver this has been modified
to work better with the intrng framework.

This has been tested on the ThunderX servers in the netperf cluster and has
been used to boot them for other testing, including DTrace and hwpmc.

With this we can use intrng on all supported arm64 platforms I was able to
test on. It is expected we will move to intrng soon, and disable the old
arm64 interrupt framework.

Obtained from: ABT Systems Ltd
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D6437