arm64/gicv3: Check if the hardware supports LPIs

Some simulators have the ITS in the DTB passed to the kernel, however
it is a runtime configuration option to enable it.

Check the GICD_TYPER register to see if LPIs are enabled before
attaching the ITS driver.

Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D44914

sys: Automated cleanup of cdefs and other formatting

Apply the following automated changes to try to eliminate
no-longer-needed sys/cdefs.h includes as well as now-empty
blank lines in a row.

Remove /^#if.*\n#endif.*\n#include\s+<sys/cdefs.h>.*\n/
Remove /\n+#include\s+<sys/cdefs.h>.*\n+#if.*\n#endif.*\n+/
Remove /\n+#if.*\n#endif.*\n+/
Remove /^#if.*\n#endif.*\n/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/types.h>/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/param.h>/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/capsicum.h>/

Sponsored by: Netflix

sys: Remove $FreeBSD$: one-line .c pattern

Remove /^[\s*]*__FBSDID\("\$FreeBSD\$"\);?\s*\n/

Remove MAXCPUS from the gicv3 driver

We create a static array of pointers to per-CPU data. Because the cpuid
space on arm64 is not sparse there is no need to add an extra level of
indirection. Move to use mallocarray to allocate the redistributors as
a single array.

Sponsored by: Arm Ltd

gicv3: Use an offset to find the redist registers

To find the redistributor registers use the resource we have already
found and add an offset. This removed the need to create a
per-redistributor resource as it can now be a pointer to the resource
found in attach.

While here check the offset is within the bounds of the resource. Some
ACPI tables list each redistributor as a separate memory range, even
if they are physically contiguous. In this case we may not have each
resource virtually contiguous with neighbouring resources. This can
lead to a data abort when reading past the resource range.

Reviewed by: kevans
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D40263

arm64: gicv3: setup PPIs on all APs after they're online

For all PPIs setup earlier than SI_SUB_SMP, PIC_INIT_SECONDARY ends up
cleaning these up for each AP as it comes online. Once they're online,
we don't currently do anything to make sure they're configured for other
APs. Fix it by using smp_rendezvous for the meaty bits of configuring a
PPI, which will just do single-thread behavior before APs are online but
do the right thing for other CPUs after.

While we're here, make sure redistributor config is correct for other
APs as they come online in gic_v3_init_secondary.

Reported/Tested by: Souradeep Chakrabarti (Microsoft/Hyper-V)
Reviewed by: andrew (before slight refactor)
Differential Revision: https://reviews.freebsd.org/D40112

arm: remove passing trapframe to intr_ipi_dispatch()

This was needed before INTRNG was in place and handling the push of
curthread->td_intr_frame. Since INTRNG now handles this, there is no
longer and need for playing around with the frame inside IPI interrupts.

arm64: Hyper-V: vPCI: SPI MSI mapping for gic v3 acpi in arm64

Microsoft Azure Hyper-V uses SPI to map MSI in ARM64, instead of
using LPI IDS. To enable that we need to have gic registered with
ACPI_MSI_XREF and gic acpi to map SPI for MSI.

This is the 1st of the three patchs to enable Hyper-V vPCI support
in arm64.

Reviewed by: andrew, emaste, whu
Tested by: Souradeep Chakrabarti <schakrabarti@microsoft.com>
Obtained from: Souradeep Chakrabarti <schakrabarti@microsoft.com>
Sponsored by: Microsoft
Differential Revision: https://reviews.freebsd.org/D37763

Teach the GICv3 driver to translate memory ranges

As with the GICv1/2 driver teach the GICv3 driver to translate memory
ranges of children. This allows us to create a common
bus_alloc_resource implementation for bot hACPI and FDT attachments.

Sponsored by: The FreeBSD Foundation

Move the GICv3 bus_print_child function to the parent

This should be common for both ACPI and FDT. Move this to the common
part of the driver.

Sponsored by: The FreeBSD Foundation

Teach the GICv3 driver about a vgic child

This will be used by bhyve to attach a virtual GIC driver.

Sponsored by: Innovate UK
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D36590

Fix set but not used warnings in arm64 core code

Add a gic interface to allocate MSI interrupts

The previous update to handle the gicv2m as a child of the gicv3 driver
assumed there was only a single gicv2m child. On some hardware there
are multiple children. Support this by removing the mbi ivars and
adding a new interface to handle MSI allocation in a given range.

Tested by: mw, trasz
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D32224

Remove a bogus assertion from the gic drivers

When setting a message based interrupt range we set the base and count.
In an earlier the count was implemented as an end value, however the
asserts used to check this value was correct were incorrectly left in.

Reported by: emaste
Sponsored by: The FreeBSD Foundation

Add support for gicv2m as a child of gicv3

On some systems, e.g. Parallels set to host a Linux VM under an M1 Mac,
there is a GICv2m as a child of the GICv3. We previously assumed the
GICv2m was always a child of a GICv2. Fix this by adding the needed
support to the GICv3 driver.

PR: 258136
Reported by: trasz
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D31768

Remove "All Rights Reserved" from FreeBSD Foundation sys/ copyrights

These ones were unambiguous cases where the Foundation was the only
listed copyright holder (in the associated license block).

Sponsored by: The FreeBSD Foundation

Add more GIC and GICv3 registers

These aren't used by either driver, however they will be needed by
bhyve on arm64 to emulate a GICv3 interrupt controller.

Sponsored by: Innovate UK

gic_v3: add message based interrupts support

Pull Request: https://github.com/freebsd/freebsd-src/pull/451

gic_v3: Use 64-bit writes to GICD_IROUTERn

While 32-bit accesses to GICD_IROUTERn are allowed,
this is a 64-bit register.

Pull Request: https://github.com/freebsd/freebsd-src/pull/451

make maximum interrupt number tunable on ARM, ARM64, MIPS, and RISC-V

Use a machdep.nirq tunable intead of compile-time constant NIRQ
as a value for maximum number of interrupts. It allows keep a system
footprint small by default with an option to increase the limit
for large systems like server-grade ARM64

Reviewd by: mhorne
Differential Revision: https://reviews.freebsd.org/D27844
Submitted by: Klara, Inc.
Sponsored by: Ampere Computing

arm64 gic: Drop unused GICV3_IVAR_REDIST_VADDR

Now that GICV3_IVAR_REDIST is available, GICV3_IVAR_REDIST_VADDR
is unused and can be removed. Drop the define and add a comment.

Reviewed by: andrew
Differential Revision: https://reviews.freebsd.org/D20454

gicv3_its: do LPI init only once per CPU

The initialization required for LPIs (setting up pending tables etc.)
has to be done just once per CPU, even in the case where there are
multiple ITS blocks associated with the CPU.

Add a flag lpi_enabled in the per-cpu distributor info for this and
use it to ensure that we call its_init_cpu_lpi() just once.

This enables us to support platforms where multiple GIC ITS blocks
can generate LPIs to a CPU.

Reviewed by: andrew
Differential Revision: https://reviews.freebsd.org/D19844

gic_v3: consolidate per-cpu redistributor information

Update 'struct gic_redists' to consolidate all per-cpu redistributor
information into a new 'struct redist_pcpu'. Provide a new interface
(GICV3_IVAR_REDIST) for the GIC driver, which can be used to retrieve
the per-cpu data.

This per-cpu redistributor struct will be later used to improve the
GIC ITS setup.

While there, remove some unused fields in gic_v3_var.h interface.
No functional changes.

Reviewed by: andrew
Differential Revision: https://reviews.freebsd.org/D19842

Rename the ThunderX CPU identification macros to include the X. This is the
name people know the product by, and is consistent with the later SoC ID
macros.

Sponsored by: DARPA, AFRL

iAdd ACPI attachments the the GIC and GICv3 interrupt controller drivers.
For each we need to walk the MADT to find which we have, then add the
driver as needed. As each may have a child they will each walk the same
table to find these details.

Reviewed by: mmel
Obtained from: ABT Systems Ltd
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D8720

Do pass removing some write-only variables from the kernel.

This reduces noise when kernel is compiled by newer GCC versions,
such as one used by external toolchain ports.

Reviewed by: kib, andrew(sys/arm and sys/arm64), emaste(partial), erj(partial)
Reviewed by: jhb (sys/dev/pci/* sys/kern/vfs_aio.c and sys/kern/kern_synch.c)
Differential Revision: https://reviews.freebsd.org/D10385

As with r325242 use mp_maxid when iterating over CPUs in the GICv3 driver.

Sponsored by: DARPA, AFRL

Add support for multiple GICv3 ITS devices. For this we add sc_irq_base
and sc_irq_length to the softc to handle the base number of IRQs available,
make gicv3_get_nirqs return the number of available interrupt IDs, and
limit which CPUs we send interrupts to based on the numa domain.

The last point is only strictly needed on a dual socket ThunderX where we
are unable to send MSI/MSI-X interrupts between sockets.

Sponsored by: DARPA, AFRL

Start to teach the GICv3 driver about NUMA. On ThunderX we may have
multiple ITS devices, however we only want a single ITS device to be
configured on each CPU. To fix this only enable ITS when the node matches
the CPUs node.

Sponsored by: DARPA, AFRL

Use proper fields to check for interrupt trigger mode.

Initialize GIC[DR]_IGROUPRn registers for GICv3

In case where GICD_CTLR.DS is 1, the IGROUPR registers are RW in
non-secure state and has to be initialized to 1 for the
corresponding interrupts to be delivered as Group 1 interrupts.

Update gic_v3_dist_init() and gic_v3_redist_init() to initialize
GICD_IGROUPRn and GICR_IGROUPRn respectively to address this. The
registers can be set unconditionally since the writes are ignored
in non-secure state when GICD_CTLR.DS is 0.

This fixes the hang on boot seen when running qemu-system-aarch64
with machine virt,gic-version=3

Create two GIC ivars to find the bus type and GIC hardware version. These
will be used by the gicv2m and ITS ACPI drivers to only attach to the
correct parent.

Obtained from: ABT Systems Ltd
Sponsored by: The FreeBSD Foundation

Move the common bit manipulation macros from the GICv3 header to the
common GIC header file.

Obtained from: ABT Systems Ltd
Sponsored by: The FreeBSD Foundation

Pull the common FDT interrupt values into a new header rather than be magic
numbers.

Sponsored by: DARPA, AFRL

Add support for SPI-mapped MSI interrupts in GICv3.

PIC_SETUP_INTR implementation in GICv3 did not allow
for setting up interrupts without included FDT
description. GICv2m-like MSI interrupts, which map
MSI messages to SPI interrupt lines, may not have
a description in FDT. Add support for such interrupts
by setting the trigger and polarity to the appropriate
values for MSI (edge, high) and get the hardware
IRQ number from the corresponding ISRC.

Obtained from: Semihalf
Submitted by: Michal Stanek <mst@semihalf.com>
Sponsored by: Annapurna Labs
Reviewed by: wma
Differential Revision: https://reviews.freebsd.org/D7662

Fix the arm64 non-SMP build, active_irq is a uint64_t so cast it through
a uintmax_t.

Obtained from: ABT Systems Ltd
MFC after: 1 week
Sponsored by: The FreeBSD Foundation

Move gic_v3_irqsrc into the GICv3 driver source as it's only needed there.
Remove unused macros from the GICv3 header.

Obtained from: ABT Systems Ltd
MFC after: 1 month
Sponsored by: The FreeBSD Foundation

Remove the non-INTRNG support from the GICv3 interrupt controller driver.
This is no longer needed.

Obtained from: ABT Systems Ltd
MFC after: 1 month
Sponsored by: The FreeBSD Foundation

INTRNG - change the way how an interrupt mapping data are provided
to the framework in OFW (FDT) case.

This is a follow-up to r301451.

Differential Revision: https://reviews.freebsd.org/D6634

Add the GICv3 ITS intrng driver. As the interface to the interrupt
framework has significantly changed the driver has moved to a new file.
While it shares some code with the existing driver this has been modified
to work better with the intrng framework.

This has been tested on the ThunderX servers in the netperf cluster and has
been used to boot them for other testing, including DTrace and hwpmc.

With this we can use intrng on all supported arm64 platforms I was able to
test on. It is expected we will move to intrng soon, and disable the old
arm64 interrupt framework.

Obtained from: ABT Systems Ltd
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D6437

Clean up the GICv3 intrng code:
* In gic_v3_attach free the correct data on failure.
* Implement gic_v3_teardown_intr.
* Update the panic string when enabling/disabling an invalid interrupt.

Obtained from: ABT Systems Ltd
Sponsored by: The FreeBSD Foundation

Add intrng support to the GICv3 driver. It lacks ITS support so won't handle
MSI or MSI-X interrupts, however this is enought to boot FreeBSD under the
ARM Foundation Model with a GICv3 interrupt controller.

Approved by: ABT Systems Ltd
Relnotes: yes
Sponsored by: The FreeBSD Foundation

Fix GICv3 build after r299090

Obtained from: Semihalf
Sponsored by: Cavium

Support interrupts binding in GICv3 and ITS

- Add MOVI command and routine for the LPI migration
- Allow to search for the ITS device descriptor using
not only devID but also LPI number.
- Bind SPIs in the Distributor
- Don't bind its_dev to collection. Keep track of the collection
IDs for each LPI.

Reviewed by: wma
Obtained from: Semihalf
Sponsored by: Cavium
Differential Revision: https://reviews.freebsd.org/D5231

Initially bind all interrupts to the boot CPU when using GICv3

This should be done by routing all interrupts to CPU0,
different assignment will be induced by either interrupts
shuffling or bus_bind_intr().

Reviewed by: wma
Obtained from: Semihalf
Sponsored by: Cavium
Differential Revision: https://reviews.freebsd.org/D5229

Minor clean-ups for ARM64 GICv3 and GIC drivers

GICv3:
- move ICC_SGI1R_EL1 definitions to armreg.h and use proper system
register's names
GIC:
- remove unused functions

Reviewed by: andrew
Obtained from: Semihalf
Sponsored by: Cavium
Differential Revision: https://reviews.freebsd.org/D5119

Simplify GICv3 related drivers' naming

Rename gic_v3_ instances to simply use 'gic' and 'its'.
The information about the controller's revision is printed
in the device announcement during boot anyway.
The intention behind this change is to avoid somewhat misleading
GIC instances naming such as:
gic_v30
gic_v31
...
etc.

Submitted by: Zbigniew Bodek <zbb@semihalf.com>
Obtained from: Semihalf
Sponsored by: Cavium
Differential Revision: https://reviews.freebsd.org/D5016

Correct arm64 gic_v3 sizeof argument

No functional change as 'struct resource *' and 'struct resource **'
have the same size, but the former is the proper type.

PR: 204768
Submitted by: David Binderman

Add SMP support to GICv3 and ITS drivers

Introduce supprot for SMP to GICv3 and ITS drivers.

Obtained from: Semihalf
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D3299

Apply erratum for mrs ICC_IAR1_EL1 speculative execution on ThunderX

ERRATUM: 22978, 23154
PASS (rev.): 1.0/1.1

Reviewed by: imp
Obtained from: Semihalf
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D3184

Introduce ITS support for ARM64

Add ARM ITS (Interrupt Translation Services) support required
to bring-up message signalled interrupts on some ARM64 platforms.

Obtained from: Semihalf
Sponsored by: The FreeBSD Foundation

Add support for ARM GICv3 interrupt controller used in some ARM64 chips

GICv3 allows to distribute interrupts to more than 8 cores served by
the previous GIC revisions. GICv3 introduces additional logic in form
of Re-Distributors associated with particular CPUs to determine
the highest priority interrupts and manage PPIs and LPIs
(Locality-specific Peripheral Interrupts). Interrupts routing is
based on CPUs' affinity numbers. CPU interface was changed to be
accessible via CPU System Registers and this is the preferred
(and supported) method in this driver.

Obtained from: Semihalf
Reviewed by: andrew, emaste, ian, imp
Sponsored by: The FreeBSD Foundation