intrng: switch flag arguments to unsigned

The flag variables behind these are all unsigned. As such adjust the
declarations to match reality and reduce the number of mismatches.

Reviewed by: imp
Pull Request: https://github.com/freebsd/freebsd-src/pull/1126

arm64: Make KMSAN aware of exceptions

- Call kmsan_intr_enter() when an exception occurs. This ensures that
code running in the exception context does not clobber thread-local
KMSAN state.
- Ensure that stack memory containing trap frames is treated as
initialized.

Co-authored-by: Alexander Stetsenko <alex.stetsenko@klarasystems.com>
Reviewed by: imp
MFC after: 2 weeks
Sponsored by: Klara, Inc.
Sponsored by: Juniper Networks, Inc.
Differential Revision: https://reviews.freebsd.org/D43155

intrng: Allow alternative IPI PICs to be registered and used

On RISC-V, the root PIC (whether the PLIC or, as will be the case in
future, the local interrupt controller) cannot send IPIs, relying on
another means to trigger the necessary software interrupts (firmware
calls), but there are upcoming standard devices that will be able to
inject them, so we can't just put the firmware calls in the root PIC
driver.

Thus, split out a new intr_ipi_dev from intr_irq_root_dev to use for
sending IPIs. New devices can be registered with a given priority up
until the first IPI is set up, when the best device seen so far gets
frozen as the IPI device to use.

Reviewed by: mhorne
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D35899

intrng: Extract arm/arm64 IPI->PIC glue code

The arm and arm64 implementations of dispatching IPIs via PIC_IPI_SEND
are almost identical, and entirely MI with the lone exception of a
single store barrier on arm64 (that is likely either redundant or needed
on arm too). Thus, de-duplicate this code by moving it to INTRNG as a
generic IPI glue framework. The ipi_* functions remain declared in MD
smp.h headers and implemented in MD code, but are trivial wrappers
around intr_ipi_send that could be made MI, at least for INTRNG ports,
at a later date.

Note that, whilst both arm and arm64 had an ii_send member in intr_ipi
to abstract over how to send interrupts,, they were always ultimately
using PIC_IPI_SEND, and so this complexity has been removed. A follow-up
commit will re-introduce the same flexibility by instead allowing a
device other than the root PIC to be registered as the IPI sender.

As part of this, strengthen a MAXCPU assertion that was missed in commit
2f0b059eeafc ("intrng: switch from MAXCPU to mp_ncpus") (which itself is
mis-titled).

Reviewed by: mmel, mhorne
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D35898

intrng: Remove irq_root_ipicount and corresponding intr_pic_claim_root arg

The static irq_root_ipicount variable is only ever written to (with the
value passed to irq_root_ipicount), never read. Moreover, the bcm2836
driver, as used by the Raspberry Pi 2B and 3A/B (but not 4, which uses a
GIC-400, though does have the legacy interrupt controller present too)
passes 0 as ipicount, despite implementing IPIs. It's thus inaccurate
and serves no purpose, so should be removed.

Reviewed by: mmel, imp, mhorne
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D35897

intrng: switch from MAXCPU to mp_ncpus

MAXCPU could be on the large side, while the hardware may not have
many processors at all. As such only allocate counters for processors
which actually exist, rather than always allocating for the maximum
potentially allowed number.

Reviewed by: markj, emaste
Differential Revision: https://reviews.freebsd.org/D41462

sys: Remove $FreeBSD$: one-line .c pattern

Remove /^[\s*]*__FBSDID\("\$FreeBSD\$"\);?\s*\n/

Continue searching for an irq map from the start

When searching for a free irq map location continue the search from the
beginning of the list. There may be holes in the map before
irq_map_first_free_idx, e.g. when removing an entries in order will
increase the index past the current free entry.

PR: 271990
Reviewed by: mhorne
Sponsored by: Arm Ltd

Differential Revision: https://reviews.freebsd.org/D40768

Hide irq_next_free, it's not used out of this file

Reviewed by: mhorne
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D40767

arm64: add KASAN support

This entails:
- Marking some obvious candidates for __nosanitizeaddress
- Similar trap frame markings as amd64, for similar reasons
- Shadow map implementation

The shadow map implementation is roughly similar to what was done on
amd64, with some exceptions. Attempting to use available space at
preinit_map_va + PMAP_PREINIT_MAPPING_SIZE (up to the end of that range,
as depicted in the physmap) results in odd failures, so we instead
search the physmap for free regions that we can carve out, fragmenting
the shadow map as necessary to try and fit as much as we need for the
initial kernel map. pmap_bootstrap_san() is thus after
pmap_bootstrap(), which still included some technically reserved areas
of the memory map that needed to be included in the DMAP.

The odd failure noted above may be a bug, but I haven't investigated it
all that much.

Initial work by mhorne with additional fixes from kevans and markj.

Reviewed by: andrew, markj
Sponsored by: Juniper Networks, Inc.
Sponsored by: Klara, Inc.
Differential Revision: https://reviews.freebsd.org/D36701

intrng: track counter allocation with a bitmap

Crucially, this allows releasing counters, and interrupt sources by
extension. Where before we were incrementing intrcnt_index with atomics,
now we protect the bitmap using the existing isrc_table_lock mutex.

Reviewed by: mmel
MFC after: 2 weeks
Differential Revision: https://reviews.freebsd.org/D38437

intrng: sort includes

MFC after: 3 days

ddb: annotate some commands with DB_CMD_MEMSAFE

This is not completely exhaustive, but covers a large majority of
commands in the tree.

Reviewed by: markj
Sponsored by: Juniper Networks, Inc.
Sponsored by: Klara, Inc.
Differential Revision: https://reviews.freebsd.org/D35583

intrng: Use less confusing return value for intr_pic_add_handler

Currently intr_pic_add_handler either returns the PIC you gave it (which
is useless and risks causing confusion about whether it's creating
another PIC) or, on error, NULL. Instead, convert it to return an int
error code as one would expect.

Note that the only consumer of this API, arm64's gicv3_its, does not use
the return value, so no uses need updating to work with the revised API.

Reviewed by: markj, mmel
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D33341

intrng: remove now redundant shadow variable.

Should not be a functional change.

Submitted by: ehem_freebsd@m5p.com
Discussed in: https://reviews.freebsd.org/D29310
MFC after: 4 weeks

intrng: Releasing interrupt source should clear interrupt table full state.

The first release of an interrupt in a situation where the interrupt table
is full should schedule a full table check the next time an interrupt is
allocated. A full check is necessary to ensure maximum separation between
the order of allocation and the order of release.

Submitted by: ehem_freebsd@m5p.com (initial version)
Discussed in: https://reviews.freebsd.org/D29310
MFC after: 4 weeks

kern/intr: declare interrupt vectors unsigned

These should never get values large enough for sign to matter, but one
of them becoming negative could cause problems.

MFC after: 1 week
Reviewed by: markj
Differential Revision: https://reviews.freebsd.org/D29327

make maximum interrupt number tunable on ARM, ARM64, MIPS, and RISC-V

Use a machdep.nirq tunable intead of compile-time constant NIRQ
as a value for maximum number of interrupts. It allows keep a system
footprint small by default with an option to increase the limit
for large systems like server-grade ARM64

Reviewd by: mhorne
Differential Revision: https://reviews.freebsd.org/D27844
Submitted by: Klara, Inc.
Sponsored by: Ampere Computing

Also pass interrupt binding request to non-root interrupt controllers.
There are message based controllers that can bind interrupts even if they are
not implemented as root controllers (such as the ITS subblock of GIC).

MFC after: 3 weeks

Add a method to determine whether given interrupt is per CPU or not.

MFC after: 2 weeks

o Add iommu de-initialization method for MSI interface.
o Add iommu_unmap_msi() to release the msi GAS entry.
o Provide default implementations for iommu init/deinit methods.

Reviewed by: kib
Sponsored by: Innovate DSbD
Differential Revision: https://reviews.freebsd.org/D26906

Manage MSI iommu pages.

This allows the interrupt controller driver only need a small change to
create a map for the page the device will write to raise an interrupt.

Submitted by: andrew
Reviewed by: kib
Sponsored by: Innovate DSbD
Differential Revision: https://reviews.freebsd.org/D26705

kern: clean up empty lines in .c and .h files

INTRNG: only shuffle for !EARLY_AP_STARTUP

During device attachment, all interrupt sources will bind to the BSP,
as it is the only processor online. This means interrupts must be
redistributed ("shuffled") later, during SI_SUB_SMP.

For the EARLY_AP_STARTUP case, this is no longer true. SI_SUB_SMP will
execute much earlier, meaning APs will be online and available before
devices begin attachment, and there will therefore be nothing to
shuffle.

All PIC-conforming interrupt controllers will handle this early
distribution properly, except for RISC-V's PLIC. Make the necessary
tweak to the PLIC driver.

While here, convert irq_assign_cpu from a boolean_t to a bool.

Reviewed by: markj
Differential Revision: https://reviews.freebsd.org/D25693

Simplify the flow when getting/setting an isrc

Rather than unlocking and returning we can just perform the needed action
only when the interrupt source is valid and reuse the unlock in both the
valid irq and invalid irq cases.

Sponsored by: Innovate UK

Fix arm64 kernel build with DEBUG on

Submitted by: Greg V <greg@unrelenting.technology>, andrew
Differential Revision: https://reviews.freebsd.org/D24986

Remove duplicated empty lines from kern/*.c

No functional changes.

Extract eventfilter declarations to sys/_eventfilter.h

This allows replacing "sys/eventfilter.h" includes with "sys/_eventfilter.h"
in other header files (e.g., sys/{bus,conf,cpu}.h) and reduces header
pollution substantially.

EVENTHANDLER_DECLARE and EVENTHANDLER_LIST_DECLAREs were moved out of .c
files into appropriate headers (e.g., sys/proc.h, powernv/opal.h).

As a side effect of reduced header pollution, many .c files and headers no
longer contain needed definitions. The remainder of the patch addresses
adding appropriate includes to fix those files.

LOCK_DEBUG and LOCK_FILE_LINE_ARG are moved to sys/_lock.h, as required by
sys/mutex.h since r326106 (but silently protected by header pollution prior
to this change).

No functional change (intended). Of course, any out of tree modules that
relied on header pollution for sys/eventhandler.h, sys/lock.h, or
sys/mutex.h inclusion need to be fixed. __FreeBSD_version has been bumped.

Fix compile error with option DEBUG. This is fallout from some long-ago
INTRNG refactoring that didn't get caught at the time because code in a
debugf() statement isn't compiled unless DEBUG is defined.

PR: 221557

Only return the current cpu if it's in the cpumask. When we restrict the
cpumask it probably means we are unable to sent interrupts to CPUs outside
the map. As such only return the current CPU when it's within the mask
otherwise return the first valid CPU.

This is needed on ThunderX as, in a dual socket configuration, we are
unable to send MSI/MSI-X interrupts between sockets.

Reviewed by: mmel
Sponsored by: DARPA, AFRL
Differential Revision: https://reviews.freebsd.org/D11957

- Remove 'struct vmmeter' from 'struct pcpu', leaving only global vmmeter
in place. To do per-cpu stats, convert all fields that previously were
maintained in the vmmeters that sit in pcpus to counter(9).
- Since some vmmeter stats may be touched at very early stages of boot,
before we have set up UMA and we can do counter_u64_alloc(), provide an
early counter mechanism:
o Leave one spare uint64_t in struct pcpu, named pc_early_dummy_counter.
o Point counter(9) fields of vmmeter to pcpu[0].pc_early_dummy_counter,
so that at early stages of boot, before counters are allocated we already
point to a counter that can be safely written to.
o For sparc64 that required a whole dummy pcpu[MAXCPU] array.

Further related changes:
- Don't include vmmeter.h into pcpu.h.
- vm.stats.vm.v_swappgsout and vm.stats.vm.v_swappgsin changed to 64-bit,
to match kernel representation.
- struct vmmeter hidden under _KERNEL, and only vmstat(1) is an exclusion.

This is based on benno@'s 4-year old patch:
https://lists.freebsd.org/pipermail/freebsd-arch/2013-July/014471.html

Reviewed by: kib, gallatin, marius, lidl
Differential Revision: https://reviews.freebsd.org/D10156

All these files need sys/vmmeter.h, but now they got it implicitly
included via sys/pcpu.h.

Only allow the pic type to be either a PIC or MSI type. All interrupt
controller drivers handle either MSI/MSI-X interrupts, or regular
interrupts, as such enforce this in the interrupt handling framework.
If a later driver was to handle both it would need to create one of each.

This will allow future changes to allow the xref space to overlap, but
refer to different drivers.

Obtained from: ABT Systems Ltd
Sponsored by: The FreeBSD Foundation
X-Differential Revision: https://reviews.freebsd.org/D8616

INTRNG - fix MSI/MSIX release path

Use isrc in attached MSI data structure instead of using map's
isrc directly. map's isrc is set to NULL on IRQ deactivation
which happens prior to pci_release_msi so MSI_RELEASE_MSI
receives array of NULLs

Reviewed by: mmel
Differential Revision: https://reviews.freebsd.org/D8206

INTRNG: Rework handling with resources. Partially revert r301453.
- Read interrupt properties at bus enumeration time and store
it into global mapping table.
- At bus_activate_resource() time, given mapping entry is resolved and
connected to real interrupt source. A copy of mapping entry is attached
to given resource.
- At bus_setup_intr() time, mapping entry stored in resource is used
for delivery of requested interrupt configuration.
- For MSI/MSIX interrupts, mapping entry is created within
pci_alloc_msi()/pci_alloc_msix() call.
- For legacy PCI interrupts, mapping entry must be created within
pcib_route_interrupt() by pcib driver itself.

Reviewed by: nwhitehorn, andrew
Differential Revision: https://reviews.freebsd.org/D7493

Remove temporary solution for storing interrupt mapping data as
it's not needed after r301451 and follow-ups r301453, r301539.

This makes INTRNG clean of all additions related to various buses.

INTRNG: As follow up of r301451, implement mapping and configuration
of gpio pin interrupts by new way.

Note: This removes last consumer of intr_ddata machinery and we remove it
in separate commit.

INTRNG - change the way how an interrupt mapping data are provided
to the framework in OFW (FDT) case.

This is a follow-up to r301451.

Differential Revision: https://reviews.freebsd.org/D6634

(1) Add a new bus method to get a mapping data for an interrupt.

BUS_MAP_INTR() is used to get an interrupt mapping data according
to provided hints. The hints could be modified afterwards, but only
if mapping data was allocated. This method is intended to be called
before BUS_ALLOC_RESOURCE().

An interrupt mapping data describes an interrupt - hardware number,
type, configuration, cpu binding, and whatever is needed to setup it.

(2) Introduce a method which allows storing of an additional data
in struct resource to be available for bus drivers. This method is
convenient in two ways:
- there is no need to rework existing bus drivers as they can simply
be extended to provide an additional data,
- there is no need to modify any existing bus methods as struct
resource is already passed to them as argument and thus stored data
is simply accessible by other bus drivers.
For now, implement this method only for INTRNG.

This is motivated by needs of modern SOCs where hardware initialization
is not straightforward and resources descriptions are complex, opaque
for everyone but provider, and may vary from SOC to SOC. Typical
situation is that one bus driver can fetch a resource description for
its child device, but it's opaque for this driver. Another bus driver
knows a provider for this kind of resource and can pass this resource
description to it. In fact, something like device IVARS would be
perfect for that if implemented generally enough. Unfortunatelly, IVARS
are usable only by their owners now. Only owner knows its IVARS layout,
thus other bus drivers are not able to use them.

Differential Revision: https://reviews.freebsd.org/D6632

Add an interface to handle interrupt controllers that have a contiguous
range of interrupts they pass to a second controller driver to handle.
The parent driver is expected to detect when one of these interrupts has
been triggered and call intr_child_irq_handler to pass the interrupt to
a child. The children controllers are then expected to manage the range
by allocating interrupts as needed.

This will initially be used by the ARM GICv3 driver, but is is expected to
be useful for other driver where this type of allocation applies.

Obtained from: ABT Systems Ltd
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D6436

Limit calling pmc_hook to when the interrupt comes while running userspace.
We may enable interrupts from within the callback, e.g. in a data abort
during copyin. If we receive an interrupt at that time pmc_hook will be
called again and, as it is handling userspace stack tracing, will hit a
KASSERT as it checks if the trapframe is from userland.

With this I can run hwpmc with intrng on a ThunderX and have it trace all
CPUs.

Obtained from: ABT Systems Ltd
Sponsored by: The FreeBSD Foundation

Add the needed hwpmc hooks to subr_intr.c. This is needed for the correct
operation of hwpmc on, for example, arm64 with intrng.

Obtained from: ABT Systems Ltd
Sponsored by: The FreeBSD Foundation

Return the struct intr_pic pointer from intr_pic_register. This will be
needed in later changes where we may not be able to lock the pic list lock
to perform a lookup, e.g. from within interrupt context.

Obtained from: ABT Systems Ltd
Sponsored by: The FreeBSD Foundation

Introduce MSI and MSI-X support to intrng. This adds a new msi device
interface with 5 methods to mirror the 5 MSI/MSI-X methods in the pcib
interface. The pcib driver will need to perform a device specific lookup
to find the MSI controller and pass this to intrng as the xref. Intrng
will finally find the controller and have it handle the requested operation.

Obtained from: ABT Systems Ltd
MFH: yes
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D5985

Check malloc succeeded in pic_create, with M_NOWAIT it may return NULL.

Obtained from: ABT Systems Ltd
Sponsored by: The FreeBSD Foundation

Set correct size to the size member of struct intr_map_data when
initialized. As the size member is not used at the present,
it did not break anything.

INTRNG - redefine struct intr_map_data to avoid headers pollution. Each
struct associated with some type defined in enum intr_map_data_type
must have struct intr_map_data on the top of its own definition now.
When such structs are used, correct type and size must be filled in.

There are three such structs defined in sys/intr.h now. Their
definitions should be moved to corresponding headers by follow-up
commits.

While this change was propagated to all INTRNG like PICs,
pic_map_intr() method implementations were corrected on some places.
For this specific method, it's ensured by a caller that the 'data'
argument passed to this method is never NULL. Also, the return error
values were standardized there.

Remove superfluous check. The pic_dev member of struct pic
is never NULL on PIC found by pic_lookup().

sys/kern: spelling fixes in comments.

No functional change.

INTRNG: Define 'INTR_IRQ_INVALID' constant and use it consistently
as error indicator.

GPIO: Add support for gpio pin interrupts.
Add new function gpio_alloc_intr_resource(), which allows an allocation
of interrupt resource associated to given gpio pin. It also allows to
specify interrupt configuration.

Note: This functionality is dependent on INTRNG, and must be
implemented in each GPIO controller.

Fix intr_irq_shuffle(). After r297539, ISRCs doing IPI may be also
registered into global interrupt table. Thus, they must be filtered out
like per-cpu interrupts. Fortunately, it does not influence anything
on interrupt controllers which already use INTRNG.

Implement intr_isrc_init_on_cpu() and use it to replace very same
code implemented in every interrupt controller driver running SMP.
This function returns true, if provided ISRC should be enabled on
given cpu.

Fix PIC lookup by device and xref. There was not taken into account
the situation that someone has a pointer to device but not its xref.
This situation is regular now, after r297539.

Include sys/rman.h directly rather than relying on header pollution.

Obtained from: ABT Systems Ltd
Sponsored by: The FreeBSD Foundation

Remove FDT specific parts from INTRNG. Change its interface to make it
universal.

(1) New struct intr_map_data is defined as a container for arbitrary
description of an interrupt used by a device. Typically, an interrupt
number and configuration relevant to an interrupt controller is encoded
in such description. However, any additional information may be encoded
too like a set of cpus on which an interrupt should be enabled or vendor
specific data needed for setup of an interrupt in controller. The struct
intr_map_data itself is meant to be opaque for INTRNG.

(2) An intr_map_irq() function is created which takes an interrupt
controller identification and struct intr_map_data as arguments and
returns global interrupt number which identifies an interrupt.

(3) A set of functions to be used by bus drivers is created as well as
a corresponding set of methods for interrupt controller drivers. These
sets take both struct resource and struct intr_map_data as one of the
arguments. There is a goal to keep struct intr_map_data in struct
resource, however, this way a final solution is not limited to that.

(4) Other small changes are done to reflect new situation.

This is only first step aiming to create stable interface for interrupt
controller drivers. Thus, some temporary solution is taken. Interrupt
descriptions for devices are stored in INTRNG and two specific mapping
function are created to be temporary used by bus drivers. That's why
the struct intr_map_data is not opaque for INTRNG now. This temporary
solution will be replaced by final one in next step.

Differential Revision: https://reviews.freebsd.org/D5730

Generalize IPI support for ARM intrng and use it for interrupt
controller IPI provider.

New struct intr_ipi is defined which keeps all info about an IPI:
its name, counter, send and dispatch methods. Generic intr_ipi_setup(),
intr_ipi_send() and intr_ipi_dispatch() functions are implemented.

An IPI provider must implement two functions:
(1) an intr_ipi_send_t function which is able to send an IPI,
(2) a setup function which initializes itself for an IPI and
calls intr_ipi_setup() with appropriate arguments.

Differential Revision: https://reviews.freebsd.org/D5700

Remove an alternative way for dealing with root interrupt controller
which is not complete. Likely, it was forgotten and not removed before
committing.

Mark other parts of interrupt framework as INTR_SOLO option specific.

Note that isrc_arg member of struct intr_irqsrc is used only for
INTR_SOLO and IPI filter. This should be remembered if IPI filters
and their arguments will be stored on another place.

This option could be unusable very soon, if interrupt controllers
implementations will not be implemented considering it.

Move IPI related parts back to (ARM) machine specific file now, when
the interrupt framework is also going to be used by another (MIPS)
architecture. IPI implementations may vary much across different
architectures.

An IPI implementation should still define INTR_IPI_COUNT and use
intr_ipi_setup_counters() to setup IPI counters which are inside of
intrcnt[] and intrnames[] arrays. Those are used for sysctl and ddb.
Then, intr_ipi_increment_count() should be used to increment obtained
counter.

Reviewed by: imp
Differential Revision: https://reviews.freebsd.org/D5459

Remove an unused FDT header, fdt_common.h should only be needed in a few
places, mostly in sys/dev/fdt and legacy code.

Sponsored by: ABT Systems Ltd

Allow MIPS INTRNG code to be built without FDT support.

This patch allows the newly imported INTRNG code to be built without necessarily
having FDT support in the kernel. This may be useful for some MIPS platforms
that wish to move to INTRNG, but not to FDT at the same time.

Basically all the code is already within ifdef's where FDT is concerned,
it's just the headers that aren't.

Submitted by: Stanislav Galabov <sgalabov@gmail.com>
Differential Revision: https://reviews.freebsd.org/D5249

[intrng] Migrate the intrng code from sys/arm/arm to sys/kern/subr_intr.c.

The ci20 port (by kan@) is going to reuse almost all of the intrng code
since the SoC in question looks suspiciously like someone took an ARM
SoC design and replaced the ARM core with a MIPS core.

* migrate out the code;
* rename ARM_ -> INTR_;
* rename arm_ -> intr_;
* move the interrupt flush routine from intr.c / intrng.c into
arm/machdep_intr.c - removing the code duplication and removing
the ARM specific bits from here.

Thanks to the Star Wars: The Force Awakens premiere line for allowing
me a couple hours of quiet time to finish the universe builds.

Tested:

* make universe

TODO:

* The structure definitions in subr_intr.c still includes machine/intr.h
which requires one duplicates all of the intrng definitions in
the platform code (which kan has done, and I think we don't have to.)

Instead I should break out the generic things (function declarations,
common intr structures, etc) into a separate header.

* Kan has requested I make the PIC based IPI stuff optional.