MFC r340995

Prevent kernel stack disclosure in signal delivery

On arm64 and riscv platforms, sendsig() failed to zero the signal
frame before copying it out to userspace. Zero it.

On arm, I believe all the contents of the frame were initialized,
so there was no disclosure. However, explicitly zero the whole frame
because that fact could inadvertently change in the future,
it's more clear to the reader, and I could be wrong in the first place.

Security: similar to FreeBSD-EN-18:12.mem and CVE-2018-17155
Sponsored by: Dell EMC Isilon

MFC 338813: Clear all of the VFP state in fill_fpregs().

Zero the entire FP register set structure returned for ptrace() if a
thread hasn't used FP registers rather than leaking garbage in the
fp_sr and fp_cr fields.

Revert r338695: it depends on r334032, which was not MFCed.

PR: 231064

MFC r338538:
Exclude the EFI framebuffer from phys_avail[] on arm64.

PR: 231064

Revert r331022: MFC of EFI Runtime Service support on aarch64

Apologies; this is NOT an MFC'able change. It requires a good number of pcb
changes that would break KBI.

Pointy hat to: me

MFC r324495: Support the EFI Runtime Services on arm64. As with amd64 we use
the 1:1 mapping. This uses the new common code shared with amd64.

The RTC should only be accessed via EFI. There is no locking around it as
the spec only has this as a requirement for the PC-AT CMOS device.

NOTE: r326311's changes to arm64/efirt_machdep.c have been hand-applied to
this import of efirt_machdep.c due to r326311 having already been merged.

MFC r326137:

Ensure we check the program state set in the trap frame on arm and arm64.
This value may be set by userspace so we need to check it before using it.
If this is not done correctly on exception return the kernel may continue
in kernel mode with all registers set to a userspace controlled value. Fix
this by moving the check into set_mcontext, and also add the missing
sanitisation from the arm64 set_regs.

Discussed with: security-officer@
Sponsored by: DARPA, AFRL

MFC 322437: Reliably enable debug exceptions on all CPUs.

Previously, debug exceptions were only enabled on the boot CPU if
DDB was enabled in the dbg_monitor_init() function. APs also called
this function, but since mp_machdep.c doesn't include opt_ddb.h, the
APs ended up calling an empty stub defined in <machine/debug_monitor.h>
instead of the real function. Also, if DDB was not enabled in the kernel,
the boot CPU would not enable debug exceptions.

Fix this by adding a new dbg_init() function that always clears the OS
lock to enable debug exceptions which the boot CPU and the APs call.
This function also calls dbg_monitor_init() to enable hardware breakpoints
from DDB on all CPUs if DDB is enabled. Eventually base support for
hardware breakpoints/watchpoints will need to move out of the DDB-only
debug_monitor.c for use by userland debuggers.

MFC 322436: Don't panic for PT_GETFPREGS.

Only fetch the VFP state from the CPU if the thread whose registers are
being requested is the current thread. If a stopped thread's registers
are being fetched by a debugger, the saved state in the PCB is already
valid.

MFC r322627: arm64: return error instead of panic in unimplemented ptrace ops

We don't need a panic as a reminder that these need to be implemented.

Reported by: Shawn Webb
Sponsored by: The FreeBSD Foundation

MFC r316732, r316756: Enable Privileged Access Never on arm64.

r316732:
Use the unprivileged variant of the load and store instructions most
places possible in the kernel. This forces these functions to fail if
userspace is unable to access a given memory location, even if it is in
the user memory range.

This will simplify adding Privileged Access Never support later.

r316756:
In ARMv8.1 ARM has added a process state bit to disable access to userspace
from the kernel. Make use of this to restrict accessing userspace to just
the functions that explicitly handle crossing the user kernel boundary.

MFC r303261,r315059:

r303261:
Add more UEFI/e820 memory types from latest specifications.
r315059:
Split overbloated machep.c to multiple files and do basic cleanup of these
fragments.

MFC r305942:
Consolidate four efi_next_descriptor() definitions.

MFC 305285:
Add a pc_clock pcpu field and use it to implement cpu_est_clockrate. This
will allow drivers that manage the clock frequency to communicate this with
the reset of the kernel.

Sponsored by: ABT Systems Ltd

MFC r304892:
Print both the kernel read and write translation in DDB when asking for
a virtual to physical translation. These may be different, e.g. when a
page is mapped as read-only.

Sponsored by: ABT Systems Ltd

MFC r304285:
Implement userspace gettimeofday(2) with HPET timecounter.

Copy head@r302406 to stable/11 as part of the 11.0-RELEASE cycle.
Prune svn:mergeinfo from the new branch, as nothing has been merged
here.

Additional commits post-branch will follow.

Approved by: re (implicit)
Sponsored by: The FreeBSD Foundation

Move arm's devmap to some generic place, so it can be used
by other architectures.

Reviewed by: imp
Differential Revision: https://reviews.freebsd.org/D6091
Sponsored by: DARPA, AFRL
Sponsored by: HEIF5

arm64: pagezero improvement

This change has been provided to improve pagezero call performance.

Submitted by: Dominik Ermel <der@semihalf.com>
Obtained from: Semihalf
Sponsored by: Cavium
Reviewed by: kib
Differential Revision: https://reviews.freebsd.org/D5741

arm64: bzero optimization

This optimization attempts to utylize as wide as possible register store instructions to zero large buffers.
The implementation, if possible, will use 'dc zva' to zero buffer by cache lines.

Speedup: 60x faster memory zeroing

Submitted by: Dominik Ermel <der@semihalf.com>
Obtained from: Semihalf
Sponsored by: Cavium
Reviewed by: kib
Differential Revision: https://reviews.freebsd.org/D5726

Add support for 4 level pagetables. The userland address space has been
increased to 256TiB. The kernel address space can also be increased to be
the same size, but this will be performed in a later change.

To help work with an extra level of page tables two new functions have
been added, one to file the lowest level table entry, and one to find the
block/page level. Both of these find the entry for a given pmap and virtual
address.

This has been tested with a combination of buildworld, stress2 tests, and
by using sort to consume a large amount of memory by sorting /dev/zero. No
new issues are known to be present from this change.

Reviewed by: kib
Obtained from: ABT Systems Ltd
Relnotes: yes
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D5720

Get memory ranges from FDT if no EFI API is available on ARM64

Obtained from: Semihalf
Submitted by: Michal Stanek <mst@semihalf.com>
Sponsored by: Annapurna Labs
Approved by: cognet (mentor)
Reviewed by: andrew, wma
Differential revision: https://reviews.freebsd.org/D5408

Implement single stepping on arm64. We need to set the single step bits in
the processor and debug state registers. A flag has been added to the pcb
to tell us when to enable single stepping for a given thread.

Reviewed by: kib
Sponsored by: ABT Systems Ltd
Differential Revision: https://reviews.freebsd.org/D4730

Convert ss_sp in stack_t and sigstack to void *.

POSIX requires these members to be of type void * rather than the
char * inherited from 4BSD. NetBSD and OpenBSD both changed their
fields to void * back in 1998. No new build failures were reported
via an exp-run.

PR: 206503 (exp-run)
Reviewed by: kib
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D5092

Stop including fdt_common.h in the arm64 code. We don't use anything from
it, however may have relied on header pollution to pull in the needed
headers through it

Sponsored by: ABT Systems Ltd

Make the 'env' directive described in config(5) work on all architectures,
providing compiled-in static environment data that is used instead of any
data passed in from a boot loader.

Previously 'env' worked only on i386 and arm xscale systems, because it
required the MD startup code to examine the global envmode variable and
decide whether to use static_env or an environment obtained from the boot
loader, and set the global kern_envp accordingly. Most startup code wasn't
doing so. Making things even more complex, some mips startup code uses an
alternate scheme that involves calling init_static_kenv() to pass an empty
buffer and its size, then uses a series of kern_setenv() calls to populate
that buffer.

Now all MD startup code calls init_static_kenv(), and that routine provides
a single point where envmode is checked and the decision is made whether to
use the compiled-in static_kenv or the values provided by the MD code.

The routine also continues to serve its original purpose for mips; if a
non-zero buffer size is passed the routine installs the empty buffer ready
to accept kern_setenv() values. Now if the size is zero, the provided buffer
full of existing env data is installed. A NULL pointer can be passed if the
boot loader provides no env data; this allows the static env to be installed
if envmode is set to do so.

Most of the work here is a near-mechanical change to call the init function
instead of directly setting kern_envp. A notable exception is in xen/pv.c;
that code was originally installing a buffer full of preformatted env data
along with its non-zero size (like mips code does), which would have allowed
kern_setenv() calls to wipe out the preformatted data. Now it passes a zero
for the size so that the buffer of data it installs is treated as
non-writeable.

Add support for usermode (vdso-like) gettimeofday(2) and
clock_gettime(2) on ARMv7 and ARMv8 systems which have architectural
generic timer hardware. It is similar how the RDTSC timer is used in
userspace on x86.

Fix a permission problem where generic timer access from EL0 (or
userspace on v7) was not properly initialized on APs.

For ARMv7, mark the stack non-executable. The shared page is added for
all arms (including ARMv8 64bit), and the signal trampoline code is
moved to the page.

Reviewed by: andrew
Discussed with: emaste, mmel
Sponsored by: The FreeBSD Foundation
Differential revision: https://reviews.freebsd.org/D4209

Correctly align the stack. The early csu assumed we passed the aux vector
in through the stack pointer, however this may have been misaligned
causing some userland applications to crash. A workaround was committed in
r284707 where userland would check if the aux vector was passed using the
old or new ABI and adjust the stack if needed. As 4 months have passed it
is time to move to the new ABI, with the expectation the compat code in csu
and the runtime linker to be removed in the future.

Sponsored by: ABT Systems Ltd

Don't read the floating-point registers for now. We will need to enable the
VFP around the read of these instructions as they may raise an exception.

Sponsored by: ABT Systems Ltd

Add ddb show commands to print the special registers and to ask the
hardware to perform address translation for us. These are useful to help
track down what caused us to enter the debugger.

Sponsored by: ABT Systems Ltd

Fill in dump_avail based on the physical memory from EFI.

Obtained from: ABT Systems Ltd
Sponsored by: The FreeBSD Foundation

Pass the pcb to store the vfp state in to vfp_save_state. This fixes a bug
in savectx where it will be used to store the current state however will
pass in a pcb when vfp_save_state expected a thread pointer.

Obtained from: ABT Systems Ltd
Sponsored by: The FreeBSD Foundation

Try to put the CPU into a low power state if we failed to otherwise halt
the system.

Obtained from: ABT Systems Ltd
Sponsored by: The FreeBSD Foundation

Add ARM64TODO markers to unimplemented functionality

Reviewed by: andrew
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D2389

Clear the carry bit on the saved program state register when asked to
clear the return value, it's used to indicate an error.

Obtained from: ABT Systems Ltd
Sponsored by: The FreeBSD Foundation

Add support for SMP. This uses the FDT data to find the CPUs to start on,
and psci to start them. I expect ACPI support to be added later.

This has been tested on qemu with 2 cpus as that is the current value of
MAXCPUS. This is expected to be increased in the future as FreeBSD has
been tested on 48 cores on the Cavium ThunderX hardware.

Partially based on a patch from Robin Randhawa from ARM.

Approved by: ABT Systems Ltd
Relnotes: yes
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D3024

Remove sv_sigtbl handling from the arm64 sendsig(). There is no ABI
emulators on arm64.

Reviewed by: andrew
Review: https://reviews.freebsd.org/D2889
Sponsored by: The FreeBSD Foundation

Add basic support for ACPI. It splits out the nexus driver to two new
drivers, one for fdt, one for acpi. It then uses this to decide if it will
use fdt or acpi.

The GICv2 (interrupt controller) and Generic Timer drivers have been
updated to handle both cases.

As this is early code we still need FDT to find the kernel console, and
some parts are still missing, including PCI support.

Differential Revision: https://reviews.freebsd.org/D2463
Reviewed by: jhb, jkim, emaste
Obtained from: ABT Systems Ltd
Relnotes: Yes
Sponsored by: The FreeBSD Foundation

Bring in the start of the arm64 kernel.

This is only the minimum set of files needed to boot in qemu. As such it is
missing a few things.

The bus_dma code is currently only stub functions with a full implementation
from the development tree to follow.

The gic driver has been copied as the interrupt framework is different. It
is expected the two drivers will be merged by the arm intrng project,
however this will need to be imported into the tree and support for arm64
would need to be added.

This includes code developed by myself, SemiHalf, Ed Maste, and Robin
Randhawa from ARM. This has been funded by the FreeBSD Foundation, with
early development by myself in my spare time with assistance from Robin.

Differential Revision: https://reviews.freebsd.org/D2199
Reviewed by: emaste, imp
Relnotes: yes
Sponsored by: The FreeBSD Foundation