kmsan: Add some additional bus_space accessors

These are needed for arm64 support.

Co-authored-by: Alexander Stetsenko <alex.stetsenko@klarasystems.com>
Sponsored by: Klara, Inc.
Sponsored by: Juniper Networks, Inc.

kmsan: Add kmsan_check_uio()

This was handy for some ad-hoc debugging and fits in with other
kmsan_check_*() routines which operate on some kind of data container.

MFC after: 1 week
Sponsored by: The FreeBSD Foundation

kmsan: Correct the origin address in kmsan_shadow_check()

Otherwise a KMSAN report (which panics the system by default) could
trigger a recursive panic.

MFC after: 1 week
Fixes: ca6cd604c8fc ("kmsan: Use the correct origin bytes in kmsan_check_arg()")

kmsan: Use __builtin_memset to initialize per-thread state

Accesses to KMSAN's TLS block are not instrumented, so there's no need
to use kmsan_memset(). No functional change intended.

MFC after: 1 week
Sponsored by: Klara, Inc.
Sponsored by: Juniper Networks, Inc.

kmsan: Add corresponding sysctl knob for loader tunable

The loader tunable 'debug.kmsan.disabled' does not have corresponding
sysctl MIB entry. Add it so that it can be retrieved, and `sysctl -T`
will also report it correctly.

Reviewed by: markj
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D42138

sys: Remove $FreeBSD$: one-line .c pattern

Remove /^[\s*]*__FBSDID\("\$FreeBSD\$"\);?\s*\n/

kmsan: Use the correct origin bytes in kmsan_check_arg()

Upon discovering a violation kmsan_check_arg() passes a pointer to
function parameter shadow state to kmsan_report_hook().
kmsan_report_hook() uses that address to find the origin cells, assuming
that the passed address belongs to the kernel map. This has two
problems:
1) Function parameter origin state is also located in TLS, not in the
origin map, but kmsan_report_hook() doesn't know this.
2) KMSAN TLS for thread0 is statically allocated and thus isn't shadowed
(because the kernel itself is not shadowed).

These bugs could result in inaccuracies in KMSAN reports, or a page
fault when trying to report a KMSAN violation (which by default panics
the kernel anyway).

Fix the problem by making callers of kmsan_report_hook() provide a
pointer to origin cells.

Sponsored by: The FreeBSD Foundation

memdesc: Retire MEMDESC_BIO.

Instead, change memdesc_bio to examine the bio and return a memdesc of
a more generic type describing the data buffer.

Reviewed by: imp
Sponsored by: Chelsio Communications
Differential Revision: https://reviews.freebsd.org/D41029

memdesc: Replace md_opaque with a union of type-specific fields.

Reviewed by: imp, markj
Sponsored by: Chelsio Communications
Differential Revision: https://reviews.freebsd.org/D41027

memdesc: Retire MEMDESC_CCB.

Instead, change memdesc_ccb to examine the CCB and return a memdesc of
a more generic type describing the data buffer.

Reviewed by: imp, markj
Sponsored by: Chelsio Communications
Differential Revision: https://reviews.freebsd.org/D40880

atomic: Intercept atomic_(load|store)_bool for kernel sanitizers

Fixes: 2bed73739aac ("atomic: Add plain atomic_load/store_bool()")

Use KERNEL_PANICKED() in more places

This is slightly more optimized than checking panicstr directly. For
most of these instances performance doesn't matter, but let's make
KERNEL_PANICKED() the common idiom.

Reviewed by: mjg
MFC after: 3 days
Differential Revision: https://reviews.freebsd.org/D35373

amd64: Deduplicate routines for expanding KASAN/KMSAN shadow maps

When working on the ports these functions were slightly different, but
now there's no reason for them to be separate.

No functional change intended.

MFC after: 1 week
Sponsored by: The FreeBSD Foundation

kmsan: Export kmsan_mark_mbuf() and kmsan_mark_bio()

Sponsored by: The FreeBSD Foundation

Add the KMSAN runtime

KMSAN enables the use of LLVM's MemorySanitizer in the kernel. This
enables precise detection of uses of uninitialized memory. As with
KASAN, this feature has substantial runtime overhead and is intended to
be used as part of some automated testing regime.

The runtime maintains a pair of shadow maps. One is used to track the
state of memory in the kernel map at bit-granularity: a bit in the
kernel map is initialized when the corresponding shadow bit is clear,
and is uninitialized otherwise. The second shadow map stores
information about the origin of uninitialized regions of the kernel map,
simplifying debugging.

KMSAN relies on being able to intercept certain functions which cannot
be instrumented by the compiler. KMSAN thus implements interceptors
which manually update shadow state and in some cases explicitly check
for uninitialized bytes. For instance, all calls to copyout() are
subject to such checks.

The runtime exports several functions which can be used to verify the
shadow map for a given buffer. Helpers provide the same functionality
for a few structures commonly used for I/O, such as CAM CCBs, BIOs and
mbufs. These are handy when debugging a KMSAN report whose
proximate and root causes are far away from each other.

Obtained from: NetBSD
Sponsored by: The FreeBSD Foundation