dtrace: Add the 'oformat' libdtrace option

This option can be used to specify a format to use in DTrace output.
The following formats are supported:
- json
- xml
- html
- none (default DTrace output)

This is implemented using libxo and integrated into libdtrace. Client
code only works with the following API:

- dtrace_oformat_setup(dtrace_hdl_t *) -- to be called when output is starting.
- dtrace_oformat_teardown(dtrace_hdl_t *) -- to be called when output is finished
- dtrace_oformat(dtrace_hdl_t *) -- check if oformat is enabled.
- dtrace_set_outfp(FILE *) -- sets the output file for oformat.
- Ensure that oformat is correctly checked in the drop handler and record
processing callbacks.

This commit also adds tests which check if the generated output is
valid (JSON, XML) and extends the dtrace(1) describing the structured output.

Reviewed by: markj
Discussed with: phil
MFC after: 2 months
Sponsored by: Innovate UK
Differential Revision: https://reviews.freebsd.org/D41745

Remove $FreeBSD$: one-line sh pattern

Remove /^\s*#[#!]?\s*\$FreeBSD\$.*$\n/

dtrace: Add WITH_DTRACE_ASAN

See commit 4ae6991228105eb34989c870194ae7b0a1e23be4. This version of
the commit avoids inadvertently changing SHLIBDIR for libdtrace.so.

dtrace: Revert the addition of WITH_DTRACE_ASAN

The follow-up fix triggers a lib32 build failure, revert everything
until the problem is addressed.

libdtrace: Explicitly set SHLIBDIR and SHLIB_MAJOR

They were previously being defined by cddl/lib/Makefile.inc, and as of
commit 4ae699122810 were being overridden by defaults in bsd.own.mk,
which changed SHLIBDIR to /usr/lib.

Reported by: Domagoj Stolfa <ds815@cam.ac.uk>
Fixes: 4ae699122810 ("dtrace: Add WITH_DTRACE_ASAN")

dtrace: Add WITH_DTRACE_ASAN

This option is a blanket for all the DTrace-related software. The option
when enabled passes in -fsanitize=address -fsanitize=undeifned, enabling
ASAN and UBSAN in the following components:

- libdtrace
- dtrace(1)
- lockstat(1)
- plockstat(1)

The option defaults to "no" and is intended as a developer aid.

Reviewed by: markj
Differential Revision: https://reviews.freebsd.org/D41157

dtrace: add register bindings for arm64

Reviewed by: mhorne, markj
Approved by: markj (mentor)
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D39956

dtrace: implement dtrace_instr_size() for arm64

Reviewed by: markj
Approved by; markj (mentor)
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D39955

dtrace: add register bindings for RISC-V

Reviewed by: mhorne, markj
MFC after: 1 week
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D39611

dtrace: expose dtrace_instr_size() to userland and implement it for riscv

dtrace_instr_size() is needed by the forthcoming RISC-V port of kinst,
as well as by libdtrace in D38825 for both amd64 and RISC-V.

Reviewed by: markj, mhorne
MFC after: 1 week
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D39489

libdtrace: Add kinst support

kinst does not instantiate its probes automatically, it only does so on
demand via an ioctl interface implemented by /dev/kinst. This change
modifies libdtrace to perform that work when the script references the
kinst provider, similar to the way pid provider probes are implemented.

Reviewed by: markj
MFC after: 3 months
Sponsored by: Google, Inc. (GSoC 2022)
Differential Revision: https://reviews.freebsd.org/D36852

dtrace: Remove local mips support

Remove the stub pid probe and all the build glue.

Reviewed by: imp, jhb
Sponsored by: DARPA, AFRL
Differential Revision: https://reviews.freebsd.org/D35541

ctf: Import ctf.h from OpenBSD

Use it instead of the existing ctf.h from OpenSolaris. This makes it
easier to use CTF in the core kernel, and to extend the CTF format to
support wider type IDs.

The imported ctf.h is modified to depend only on _types.h, and also to
provide macros which use the "parent" bit of a type ID to refer to types
in a parent CTF container.

No functional change intended.

Reviewed by: Domagoj Stolfa, emaste
MFC after: 1 month
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D34358

pkgbase: Put dtrace in its own package

While dtrace is usefull some people might not want it.

Differential Revision: https://reviews.freebsd.org/D30752
Sponsored by: Diablotin Systems

libdtrace: Stop relying on lex compatibility

It does not appear to be required, and as of commit 6b7e592c215f
("lex: Do not let input() return 0 when end-of-file is reached") it
causes input to return 0 instead of EOF when end-of-input is reached.

PR: 253440
MFC after: 3 days
Sponsored by: The FreeBSD Foundation

Install dtrace.h and dependencies

This makes the minimum amount of changes to allow inclusion of dtrace.h
without all the solaris compatibility headers. Installing dtrace.h allows
compiling consumers of libdtrace (e.g. https://github.com/tmetsch/python-dtrace)
without requiring a copy of the source tree.
For python-dtrace I worked around this in https://github.com/tmetsch/python-dtrace/commit/58019c9a12022203a9ffda286dd8b41f1a5ace42
but being able to build the library without installed sources would be
extremely useful.

Reviewed By: gnn
Differential Revision: https://reviews.freebsd.org/D27884

Remove -I flag for include path that doesn't exist

Found this while trying to get macOS bootstrap to work again after OpenZFS merge.

Reviewed By: #zfs, freqlabs
Differential Revision: https://reviews.freebsd.org/D26192

Merge OpenZFS support in to HEAD.

The primary benefit is maintaining a completely shared
code base with the community allowing FreeBSD to receive
new features sooner and with less effort.

I would advise against doing 'zpool upgrade'
or creating indispensable pools using new
features until this change has had a month+
to soak.

Work on merging FreeBSD support in to what was
at the time "ZFS on Linux" began in August 2018.
I first publicly proposed transitioning FreeBSD
to (new) OpenZFS on December 18th, 2018. FreeBSD
support in OpenZFS was finally completed in December
2019. A CFT for downstreaming OpenZFS support in
to FreeBSD was first issued on July 8th. All issues
that were reported have been addressed or, for
a couple of less critical matters there are
pull requests in progress with OpenZFS. iXsystems
has tested and dogfooded extensively internally.
The TrueNAS 12 release is based on OpenZFS with
some additional features that have not yet made
it upstream.

Improvements include:
project quotas, encrypted datasets,
allocation classes, vectorized raidz,
vectorized checksums, various command line
improvements, zstd compression.

Thanks to those who have helped along the way:
Ryan Moeller, Allan Jude, Zack Welch, and many
others.

Sponsored by: iXsystems, Inc.
Differential Revision: https://reviews.freebsd.org/D25872

- Cleanups related to sparc64 removal.
- Remove remains of sparc64 files.

Reviewed by: imp
Differential Revision: https://reviews.freebsd.org/D25831

Add support for send, receive and state-change DTrace providers for
SCTP. They are based on what is specified in the Solaris DTrace manual
for Solaris 11.4.

Reviewed by: 0mp, dteske, markj
Relnotes: yes
Differential Revision: https://reviews.freebsd.org/D16839

Add a dtrace provider for UDP-Lite.

The dtrace provider for UDP-Lite is modeled after the UDP provider.
This fixes the bug that UDP-Lite packets were triggering the UDP
provider.
Thanks to dteske@ for providing the dwatch module.

Reviewed by: dteske@, markj@, rrs@
Relnotes: yes
Differential Revision: https://reviews.freebsd.org/D16377

Add D definitions for the named values in socket.h

Reviewed by: markj
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D12241

cddl: normalize paths using SRCTOP-relative paths or :H when possible

This simplifies make logic/output

While here, remove bogus CFLAGS which look for headers in cddl/lib/libumem.
There aren't any source files there (just Makefiles)

MFC after: 1 month
Sponsored by: Dell EMC Isilon

Define dependencies for some auto-generated source files in libdtrace.

Remove an unneeded beforedepend rule.

Reported by: emaste, jhb
Reviewed by: bdrewery
MFC after: 1 week

Add initial DTrace support for RISC-V.

Sponsored by: DARPA, AFRL
Sponsored by: HEIF5

Use FILES for installing the dtrace scripts.

The BSD.usr.dist mtree always creates /usr/lib/dtrace so there is no
need to check if it exists.

The FILES mechanism already supports LIBRARIES_ONLY.

Sponsored by: EMC / Isilon Storage Division

First cut of DTrace for AArch64.

Reviewed by: andrew, emaste
Sponsored by: ARM Limited
Differential Revision: https://reviews.freebsd.org/D2738

Correctly link libdtrace and convert to LIBADD
Make dtrace only link to libdtrace

Move siftr-related definitions out of tcp.d and into a separate library
which declares a dependency on siftr(4). This is necessitated by a
reference to struct pkt_node, which is defined in siftr(4): otherwise,
dtrace(1) will return an error during startup if siftr.ko is not loaded.

Initial version of DTrace on ARM32.

Submitted by: Howard Su based on work by Oleksandr Tymoshenko
Reviewed by: ian, andrew, rpaulo, markj

Move some files that are identical on i386 and amd64 to an x86 subdirectory
rather than keeping duplicate copies.

Discussed with: avg
MFC after: 1 week

Implement the ip, tcp, and udp DTrace providers. The probe definitions use
dynamic translation so that their arguments match the definitions for
these providers in Solaris and illumos. Thus, existing scripts for these
providers should work unmodified on FreeBSD.

Tested by: gnn, hiren
MFC after: 1 month

DTrace: Revert r249367

The following change from illumos brought caused DTrace to
pause in an interactive environment:

3026 libdtrace should set LD_NOLAZYLOAD=1 to help the pid provider

This was not detected during testing because it doesn't
affect scripts.

We shouldn't be changing the environment, especially since the
LD_NOLAZYLOAD option doesn't apply to our (GNU) ld.
Unfortunately the change from upstream was made in such a way
that it is very difficult to separate this change from the
others so, at least for now, it's better to just revert
everything.

Reference:
https://www.illumos.org/issues/3026

Reported by: Navdeep Parhar and Mark Johnston

Add a new LIBRARIES_ONLY make variable to disable the build and install
of files other than the actual libraries.

Use LIBRARIES_ONLY to supress the inclusion of files in the lib32
distribution that are duplicates of files in base.

Sponsored by: DARPA, AFRL
Reviewed by: emaste

Implement DTrace for PowerPC. This includes both 32-bit and 64-bit.

There is one known issue: Some probes will display an error message along the
lines of: "Invalid address (0)"

I tested this with both a simple dtrace probe and dtruss on a few different
binaries on 32-bit. I only compiled 64-bit, did not run it, but I don't expect
problems without the modules loaded. Volunteers are welcome.

MFC after: 1 month

Initial commit of an I/O provider for DTrace on FreeBSD.

These probes are most useful when looking into the structures
they provide, which are listed in io.d. For example:

dtrace -n 'io:genunix::start { printf("%d\n", args[0]->bio_bcount); }'

Note that the I/O systems in FreeBSD and Solaris/Illumos are sufficiently
different that there is not a 1:1 mapping from scripts that work
with one to the other.
MFC after: 1 month

Enable build of DTrace-related userland parts for MIPS

A new jail(8) with a configuration file, to replace the work currently done
by /etc/rc.d/jail.

MFtbemd:

Prefer MACHNE_CPUARCH to MACHINE_ARCH in most contexts where you want
to test of all the CPUs of a given family conform.

Add libdtrace support for tracing userland programs.

Summary of changes:
* Implement a compatibility shim between Solaris libproc and our
libproc and remove several ifdefs because of this.
* Port the drti to FreeBSD.
* Implement the missing DOODAD sections
* Link with libproc and librtld_db
* Support for ustack, jstack and uregs (by sson@)
* Misc bugfixing

When writing the SUWN_dof section, we had to resort to building the ELF
file layout by "hand". This is the job of libelf, but our libelf doesn't
support this yet. When libelf is fixed, we can remove the code under
#ifdef BROKEN_LIBELF.

Sponsored by: The FreeBSD Foundation

Create the altix project branch. The altix project will add support
for the SGI Altix 350 to FreeBSD/ia64. The hardware used for porting
is a two-module system, consisting of a base compute module and a
CPU expansion module. SGI's NUMAFlex architecture can be an excellent
platform to test CPU affinity and NUMA-aware features in FreeBSD.

Remove manual .includes in cddl Makefiles

- Break the dependency on ../Makefile.inc for .PATH, and include
../Makefile.inc implicitly. This is required to ...
- Set WARNS?=6 in top-level Makefile.inc
- Remove now redundant WARNS settings, add WARNS?=0 where appropriate
- Remove redundant SHLIB_MAJOR overrides
- Use NO_MAN, not MK_MAN=no
- Remove redundant inclusion of bsd.own.mk
- Order Makefiles more according to style.Makefile(9)
- Reduce diff of cddl Makefiles against each other

No objection: pjd
Approved by: ed (co-mentor)

Bump the version of all non-symbol-versioned shared libraries in
preparation for 8.0-RELEASE. Add the previous version of those
libraries to ObsoleteFiles.inc and bump __FreeBSD_Version.

Reviewed by: kib
Approved by: re (rwatson)

- Import the HEAD csup code which is the basis for the cvsmode work.

Add the DTrace libraries and D library scripts.