crashinfo: Print stack traces for all on-CPU threads

Add a python script which implements the bulk of this functionality.
Over time, this would ideally evolve into a library of python routines
which can be used to inspect kernel data structures and automate some
debugging tasks, similar to jhb's out-of-tree scripts, but written in a
somewhat nicer language and with better integration into the kgdb
command prompt.

Note that kgdb currently won't auto-load scripts in this directory.
This should perhaps change in the future. It probably also makes more
sense to have a crashinfo.py which provides all the kgdb output that we
want to include in core.txt, rather than having crashinfo.sh pipe in
several commands.

Reviewed by: avg, imp
Discussed with: jhb
MFC after: 3 weeks
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D33817

crashinfo: Get local variable values in the gdb backtrace

This should make crash reports a bit more useful without having to ask
for additional information.

Reviewed by: imp
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D42465

crashinfo: remove unused /usr/bin/gdb support

We have not had gdb 6.1 in the base system for some time; there is no
need to check for it.

Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D34478

Remove $FreeBSD$: one-line sh pattern

Remove /^\s*#[#!]?\s*\$FreeBSD\$.*$\n/

crashinfo: Add references to the gdb port/package

We intend to remove the obsolete GDB 6.1.1 from FreeBSD before FreeBSD 13.

Reviewed by jhb
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D27447

Remove nfsstat. Running nfsstat in crashinfo will give the stats of the
running kernel instead of the stats of the crashed kernel. The current
version uses sysctls to query the stats and does not work at all (anymore)
on crash dumps.

crashinfo: stop looking for gdb in /usr/bin/gdb

As of r359457 we removed the GDB_LIBEXEC option, always installing in-tree
gdb into /usr/libexec/. Thus, there is now no need for crashinfo to include
/usr/bin/gdb in the list of pathnames to check when looking for gdb.

Support compressed crash dumps in crashinfo(8).

Temporarily decompress a copy of a crash dump compressed with either
gzip or zstd and run various tools against the decompressed copy while
generating the crash information. The uncompressed copy is deleted when
the script exits.

Note that crashinfo is enabled by default, so this will attempt to
decompress the most recent compressed crash dump after a crash that
generates a compressed crash dump. Users who wish to only do offline
analysis of compressed crash dumps can disable crashinfo in rc.conf.

Tested by: ler
Reviewed by: markj
MFC after: 2 weeks

General further adoption of SPDX licensing ID tags.

Mainly focus on files that use BSD 3-Clause license.

The Software Package Data Exchange (SPDX) group provides a specification
to make it easier for automated tools to detect and summarize well known
opensource licenses. We are gradually adopting the specification, noting
that the tags are considered only advisory and do not, in any way,
superceed or replace the license texts.

Special thanks to Wind River for providing access to "The Duke of
Highlander" tool: an older (2014) run over FreeBSD tree was useful as a
starting point.

Allow matches of truncated version strings.

Long objdir paths make it easy to hit the version string length limit in
kernel dump headers. The build number and timestamp are unlikely to be
truncated and ought to be sufficient to protect against false positives.

Discussed with: jhb
MFC after: 1 week

crashinfo: add "batch" mode and use it during boot

In batch mode, most messages go into the core.txt.N file instead of stdout.

Reviewed by: jhb
MFC after: 3 days
Sponsored by: Dell EMC
Differential Revision: https://reviews.freebsd.org/D10429

Add a new GDB_LIBEXEC option to install gdb and kgdb to /usr/libexec.

When this option is enabled, only gdb and kgdb are installed to
/usr/libexec for use by crashinfo(8). Other bits of GDB such as
gdbserver and gdbtui are not installed. For this option to be
effective, GDB must be enabled.

Rework r317094 to re-enable GDB on all platforms but enable
GDB_LIBEXEC on platforms for which the GDB in ports is a superset of
functionality.

Reviewed by: emaste, kib
Suggested by: kib
Relnotes: yes
Differential Revision: https://reviews.freebsd.org/D10449

Update crashinfo to work with newer gdb from ports.

If gdb from ports is installed, use it instead of the base system gdb
to extract variables from a kernel. Note that base gdb and ports gdb
do not support the same options for invoking a single command in batch
mode, so a wrapper shell function is used. In addition, prefer kgdb
from ports when generating a backtrace if present.

PR: 193335
Reviewed by: emaste
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D7218

Remove 'netstat -anr' from crashinfo.

Do not truncate the ``command'' column in ``ps'' output.

Reviewed by: jhb
MFC after: 1 week

Remove 'netstat -i' from crashinfo, as this no longer is supported.

Allow more space for interface names.

MFC after: 1 week

Improve multi-line kernel ident parsing

Instead of blindly grabbing the line with 'Version string' and the
following one from the core info file, take all lines after 'Version
string' until the one one that matches the field format in the core info
file. This provides compatibility with VendorBSD modifications that have
a different kernel ident format.

Reviewed by: jhb

Teach crashinfo(8) to look at the currently running kernel (based on the
kern.bootfile sysctl). Prior to this commit, crashinfo(8) only looks at
/boot/*/kernel; this includes the usual places where kernels reside, so
for most systems this will have no effect.

Approved by: jhb
MFC after: 3 days

A new jail(8) with a configuration file, to replace the work currently done
by /etc/rc.d/jail.

Create the altix project branch. The altix project will add support
for the SGI Altix 350 to FreeBSD/ia64. The hardware used for porting
is a two-module system, consisting of a base compute module and a
CPU expansion module. SGI's NUMAFlex architecture can be an excellent
platform to test CPU affinity and NUMA-aware features in FreeBSD.

MFC r198846:

Set umask to 0x077 instead of the default. This prevents non-root user
from reading crashinfo output, which could contain some sensitive
information.

MFC 198586: Include the output of the ddb(4) capture buffer.

Set umask to 0x077 instead of the default. This prevents non-root user
from reading crashinfo output, which could contain some sensitive
information.

Reviewed by: jhb
MFC after: 1 week

Include the output of the ddb(4) capture buffer.

Submitted by: Mikolaj Golub to my trociny of gmail
MFC after: 3 days

- Import the HEAD csup code which is the basis for the cvsmode work.

Add a script to perform simple analysis of a crash dump (either a full
dump or minidump). When the script is run, it generates a text file
containing the output of several commands run againt the core dump such
as kgdb (stack trace), ps, netstat, vmstat, iostat, dmesg, and fstat.

Obtained from: Yahoo!
MFC after: 2 weeks