usr.sbin: Automated cleanup of cdefs and other formatting

Apply the following automated changes to try to eliminate
no-longer-needed sys/cdefs.h includes as well as now-empty
blank lines in a row.

Remove /^#if.*\n#endif.*\n#include\s+<sys/cdefs.h>.*\n/
Remove /\n+#include\s+<sys/cdefs.h>.*\n+#if.*\n#endif.*\n+/
Remove /\n+#if.*\n#endif.*\n+/
Remove /^#if.*\n#endif.*\n/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/types.h>/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/param.h>/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/capsicum.h>/

Sponsored by: Netflix

Remove $FreeBSD$: one-line .c pattern

Remove /^[\s*]*__FBSDID\("\$FreeBSD\$"\);?\s*\n/

pmcstat: Fix typos

Signed-off-by: Elyes Haouas <ehaouas@noos.fr>
Reviewed by: imp
Pull Request: https://github.com/freebsd/freebsd-src/pull/653

pmc: Rework PROCEXEC event to support PIEs

Currently the PROCEXEC event only reports a single address, entryaddr,
which is the entry point of the interpreter in the typical dynamic case,
and used solely to calculate the base address of the interpreter. For
PDEs this is fine, since the base address is known from the program
headers, but for PIEs the base address varies at run time based on where
the kernel chooses to load it, and so pmcstat has no way of knowing the
real address ranges for the executable. This was less of an issue in the
past since PIEs were rare, but now they're on by default on 64-bit
architectures it's more of a problem.

To solve this, pass through what was picked for et_dyn_addr by the
kernel, and use that as the offset for the executable's start address
just as is done for everything in the kernel. Since we're changing this
interface, sanitise the way we determine the interpreter's base address
by passing it through directly rather than indirectly via the entry
point and having to subtract off whatever the ELF header's e_entry is
(and anything that wants the entry point in future can still add that
back on as needed; this merely changes the interface to directly provide
the underlying variables involved).

This will be followed up by a bump to the pmc major version.

Reviewed by: jhb
Differential Revision: https://reviews.freebsd.org/D39595

spdx: The BSD-2-Clause-FreeBSD identifier is obsolete, drop -FreeBSD

The SPDX folks have obsoleted the BSD-2-Clause-FreeBSD identifier. Catch
up to that fact and revert to their recommended match of BSD-2-Clause.

Discussed with: pfg
MFC After: 3 days
Sponsored by: Netflix

Add the missing pmcstat log entries

Sponsored by: Innovate UK

pmcstat: implement showing offsets into symbols in top mode

The -I option (and hotkey) is reused for this. Skipping symbol resolution is
moved to the new -A option (and hotkey).

While arguably this violates POLA I think it's a change for the better.
ALso note the -I option was added in head.

Differential Revision: https://reviews.freebsd.org/D21658

pmc(3)/hwpmc(4): update supported Intel processors to rely fully on the
vendor provided pmu-events tables and sundry cleanups.

The vendor pmu-events tables provide counter descriptions, default
sample rates, event, umask, and flag values for all the counter
configuration permutations. Using this gives us:

- much simpler kernel code for the MD component
- helpful long and short event descriptions
- simpler user code
- sample rates that won't overload the system

Update man page with newer sample types and remove unused sample type.

Revert r334242 "pmc(3)/hwpmc(4): update supported Intel processors to rely fully on the"
because of squash commit messages

pmc(3)/hwpmc(4): update supported Intel processors to rely fully on the
vendor provided pmu-events tables and sundry cleanups.

The vendor pmu-events tables provide counter descriptions, default
sample rates, event, umask, and flag values for all the counter
configuration permutations. Using this gives us:

- much simpler kernel code for the MD component
- helpful long and short event descriptions
- simpler user code
- sample rates that won't overload the system

Update man page with newer sample types and remove unused sample type.

Squashed commit of the following:

commit 4459d43eff815bec08ccc5533dbe5de846f03128
Author: Matt Macy <mmacy@mattmacy.io>
Date: Sat May 26 00:06:31 2018 -0700

libpmc: fix pmu function signatures for non amd64

commit a2cb8bbc586c65d41f9b291430a2261ec67b59fe
Author: Matt Macy <mmacy@mattmacy.io>
Date: Fri May 25 22:38:11 2018 -0700

pmcstat: fix indentation of usage

commit f686954b15ff56a833ac80404898977cb80a265b
Author: Matt Macy <mmacy@mattmacy.io>
Date: Fri May 25 22:19:49 2018 -0700

pmclog(3): add callchain and pmcallocatedyn, remove pcsample

commit 73e13a0d2e9498c81c150d14d022050cee7511bb
Author: Matt Macy <mmacy@mattmacy.io>
Date: Fri May 25 22:19:00 2018 -0700

pmclog.h: GC pcsample field

commit 3e93ffd65da641fa657539dad3c48e281f8b5798
Author: Matt Macy <mmacy@mattmacy.io>
Date: Fri May 25 22:05:57 2018 -0700

hwpmc: make Intel core CPUs use external event tables

commit 634f5fae1e1644ac324003136c66cd9c619d1c93
Author: Matt Macy <mmacy@mattmacy.io>
Date: Fri May 25 22:00:06 2018 -0700

pmclog: update log record types, bump PMC_MAJOR
- explicitly make log record types a multiple of 8 bytes
- hook in pmu event types for pmc_allocate records
- remove references to no longer PCSAMPLE record

commit 83d84fcd2d65bdf6ddcb2e155a22f0cfa2a9c225
Author: Matt Macy <mmacy@mattmacy.io>
Date: Fri May 25 21:52:10 2018 -0700

libpmc: add support for having vendor table driven pmc_allocate

commit 9e6ad63c40c2fce8404847ace5078ca6cb33a736
Author: Matt Macy <mmacy@mattmacy.io>
Date: Fri May 25 19:11:33 2018 -0700

hwpmc_core: add accessors for EVSEL & UMASK, make IAP_UMASK useful to user

commit 859dceb93daa6419a48c794db99b6758e5b041c9
Author: Matt Macy <mmacy@mattmacy.io>
Date: Fri May 25 19:09:45 2018 -0700

pmcstat: update usage and man page as well as make -L consistent with pmccontrol

commit 79c7d8597e28c2eb13f5f9113e65ec2792ca57b1
Author: Matt Macy <mmacy@mattmacy.io>
Date: Fri May 25 18:07:03 2018 -0700

pmu_util: add support for all current intel event keywords

commit d8089c7f6a6c8527f38324252b1ffb47004694c6
Author: Matt Macy <mmacy@mattmacy.io>
Date: Fri May 25 17:45:00 2018 -0700

add description for new arguments

commit 058336740bab53c62ec88a3a026ea848cf3878c6
Author: Matt Macy <mmacy@mattmacy.io>
Date: Fri May 25 17:38:15 2018 -0700

libpmc: move pmu_events table and pmu_utils out of libpmcstat so that they can be used by pmc_allocate

commit 049b66b382e2f833c3f47bc8df9e750cb265709f
Author: Matt Macy <mmacy@mattmacy.io>
Date: Fri May 25 16:12:41 2018 -0700

pmcstat: hook pmu_events counter description utility routines in

commit f5e01e7b37a691dc045e1aa16b3ebdd162515de8
Author: Matt Macy <mmacy@mattmacy.io>
Date: Fri May 25 16:11:59 2018 -0700

pmu_events: add utility routines for listing counters and their descriptions

commit cba4d4f8907f772279f86f18f915e0d74d33ac56
Author: Matt Macy <mmacy@mattmacy.io>
Date: Fri May 25 16:09:50 2018 -0700

pmu-events: expand out skylake regex to simplify string matches

pmc: bump PMC major version to reflect ABI breakage and make warning not require verbose

various: general adoption of SPDX licensing ID tags.

Mainly focus on files that use BSD 2-Clause license, however the tool I
was using misidentified many licenses so this was mostly a manual - error
prone - task.

The Software Package Data Exchange (SPDX) group provides a specification
to make it easier for automated tools to detect and summarize well known
opensource licenses. We are gradually adopting the specification, noting
that the tags are considered only advisory and do not, in any way,
superceed or replace the license texts.

No functional change intended.

Extract a set of pmcstat functions and interfaces to the new internal
library -- libpmcstat.

This includes PMC logging module, symbols lookup functions,
ELF parsing, process management, PMC attachment, etc.

This allows to reuse code while building new hwpmc(4)-based applications.

Also add pmcstat_symbol_search_by_name() function that allows to find
mapped IP range for a given function name.

Reviewed by: kib
Sponsored by: DARPA, AFRL
Differential Revision: https://reviews.freebsd.org/D12718

pmcstat(8); unsign some allocation variables and use reallocarray(3).

Use unsigned values in some internal variables that will be used during
allocation. The variables are used in reduced scope and have no chance of
becoming negative.

Provide bounds checking through reallocarray(3).

MFC after: 2 weeks

pmcstat: minor spelling fixes.

Mostly comments bur also a user-visible string.

MFC after: 2 weeks

Fix for r288176 changes related to debug symbols move.

Reviewed by: emaste
Sponsored by: Stormshield

Fix pmcstat symbol resolution for userland processes.

When examining existing processes pmcstat fails to
correctly determine the locations of executable sections
of the process due to a miscalculated virtual load address.
This does not affect the newly launched processes as the
same value passed as a "start address" to the pmcstat_image_link()
thus nullifying the effect of it. The issue manifests itself
in processes not being reported in the pmcstat(8) output and
"dubious frames" being reported.

Fix it for now by ignoring all the sections except the executable
one. This won't fix the issue for objects with multiple
executable sections but helps in majority of real world usecases.
The real solution would be to modify the MAP-IN event to include
the appropriate load address so pmcstat(8) won't have to manually
parse object files to try to determine it.

PR: 198147, 198148
Reviewed by: jhb, rpaulo
MFC after: 2 weeks

Fix the double space in comment.

While here, change the comment style to match the rest of the file.

Speed up pmcstat by improving string hash

In one case generating callgraph output from a 24MB system-wide sampling
data file took 17.4 seconds on average. Profiling showed pmcstat
spending a lot of time in strcmp, due to hash collisions.

Replacing the XOR-only hash with FNV-1a reduces the run time for my
test by 40%.

Add a new option - 'a <file>' - which spits out annotated callgraphs.

'-m <file>' spits out the given stream into <file> (eg, /dev/stdout).
However, it only resolves the first symbol; it doesn't parse the entire
callgraph. If it fails to lookup then it doesn't print anything.

'-a' instead does a symbol and file:line lookup for each address in each
callgraph and will happily print the address itself with no lookup
information if it couldn't look things up.

This makes it much easier to pull out individual records from a
pmc data file and look at the callgraph information without having to
hand-decode the addresses.

Sponsored by: Netflix, Inc.

More -Wmissing-variable-declarations fixes.

In addition to adding `static' where possible:

- bin/date: Move `retval' into extern.h to make it visible to date.c.
- bin/ed: Move globally used variables into ed.h.
- sbin/camcontrol: Move `verbose' into camcontrol.h and fix shadow warnings.
- usr.bin/calendar: Remove unneeded variables.
- usr.bin/chat: Make `line' local instead of global.
- usr.bin/elfdump: Comment out unneeded function.
- usr.bin/rlogin: Use _Noreturn instead of __dead2.
- usr.bin/tset: Pull `Ospeed' into extern.h.
- usr.sbin/mfiutil: Put global variables in mfiutil.h.
- usr.sbin/pkg: Remove unused `os_corres'.
- usr.sbin/quotaon, usr.sbin/repquota: Remove unused `qfname'.

Remove spurious ARM symbols from lookup table.

MFC after: 3 days

Don't crash trying to load symbols from striped file.

PR: bin/167361
Submitted by: Slawa Olhovchenkov <slw zxy.spb.ru>
Silence from: jkoshy

Add software PMC support.

New kernel events can be added at various location for sampling or counting.
This will for example allow easy system profiling whatever the processor is
with known tools like pmcstat(8).

Simultaneous usage of software PMC and hardware PMC is possible, for example
looking at the lock acquire failure, page fault while sampling on
instructions.

Sponsored by: NETASQ
MFC after: 1 month

- Support inlined location in calltree output.
In case of multiple level of inlining all the locations are flattened.
Require recent binutils/addr2line (head works or binutils from ports
with the right $PATH order).
- Multiple fixes in the calltree output (recursion case, ...)
- Fix the calltree top view that previously hide some shared nodes.

Tested with Kcachegrind(kdesdk4)/qcachegrind(head).

Sponsored by: NETASQ

Fix base vaddr detection for ELF binaries. PT_LOAD with offset 0 is not
mandatory for ELF binaries so we'll use the segment with offset less then
alignment and align it appropriately (which covers pt_offset == 0 case)

Spelling fixes for usr.sbin/

KNF

Improve the chances of matching an outputted string with the line of code.

Add a flush of the current PMC log buffer before displaying the next top.

As the underlying block is 4KB if the PMC throughput is low the measurement
will be reported on the next tick. pmcstat(8) use the modified flush API to
reclaim current buffer before displaying next top.

MFC after: 1 month

Convert pmcstat about using cpuset_t rather than relying on plain 32 bit
ints. That fixes a first bug where pmcstat wasn't using the old
cpumask_t interface and now also brings the full support for more
than 32 cpus.

While here, make the functions pmcstat_clone_event_descriptor() and
pmcstat_get_cpumask() private to pmcstat.

The problem of assuming cpu dense masks still persists and should be
eventually fixed, as reported by avg.

Tested by: pluknet
Reviewed by: gnn
Approved by: re (kib)

A new jail(8) with a configuration file, to replace the work currently done
by /etc/rc.d/jail.

When an asm location cannot be resolved to a function the cost
will be spread as small value and then filtered by the threshold.
As a first step solution display the number of event that cannot
be resolved as a valid function location.

MFC after: 1week

- Do not use the runtime mask when logfile is specified.
- Revert the fix on rtld path that is not necessary.

MFC after: 1 week

Allow file as a top source, it works with socket now.
This will allow top monitoring using socket/ssh tunnelling
of system without local symbols.

client:
pmcstat -R <ip>:<port> -T -r <symbolspath>
monitored device:
pmcstat -Sinstructions -O <ip>:<port>

- Move the file read in the event loop
- Initialize and clean log in all cases
- Preserve global stats value during top refresh
- Fix the rtld/line resolver that ignore '-r' prefix
- Support socket for '-R' (server mode)
- Display the statistics when exiting top mode

Fix warnings found by Coverity.

Found with: Coverity Prevent(tm)
MFC after: 1 month

MFC r207731:
Exclude undefined symbol from ELF file when doing function resolve.

Exclude undefined symbol from ELF file when doing function resolve.

MFC after: 3 days

Move fatal error at the right place.
Fix exit from top mode when checking if PMC is available.

MFC after: 3 days

MFC r206090:
Improve "top" header by:
- Display sample received per PMCs (or merged PMCs).
- Display percentage vs all samples

Improve "top" header by:
- Display sample received per PMCs (or merged PMCs).
- Display percentage vs all samples

MFC r205809:
Wait for pmc name in the log before displaying data.
This will solve an abort in case of low throughput PMCs.

Wait for pmc name in the log before displaying data.
This will solve an abort in case of low throughput PMCs.

MFC after: 3days

MFC r204878:
Change the way shutdown is handled for log file.

pmc_flush_logfile is now non-blocking and just ask the kernel
to shutdown the file. From that point, no more data is
accepted by the log thread and when the last buffer is flushed
the file is closed.

This will remove a deadlock between pmcstat asking for
flush while it cannot flush the pipe itself.

Create the altix project branch. The altix project will add support
for the SGI Altix 350 to FreeBSD/ia64. The hardware used for porting
is a two-module system, consisting of a base compute module and a
CPU expansion module. SGI's NUMAFlex architecture can be an excellent
platform to test CPU affinity and NUMA-aware features in FreeBSD.

Change the way shutdown is handled for log file.

pmc_flush_logfile is now non-blocking and just ask the kernel
to shutdown the file. From that point, no more data is
accepted by the log thread and when the last buffer is flushed
the file is closed.

This will remove a deadlock between pmcstat asking for
flush while it cannot flush the pipe itself.

MFC after: 3 days

MFC 203790:
- Reorganize code in 'plugin' to share log processing.
- Kcachegrind (calltree) support with assembly/source
code mapping and call count estimator (-F).
- Top mode for calltree and callgraph plugin (-T).

- Reorganize code in 'plugin' to share log processing.
- Kcachegrind (calltree) support with assembly/source
code mapping and call count estimator (-F).
- Top mode for calltree and callgraph plugin (-T).

MFC after: 1 month

MFC r201145 to stable/8:
(S)LIST_HEAD_INITIALIZER takes a (S)LIST_HEAD as an argument.
Fix some wrong usages.
Note: this does not affect generated binaries as this argument is not used.

PR: 137213
Submitted by: Eygene Ryabinkin (initial version)

(S)LIST_HEAD_INITIALIZER takes a (S)LIST_HEAD as an argument.
Fix some wrong usages.
Note: this does not affect generated binaries as this argument is not used.

PR: 137213
Submitted by: Eygene Ryabinkin (initial version)
MFC after: 1 month

Add the -m option to pmcstat.
This option prints the list of sampled PCs along with the function name,
the start and end addresses of this where their live within.

Reviewed by: jkoshy
Tested by: gnn
Sponsored by: Nokia

- Import the HEAD csup code which is the basis for the cvsmode work.

Introduce pmcstat(8) changes for summarizing hwpmc(4) callchain records in
in textual form and in gmon.out format.

Update manual page.

Sponsored by: FreeBSD Foundation and Google Inc.

When printing the contents of a HWPMC log in human readable form:
- Print a warning if the version number recorded in the log is not what the
tool expects.
- Print a tidier error message when an unrecognized event is encountered
in the log.
- Don't print a spurious 'Unknown error: 0' when exiting after a parse
error.

When printing a map, print the number of samples recorded by each gmon.out file.

MFC after: 1 week

- Teach pmcstat(8) to log over the network; the -O option now
takes a host:port specification.
- Update the manual page and add an example showing how log
over the network using pmcstat(8) and nc(1). Document the
current inability to process logs in cross-platform manner.
- Have pmcstat_open_log() call err(3) directly in case
of an error; this simplifies error handling in its caller.

MFC after: 1 week

MFP4: Support for profiling dynamically loaded objects.

Kernel changes:

Inform hwpmc of executable objects brought into the system by
kldload() and mmap(), and of their removal by kldunload() and
munmap(). A helper function linker_hwpmc_list_objects() has been
added to "sys/kern/kern_linker.c" and is used by hwpmc to retrieve
the list of currently loaded kernel modules.

The unused `MAPPINGCHANGE' event has been deprecated in favour
of separate `MAP_IN' and `MAP_OUT' events; this change reduces
space wastage in the log.

Bump the hwpmc's ABI version to "2.0.00". Teach hwpmc(4) to
handle the map change callbacks.

Change the default per-cpu sample buffer size to hold
32 samples (up from 16).

Increment __FreeBSD_version.

libpmc(3) changes:

Update libpmc(3) to deal with the new events in the log file; bring
the pmclog(3) manual page in sync with the code.

pmcstat(8) changes:

Introduce new options to pmcstat(8): "-r" (root fs path), "-M"
(mapfile name), "-q"/"-v" (verbosity control). Option "-k" now
takes a kernel directory as its argument but will also work with
the older invocation syntax.

Rework string handling in pmcstat(8) to use an opaque type for
interned strings. Clean up ELF parsing code and add support for
tracking dynamic object mappings reported by a v2.0.00 hwpmc(4).

Report statistics at the end of a log conversion run depending
on the requested verbosity level.

Reviewed by: jhb, dds (kernel parts of an earlier patch)
Tested by: gallatin (earlier patch)

When generating profiles (-g) warn about shared objects that were
not found.

Handle 32 bit executables better on AMD64 platforms.

Determine where the runtime loader got mapped to when a dynamically
linked process was exec()'ed, and map samples in that address range
into the appropriate buckets in the runtime loader's profile.
Improve a few comments.

MFC after: 3 days

Fix a bug that caused generated gmon.out profiles to be out of
whack. Use saturating arithmetic when incrementing a bucket.
Improve a few comments.

MFC after: 3 days

Fix a bug that affected on-the-fly gmon.out file generation.

Approved by: re (scottl)

Unbreak tinderbox builds. Remove a debug printf.

Approved by: re (blanket hwpmc)

MFP4:

- pmcstat(8) gprof output mode fixes:

lib/libpmc/pmclog.{c,h}, sys/sys/pmclog.h:
+ Add a 'is_usermode' field to the PMCLOG_PCSAMPLE event
+ Add an 'entryaddr' field to the PMCLOG_PROCEXEC event,
so that pmcstat(8) can determine where the runtime loader
/libexec/ld-elf.so.1 is getting loaded.

sys/kern/kern_exec.c:
+ Use a local struct to group the entry address of the image being
exec()'ed and the process credential changed flag to the exec
handling hook inside hwpmc(4).

usr.sbin/pmcstat/*:
+ Support "-k kernelpath", "-D sampledir".
+ Implement the ELF bits of 'gmon.out' profile generation in a new
file "pmcstat_log.c". Move all log related functions to this
file.
+ Move local definitions and prototypes to "pmcstat.h"

- Other bug fixes:
+ lib/libpmc/pmclog.c: correctly handle EOF in pmclog_read().
+ sys/dev/hwpmc_mod.c: unconditionally log a PROCEXIT event to all
attached PMCs when a process exits.
+ sys/sys/pmc.h: correct a function prototype.
+ Improve usage checks in pmcstat(8).

Approved by: re (blanket hwpmc)