MFC r325671:
Check that the pmc index is less than the number of hardware PMCs,
instead of asserting the condition.

MFC r322256:
Fix logic error in the the assert, causing the condition to be always true.

PR: 217741
Approved by: re (gjb)

MFC r311224: Fix PMC architecture check to handle later IPAs including Skylake

MFC r292070: More fixes in the various intel processors, fixing missing
IAP_F_FM's...

MFC r291494: Add support for Intel Skylake and Intel Broadwell PMC's.

MFC r281101: Remove whitespace [in hwpmc]

MFC r279939: hwpmc: Fix event number to match enum name

MFC r279836: Add missing counter definitions

MFC r279835: Fix Ivy Bridge+ MEM_UOPS_RETIRED counters

MFC r279834: Support architectural events on Haswell/Ivy Bridge

MFC r279832: Fix Sandy Bridge+ hwpmc branch counters

MFC r279831: Fix pmc unit restrictions to match documentation

MFC r279830: Fix various bugs in Haswell counter definitions

MFC r277524: style(9) cleanup in hwpmc

MFC 281102: hwpmc: add initial Intel Broadwell support.

MFC r308480: pmc_process_csw_out: ignore deleted counters

MFC r308101: hwpmc: fix a race between amd_stop_pmc and amd_intr

MFH: r259647

o Remove assertions on ipa_version as sometimes the version detection
using cpuid can be quirky (this is the case of VMWare without the
vPMC support) but fail to probe hwpmc.
o Apply the fix for XEON family of processors as established by
315338-020 document (bug AJ85).

Sponsored by: EMC / Isilon storage division
Reviewed by: fabient
MFC courtesy of panzura.

MFC 303720: Apply the fix from r232612 to fixed function counters.

MFC r298931, r298981, r299375:

Minor spelling fixes in:
sys/dev, sys/sys

Many of these have user-visible strings.

MFC r295352:
Do not call vn_fullpath(9) (through the pmc_getfilename() wrapper)
when its result is immediately ignored, i.e. for kernel processes
forked from the user process. Do not test for non-null before freeing
string.

MFC r290811:
Fix hwpmc "stalled" behavior

Currently, there is a single pm_stalled flag that tracks whether a
performance monitor was "stalled" due to insufficent ring buffer
space for samples. However, because the same performance monitor
can run on multiple processes or threads at the same time, a single
pm_stalled flag that impacts them all seems insufficient.

In particular, you can hit corner cases where the code fails to stop
performance monitors during a context switch out, because it thinks
the performance monitor is already stopped. However, in reality,
it may be that only the monitor running on a different CPU was stalled.

This patch attempts to fix that behavior by tracking on a per-CPU basis
whether a PM desires to run and whether it is "stalled". This lets the
code make better decisions about when to stop PMs and when to try to
restart them. Ideally, we should avoid the case where the code fails
to stop a PM during a context switch out.

MFC r290813:
Optimizations to the way hwpmc gathers user callchains

Changes to the code to gather user stacks:
* Delay setting pmc_cpumask until we actually have the stack.
* When recording user stack traces, only walk the portion of the ring
that should have samples for us.

MFC r290929:
Change the driver stats to what they really are: unsigned values.

When pmcstat exits after some samples were dropped, give the user an
idea of how many were lost. (Granted, these are global numbers, but
they may still help quantify the scope of the loss.)

MFC r290930:
Improve accuracy of PMC sampling frequency

The code tracks a counter which is the number of events until the next
sample. On context switch in, it loads the saved counter. On context
switch out, it tries to calculate a new saved counter.

Problems:

1. The saved counter was shared by all threads in a process. However, this
means that all threads would be initially loaded with the same saved
counter. However, that could result in sampling more often than once every
X number of events.

2. The calculation to determine a new saved counter was backwards. It
added when it should have subtracted, and subtracted when it should have
added. Assume a single-threaded process with a reload count of 1000
events. Assuming the counter on context switch in was 100 and the counter
on context switch out was 50 (meaning the thread has "consumed" 50 more
events), the code would calculate a new saved counter of 150 (instead of
the proper 50).

Fix:

1. As soon as the saved counter is used to initialize a monitor for a
thread on context switch in, set the saved counter to the reload count.
That way, subsequent threads to use the saved counter will get the full
reload count, assuring we sample at least once every X number of events
(across all threads).

2. Change the calculation of the saved counter. Due to the change to the
saved counter in #1, we simply need to add (modulo the reload count) the
remaining counter time we retrieve from the CPU when a thread is context
switched out.

MFC r291016:
Support a wider history counter in pmcstat(8) gmon output

pmcstat(8) contains an option to output sampling data in a gmon format
compatible with gprof(1). Currently, it uses the default histcounter,
which is an (unsigned short). With large sets of sampling data, it
is possible to overflow the maximum value provided by an (unsigned
short).

This change adds the -e argument to pmcstat. If -e and -g are both
specified, pmcstat will use a histcounter type of uint64_t.

MFC r291017:
Fix the date on the pmcstat(8) man page from r291016.

MFC r283924

Provide vnode in memory map info for files on tmpfs

When providing memory map information to userland, populate the vnode pointer
for tmpfs files. Set the memory mapping to appear as a vnode type, to match
FreeBSD 9 behavior.

This fixes the use of tmpfs files with the dtrace pid provider,
procstat -v, procfs, linprocfs, pmc (pmcstat), and ptrace (PT_VM_ENTRY).

Submitted by: Eric Badger <eric@badgerio.us> (initial revision)
Obtained from: Dell Inc.
PR: 198431

MFC 283121:
Use the proper mask when reloading sampling PMCs for Core CPUs.

MFC 283123:
Fix two bugs that could result in PMC sampling effectively stopping.
In both cases, the the effect of the bug was that a very small positive
number was written to the counter. This means that a large number of
events needed to occur before the next sampling interrupt would trigger.
Even with very frequently occurring events like clock cycles wrapping all
the way around could take a long time. Both bugs occurred when updating
the saved reload count for an outgoing thread on a context switch.

First, the counter-independent code compares the current reload count
against the count set when the thread switched in and generates a delta
to apply to the saved count. If this delta causes the reload counter
to go negative, it would add a full reload interval to wrap it around to
a positive value. The fix is to add the full reload interval if the
resulting counter is zero.

Second, occasionally the raw counter value read during a context switch
has actually wrapped, but an interrupt has not yet triggered. In this
case the existing logic would return a very large reload count (e.g.
2^48 - 2 if the counter had overflowed by a count of 2). This was seen
both for fixed-function and programmable counters on an E5-2643.
Workaround this case by returning a reload count of zero.

PR: 198149
Sponsored by: Norse Corp, Inc.

MFC 282641,282658:
- Move hwpmc(4) debugging code under a new HWPMC_DEBUG option instead of
the broader DEBUG option.
- Convert hwpmc(4) debug printfs over to KTR.

Sponsored by: Norse Corp, Inc.

MFC of r277177 and r279894 with the fixes for the PMC for Haswell.

Sponsored by: Netflix Inc.

Merge r263233 from HEAD to stable/10:

Update kernel inclusions of capability.h to use capsicum.h instead; some
further refinement is required as some device drivers intended to be
portable over FreeBSD versions rely on __FreeBSD_version to decide whether
to include capability.h.

Sponsored by: Google, Inc.

Clamp too large hwpmc callchaindepth to maximum

If the depth requested by the user is too large, it's better to provide
the maximum than the smaller default.

MFC of: r274766
Sponsored by: The FreeBSD Foundation

MFC r273236:
Use pmc_destroy_pmc_descriptor() to actually free the pmc, which is
consistent with pmc_destroy_owner_descriptor(). Also be sure to destroy
PMCs if a process exits or execs without explicitly releasing them.

MFC r272713:

Since introducing the extra mapping in r250103 (head) for architectural performance
events we have actually counted 'Branch Instruction Retired' when people
asked for 'Unhalted core cycles' using the 'unhalted-core-cycles' event mask
mnemonic.

Reviewed by: jimharris
Discussed with: gnn, rwatson
Sponsored by: DARPA/AFRL

MFC r267062:
Disable existing uncore hwpmc code for Nehalem and Westmere EX.

MFC r263446

Update hwpmc to support core events for Atom Silvermont microarchitecture.
(Model 0x4D as per Intel document 330061-001 01/2014)

MFC r266195:

Remove some prototypes for undefined functions.

MFC r263080:
Use correct types for sizeof() in the calculations for the malloc(9) sizes.

MFC r262547

Fix callchain capture for hwpmc(4). While here, some style(9) fixes, too.

MFC r261342

Add hwpmc(4) support for the PowerPC 970 class processors, direct events.
This also fixes asserts on removal of the module for the mpc74xx.

The PowerPC 970 processors have two different types of events: direct events
and indirect events. Thus far only direct events are supported. I included
some documentation in the driver on how indirect events work, but support is
for the future.

MFC r261173

MPC74xx should not fall through, to the error case.

MFC r258779,r258780,r258787,r258822:

Fix undefined behavior: (1 << 31) is not defined as 1 is an int and this
shifts into the sign bit. Instead use (1U << 31) which gets the
expected result.

Similar to the (1 << 31) case it is not defined to do (2 << 30).

This fix is not ideal as it assumes a 32 bit int, but does fix the issue
for most cases.

A similar change was made in OpenBSD.

MFC r259394,r259395,r259699

r259394:
Rebase the PMC indices at 1, since PMC_SOFT is at 0.

r259395,r259699:
Add userland PMC backtracing, and use the PMC trapframe macros for kernel
backtraces.

Copy head (r256279) to stable/10 as part of the 10.0-RELEASE cycle.

Approved by: re (implicit)
Sponsored by: The FreeBSD Foundation

Remove local change leftover, this should never have been part of
r255745.

Pointy-hat to: davide
Approved by: re (implicit)

Fix lc_lock/lc_unlock() support for rmlocks held in shared mode. With
current lock classes KPI it was really difficult because there was no
way to pass an rmtracker object to the lock/unlock routines. In order
to accomplish the task, modify the aforementioned functions so that
they can return (or pass as argument) an uinptr_t, which is in the rm
case used to hold a pointer to struct rm_priotracker for current
thread. As an added bonus, this fixes rm_sleep() in the rm shared
case, which right now can communicate priotracker structure between
lc_unlock()/lc_lock().

Suggested by: jhb
Reviewed by: jhb
Approved by: re (delphij)

Fix the build.

Change the cap_rights_t type from uint64_t to a structure that we can extend
in the future in a backward compatible (API and ABI) way.

The cap_rights_t represents capability rights. We used to use one bit to
represent one right, but we are running out of spare bits. Currently the new
structure provides place for 114 rights (so 50 more than the previous
cap_rights_t), but it is possible to grow the structure to hold at least 285
rights, although we can make it even larger if 285 rights won't be enough.

The structure definition looks like this:

struct cap_rights {
uint64_t cr_rights[CAP_RIGHTS_VERSION + 2];
};

The initial CAP_RIGHTS_VERSION is 0.

The top two bits in the first element of the cr_rights[] array contain total
number of elements in the array - 2. This means if those two bits are equal to
0, we have 2 array elements.

The top two bits in all remaining array elements should be 0.
The next five bits in all array elements contain array index. Only one bit is
used and bit position in this five-bits range defines array index. This means
there can be at most five array elements in the future.

To define new right the CAPRIGHT() macro must be used. The macro takes two
arguments - an array index and a bit to set, eg.

#define CAP_PDKILL CAPRIGHT(1, 0x0000000000000800ULL)

We still support aliases that combine few rights, but the rights have to belong
to the same array element, eg:

#define CAP_LOOKUP CAPRIGHT(0, 0x0000000000000400ULL)
#define CAP_FCHMOD CAPRIGHT(0, 0x0000000000002000ULL)

#define CAP_FCHMODAT (CAP_FCHMOD | CAP_LOOKUP)

There is new API to manage the new cap_rights_t structure:

cap_rights_t *cap_rights_init(cap_rights_t *rights, ...);
void cap_rights_set(cap_rights_t *rights, ...);
void cap_rights_clear(cap_rights_t *rights, ...);
bool cap_rights_is_set(const cap_rights_t *rights, ...);

bool cap_rights_is_valid(const cap_rights_t *rights);
void cap_rights_merge(cap_rights_t *dst, const cap_rights_t *src);
void cap_rights_remove(cap_rights_t *dst, const cap_rights_t *src);
bool cap_rights_contains(const cap_rights_t *big, const cap_rights_t *little);

Capability rights to the cap_rights_init(), cap_rights_set(),
cap_rights_clear() and cap_rights_is_set() functions are provided by
separating them with commas, eg:

cap_rights_t rights;

cap_rights_init(&rights, CAP_READ, CAP_WRITE, CAP_FSTAT);

There is no need to terminate the list of rights, as those functions are
actually macros that take care of the termination, eg:

#define cap_rights_set(rights, ...) \
__cap_rights_set((rights), __VA_ARGS__, 0ULL)
void __cap_rights_set(cap_rights_t *rights, ...);

Thanks to using one bit as an array index we can assert in those functions that
there are no two rights belonging to different array elements provided
together. For example this is illegal and will be detected, because CAP_LOOKUP
belongs to element 0 and CAP_PDKILL to element 1:

cap_rights_init(&rights, CAP_LOOKUP | CAP_PDKILL);

Providing several rights that belongs to the same array's element this way is
correct, but is not advised. It should only be used for aliases definition.

This commit also breaks compatibility with some existing Capsicum system calls,
but I see no other way to do that. This should be fine as Capsicum is still
experimental and this change is not going to 9.x.

Sponsored by: The FreeBSD Foundation

Fix hwpmc(4) for 32-bit PowerPC.

Refactor PowerPC hwpmc(4) driver into generic and specific. More refactoring
will likely be done as more drivers are added, since AIM-compatible processors
have similar PMC configuration logic.

Complete r250105. Do not zero fields if M_ZERO flag is specified to
malloc(9).

Reported by: pluknet, glebius

Remove the duplicate LLC_MISS event and put it in the right order.

Update the mis-predicted branch PMC names (for sandy bridge) to not clash.

The SDM (June 2013) tables on these are rather confusing. Yes, they
assign the same name (BR_MISP_RETIRED.ALL_BRANCHES) to two codes
(C5H/00H and C5H/04H.) The latter however is the PEBS version.

So, to make it easier to see the difference - and yes, we can use
both without having to actually enable the PEBS specific bits! -
just rename the PEBS one to _PS so there's no clashing.

Tested:

* Sandy bridge

Fix a >80 character long line, introduced in my previous commit.

Noticed by: hiren

Update the MEM_UOP_RETIRED PMC operation for sandy bridge and sandy
bridge Xeon.

Summary: These are PEBS events but they're also available as normal
counter/sample events. The source table (Table 19-2) lists the
base versions (LOAD, STLB_MISS, SPLIT, ALL) but it says they must
be qualified with other values. This particular commit fleshes
out those umask values.

Source:

* Linux; SDM June 2013, Volume 3B, Table 19-2 and 18-21.

Tested:

* Sandy Bridge (non-Xeon)

Rename the kld_unload event handler to kld_unload_try, and add a new
kld_unload event handler which gets invoked after a linker file has been
successfully unloaded. The kld_unload and kld_load event handlers are now
invoked with the shared linker lock held, while kld_unload_try is invoked
with the lock exclusively held.

Convert hwpmc(4) to use these event handlers instead of having
kern_kldload() and kern_kldunload() invoke hwpmc(4) hooks whenever files are
loaded or unloaded. This has no functional effect, but simplifes the linker
code somewhat.

Reviewed by: jhb

Change the name of this particular event to reflect the name used in
Linux and Intel examples.

Sourced:

* https://github.com/andikleen/pmu-tools/blob/master/snb-client.csv
* http://software.intel.com/en-us/comment/1747932#comment-1747932

Note:

* It's not currently in the Intel SDM; I need to chase down what's
going on.

Tested:

* Sandy Bridge

Correct a typo in the event mask mnemonic.

Reviewed by: gnn
MFC after: 3 days

Add in missing events for Sandy Bridge Xeon.

* Add in MEM_LOAD_UOPS_LLC_HIT_RETIRED for both sandy bridge and sandy
bridge Xeon. Right now it only is enabled for Sandy Bridge.
* D2/0F is actually a combination rather than a separate counter, so
just flip that on for the CPU types that support it.

There's an errata for using this on SB Xeon hardware - I've documented
it in kern/181346.

Tested:

* Sandy Bridge
* Sandy Bridge Xeon

Sponsored by: Netflix, Inc.

Relax the vm object locking. Use a read lock.

Sponsored by: EMC / Isilon Storage Division

Suppress a GCC warning. This warning is actually bogus and newer GCC
versions than the one in base (dim@ mentioned he tried on 4.7.3 and 4.8.1)
do not whine about it, so, at some point this workaround will be reverted.

Reported by: ache
Discussed with: dim

malloc(9) cannot return NULL if M_WAITOK flag is specified.

The Intel PMC architectural events have encodings which are identical to
those of some non-architectural core events. This is not a problem in the
general case as long as there's an 1:1 mapping between the two, but there
are few exceptions. For example, 3CH_01H on Nehalem/Westmere represents
both unhalted-reference-cycles and CPU_CLK_UNHALTED.REF_P.
CPU_CLK_UNHALTED.REF_P on the aforementioned architectures does not measure
reference (i.e. bus) but TSC, so there's the need to disambiguate.
In order to avoid the namespace collision rename all the architectural
events in a way they cannot be ambigous and refactor the architectural
events handling function to reflect this change.
While here, per Jim Harris request, rename
iap_architectural_event_is_unsupported() to iap_event_is_architectural().

Discussed with: jimharris
Reviewed by: jimharris, gnn

Complete r250097:
Do not change the initialization order in pmc_intel_initialize().

When hwpmc(4) module is unloaded it reports a double leakage. This happens
at least if FreeBSD is ran under VirtualBox. In order to avoid the leakage,
properly deallocate structures in case CPU claims that hw performance
monitoring counters are not supported.

Reported by: hiren

Fixup Westmere hwpmc(4) support: add missing CPU flag so that
intrucion-retired, llc-misses and llc-reference events can now be
allocated.

Reviewed by: jimharris, gnn

Improve/correct a comment. We now support a lot more cpu types.

PR: kern/177496
Approved by: sbruno (mentor)

Cosmetic change: make a comment reference Sandy Bridge *Xeon*

Reviewed by: sbruno
MFC after: 1 week

Trailing whitespace cleanup along with 80 column enforcemnt.

Submitted by: hiren.panchasara@gmail.com
Reviewed by: sbruno@freebsd.org
Obtained from: Yahoo! Inc.
MFC after: 2 weeks

Update hwpmc to support Haswell class processors.
0x3C: /* Per Intel document 325462-045US 01/2013. */

Add manpage to document all the goodness that is available in this
processor model.

Submitted by: hiren panchasara <hiren.panchasara@gmail.com>
Reviewed by: jimharris, sbruno
Obtained from: Yahoo! Inc.
MFC after: 2 weeks

Switch the vm_object mutex to be a rwlock. This will enable in the
future further optimizations where the vm_object lock will be held
in read mode most of the time the page cache resident pool of pages
are accessed for reading purposes.

The change is mostly mechanical but few notes are reported:
* The KPI changes as follow:
- VM_OBJECT_LOCK() -> VM_OBJECT_WLOCK()
- VM_OBJECT_TRYLOCK() -> VM_OBJECT_TRYWLOCK()
- VM_OBJECT_UNLOCK() -> VM_OBJECT_WUNLOCK()
- VM_OBJECT_LOCK_ASSERT(MA_OWNED) -> VM_OBJECT_ASSERT_WLOCKED()
(in order to avoid visibility of implementation details)
- The read-mode operations are added:
VM_OBJECT_RLOCK(), VM_OBJECT_TRYRLOCK(), VM_OBJECT_RUNLOCK(),
VM_OBJECT_ASSERT_RLOCKED(), VM_OBJECT_ASSERT_LOCKED()
* The vm/vm_pager.h namespace pollution avoidance (forcing requiring
sys/mutex.h in consumers directly to cater its inlining functions
using VM_OBJECT_LOCK()) imposes that all the vm/vm_pager.h
consumers now must include also sys/rwlock.h.
* zfs requires a quite convoluted fix to include FreeBSD rwlocks into
the compat layer because the name clash between FreeBSD and solaris
versions must be avoided.
At this purpose zfs redefines the vm_object locking functions
directly, isolating the FreeBSD components in specific compat stubs.

The KPI results heavilly broken by this commit. Thirdy part ports must
be updated accordingly (I can think off-hand of VirtualBox, for example).

Sponsored by: EMC / Isilon storage division
Reviewed by: jeff
Reviewed by: pjd (ZFS specific review)
Discussed with: alc
Tested by: pho

Add a generic way to call per event allocate / release function.

Reviewed by: mav
MFC after: 1 month

Add support for good old 8192Hz profiling clock to software PMC.

Reviewed by: fabient

Change the way how software PMC updates counters.
This at least fixes -n option of pmcstat.

Reviewed by: fabient

Update hwpmc to support the Xeon class of Ivybridge processors.
case 0x3E: /* Per Intel document 325462-045US 01/2013. */

Add manpage to document all the goodness that is available in this
processor model.

No support for uncore events at this time.

Submitted by: hiren panchasara <hiren.panchasara@gmail.com>
Reviewed by: davide, jimharris, sbruno
Obtained from: Yahoo! Inc.
MFC after: 2 weeks

Quiesce a couple of clang warnings

Submitted by: hiren panchasara <hiren.panchasara@gmail.com>
Obtained from: Yahoo! Inc

Fixup r240246: hwpmc needs to retain the pinning until ASTs are not
executed. This means past the point where userret() is generally
executed.

Skip the td_pinned check if a callchain tracing is currently happening
and add a more robust check to pmc_capture_user_callchain() in order to
catch td_pinned leak past ast() in hwpmc case.

Reported and tested by: fabient
MFC after: 1 week
X-MFC: r240246

Cleanup and rename some variables in libpmc and hwpmc.

Submitted by: hiren panchasara <hiren.panchasara@gmail.com>
Reviewed by: jimharris@ sbruno@
Obtained from: Yahoo! Inc.
MFC after: 2 weeks

Remove the support for using non-mpsafe filesystem modules.

In particular, do not lock Giant conditionally when calling into the
filesystem module, remove the VFS_LOCK_GIANT() and related
macros. Stop handling buffers belonging to non-mpsafe filesystems.

The VFS_VERSION is bumped to indicate the interface change which does
not result in the interface signatures changes.

Conducted and reviewed by: attilio
Tested by: pho

Update hwpmc to support the Xeon class of Sandybridge processors.
(Model 0x2D /* Per Intel document 253669-044US 08/2012. */)

Add manpage to document all the goodness that is available in this
processor model.

No support for uncore events at this time.

Submitted by: hiren panchasara <hiren.panchasara@gmail.com>
Reviewed by: jimharris@ fabient@
Obtained from: Yahoo! Inc.
MFC after: 2 weeks

hwpmc amd_pcpu_fini: fix a bug in code locked under DEBUG

MFC after: 16 days

Remove all the checks on curthread != NULL with the exception of some MD
trap checks (eg. printtrap()).

Generally this check is not needed anymore, as there is not a legitimate
case where curthread != NULL, after pcpu 0 area has been properly
initialized.

Reviewed by: bde, jhb
MFC after: 1 week

Complete and merge the list between Sandy/Ivy bridge of events
that can run on specific PMC.

MFC after: 1 month

Add Intel Ivy Bridge support to hwpmc(9).
Update offcore RSP token for Sandy Bridge.
Note: No uncore support.

Will works on Family 6 Model 3a.

MFC after: 1 month
Tested by: bapt, grehan

Disable hwpmc(4) support for Intel Xeon Sandy Bridge (Model 0x2D).
Due to some differences in MSRs between Xeon Sandy Bridge and Core Sandy
Bridge (Model 0x2A), wrmsr() may generate in a GP# fault exception and so a
panic of the machine.

Approved by: gnn (mentor)
MFC after: 3 days

Add ARM callchain support for hwpmc.

Sponsored by: NETASQ
MFC after: 3 days

Soft PMC support for ARM.
Callgraph is not captured, only current location.

Sample system wide profiling: "pmcstat -Sclock.hard -T"

Remove out of date KASSERT that fire with soft PMC.

MFC after: 1 week

Fix so that ,usr and ,os work correctly with fixed function (IAF)
counters.

MFC after: 1 week

Fix class malloc init for mips and powerpc that was not converted
by r233628.

Found by: monthadar, adrian
MFC after: 1 week

Add software PMC support.

New kernel events can be added at various location for sampling or counting.
This will for example allow easy system profiling whatever the processor is
with known tools like pmcstat(8).

Simultaneous usage of software PMC and hardware PMC is possible, for example
looking at the lock acquire failure, page fault while sampling on
instructions.

Sponsored by: NETASQ
MFC after: 1 month

Fix crash on VirtualBox (and probably on some real hardware):

- Do not cover error returned by pmc_core_initialize with the
result of pmc_uncore_initialize, fail right away.
- Give a user something to report instead failing silently

Reported by: Alexandr Kovalenko <never@nevermind.kiev.ua>

Fix random deadlock on pmcstat exit:
- Exit the thread when soft shutdown is requested
- Wakeup owner thread.

Reproduced/tested by looping pmcstat measurement:
pmcstat -S instructions -O/tmp/test ls

MFC after: 1 week

Add Octeon PMC hardware backend

Add list of Octeon's PMC counters obtained from cvmx-core.h

Rework MIPS PMC code:

- Replace MIPS24K-specific code with more generic framework that will
make adding new CPU support easier
- Add MIPS24K support for new framework
- Limit backtrace depth to 1 for stability reasons and add option
HWPMC_MIPS_BACKTRACE to override this limitation

- Remove unncessary type casts
- Make kernel backtrace routine more robust by refusing to backtrace
further when encountered function that is possibly modifies SP
value

This header file no longer exists when doing cross builds, so remove it.

mips24k hwpmc now compiles again.

Implement pmc_save_user_callchain and pmc_save_kernel_callchain for MIPS

Properly mask off bits that are not supported in the IAP counters.
This fixes a bug where users would see massively large counts, near
to 2**64 -1, due to the bits not being cleared.

MFC after: 3 weeks

- Add support for the Intel Sandy Bridge microarchitecture (both core and uncore counting events)
- New manpages with event lists.
- Add MSRs for the Intel Sandy Bridge microarchitecture

Reviewed by: attilio, brueffer, fabient
Approved by: gnn (mentor)
MFC after: 3 weeks

pmc_*_initialize may return NULL if the CPU is not supported, so check
that md is not null before dereferencing it.

PR: kern/156540

Clean up a switch statement for uncore events on Westmere processors.

Submitted by: Davide Italiano
Reviewed by: gnn
MFC after: 1 week

Update PMC events from October 2011 Intel documentation.

Submitted by: Davide Italiano <davide.italiano@gmail.com>
MFC after: 3 days

Add missing MSR programming for some events.

Submitted by: Davide Italiano <davide.italiano@gmail.com>
MFC after: 3 days

In sys/dev/hwpmc/hwpmc_amd.c, fix a clang warning about invalid enum
conversions.

Reviewed by: jkoshy
MFC after: 1 week

Quite the tinderbox for the holidays. Remove the assert[1].

Suggested by: jhibbits [1]
MFC after: 3 days

Implement hwpmc counting PMC support for PowerPC G4+ (MPC745x/MPC744x).
Sampling is in progress.

Approved by: nwhitehorn (mentor)
MFC after: 9.0-RELEASE

- Remove extra space

Submitted by: Davide Italiano <davide.italiano@gmail.com>
Approved by: brucec

There's a small set of events on Nehalem, that are not supported in
processors with CPUID signature 06_1AH, 06_1EH, and 06_1FH.

Refuse to allocate them on unsupported model.

Submitted by: Davide Italiano <davide.italiano@gmail.com>
MFC after: 1 month

Update Westmere uncore event exception list.

Submitted by: Davide Italiano <davide italiano at gmail com>
MFC after: 1 week

Flip on processing interrupt profile events for mips24k.

This is a bit hackish and should be made more generic (ie, support more than
two hard-coded performance counter+config register pairs) so it can be used
for mips74k and other chips.

All this does is process the initial interrupt event. It doesn't (yet) handle
callgraph events, so even if you route the exception/interrupt to this routine
and flip the bit on, it will hang and crash pmc unless you disable callgraph
support when you enable a sample based PMC.

Add a flush of the current PMC log buffer before displaying the next top.

As the underlying block is 4KB if the PMC throughput is low the measurement
will be reported on the next tick. pmcstat(8) use the modified flush API to
reclaim current buffer before displaying next top.

MFC after: 1 month

Begin implementing correct MIPS24K sampling mode behaviour.

* Add the interrupt bit in the configuration register
* Correctly set the counter register for the sampling overflow
interrupt. The interrupt is asserted when bit 31 is set.
So set the overflow value at 0x80000000 and subtract the
programmed value as appropriate.

In order to maximize the re-usability of kernel code in user space this
patch modifies makesyscalls.sh to prefix all of the non-compatibility
calls (e.g. not linux_, freebsd32_) with sys_ and updates the kernel
entry points and all places in the code that use them. It also
fixes an additional name space collision between the kernel function
psignal and the libc function of the same name by renaming the kernel
psignal kern_psignal(). By introducing this change now we will ease future
MFCs that change syscalls.

Reviewed by: rwatson
Approved by: re (bz)

Second-to-last commit implementing Capsicum capabilities in the FreeBSD
kernel for FreeBSD 9.0:

Add a new capability mask argument to fget(9) and friends, allowing system
call code to declare what capabilities are required when an integer file
descriptor is converted into an in-kernel struct file *. With options
CAPABILITIES compiled into the kernel, this enforces capability
protection; without, this change is effectively a no-op.

Some cases require special handling, such as mmap(2), which must preserve
information about the maximum rights at the time of mapping in the memory
map so that they can later be enforced in mprotect(2) -- this is done by
narrowing the rights in the existing max_protection field used for similar
purposes with file permissions.

In namei(9), we assert that the code is not reached from within capability
mode, as we're not yet ready to enforce namespace capabilities there.
This will follow in a later commit.

Update two capability names: CAP_EVENT and CAP_KEVENT become
CAP_POST_KEVENT and CAP_POLL_KEVENT to more accurately indicate what they
represent.

Approved by: re (bz)
Submitted by: jonathan
Sponsored by: Google Inc

etire the cpumask_t type and replace it with cpuset_t usage.

This is intended to fix the bug where cpu mask objects are
capped to 32. MAXCPU, then, can now arbitrarely bumped to whatever
value. Anyway, as long as several structures in the kernel are
statically allocated and sized as MAXCPU, it is suggested to keep it
as low as possible for the time being.

Technical notes on this commit itself:
- More functions to handle with cpuset_t objects are introduced.
The most notable are cpusetobj_ffs() (which calculates a ffs(3)
for a cpuset_t object), cpusetobj_strprint() (which prepares a string
representing a cpuset_t object) and cpusetobj_strscan() (which
creates a valid cpuset_t starting from a string representation).
- pc_cpumask and pc_other_cpus are target to be removed soon.
With the moving from cpumask_t to cpuset_t they are now inefficient
and not really useful. Anyway, for the time being, please note that
access to pcpu datas is protected by sched_pin() in order to avoid
migrating the CPU while reading more than one (possible) word
- Please note that size of cpuset_t objects may differ between kernel
and userland. While this is not directly related to the patch itself,
it is good to understand that concept and possibly use the patch
as a reference on how to deal with cpuset_t objects in userland, when
accessing kernland members.
- KTR_CPUMASK is changed and now is represented through a string, to be
set as the example reported in NOTES.

Please additively note that no MAXCPU is bumped in this patch, but
private testing has been done until to MAXCPU=128 on a real 8x8x2(htt)
machine (amd64).

Please note that the FreeBSD version is not yet bumped because of
the upcoming pcpu changes. However, note that this patch is not
targeted for MFC.

People to thank for the time spent on this patch:
- sbruno, pluknet and Nicholas Esborn (nick AT desert DOT net) tested
several revision of the patches and really helped in improving
stability of this work.
- marius fixed several bugs in the sparc64 implementation and reviewed
patches related to ktr.
- jeff and jhb discussed the basic approach followed.
- kib and marcel made targeted review on some specific part of the
patch.
- marius, art, nwhitehorn and andreast reviewed MD specific part of
the patch.
- marius, andreast, gonzo, nwhitehorn and jceel tested MD specific
implementations of the patch.
- Other people have made contributions on other patches that have been
already committed and have been listed separately.

Companies that should be mentioned for having participated at several
degrees:
- Yahoo! for having offered the machines used for testing on big
count of CPUs.
- The FreeBSD Foundation for having sponsored my devsummit attendance,
which has been instrumental.
- Sandvine for having offered offices and infrastructure during
development.

(I really hope I didn't forget anyone, if it happened I apologize in
advance).

Merge r221279,221280 from largeSMP project:
pmc_mask doesn't need to use memory barriers.

Reviewed by: fabient
Tested by: several
MFC after: 1 week

Fix two aliases that had the same name but were pointing to different
events. These are now disamiguated.

MFC after: 1 week

Fix invalid class removal when IAF is not the last class.
Keep IAF class with 0 PMC and change the alias in libpmc to IAP.

MFC after: 1 week

Make sure that we clear the correct bits when we turn off
a PMC. It was possible that we could have turned a bit on but
never cleared it.

Extend the calls to rdmsr() to all necessary functions, not
just those which previously caused a panic.

Pointed out by: jhb@
MFC after: 1 week

Fix a panic brought about by writing an MSR without a proper mask.
All of the necessary wrmsr calls are now preceded by a rdmsr
and we leave the reserved bits alone.
Document the bits in the relevant registers for future reference.

Tested by: mdf
MFC after: 1 week

Convert pm_runcount to int to correctly check for negative value.
Remove uncessary check for error.

Found with: Coverity Prevent(tm)
MFC after: 1 month

When configuring a system-wide couting PMC, hwpmc was incorrectly logging process mappings for that PMC. Nothing ever reads pmc logs out of a counting PMC, so the log buffers were leaked when the PMC was deconfigured. The process mappings are only useful for sampling PMCs anyway, so only log the mappings if the PMC is a sampling PMC.

This bug would cause allocating sample-mode PMCs to fail with ENOMEM after allocating several counting-mode PMCs.

Approved by: jkoshy (mentor)
MFC after: 2 weeks

- Fix a typo OFFCORE_REQUESTS.ANY.RFO is B0H10H and not 80H10H.
- Enable missing PARTIAL_ADDRESS_ALIAS for Core i7.

MFC after: 3 days

- Support for uncore counting events: one fixed PMC with the uncore
domain clock, 8 programmable PMC.
- Westmere based CPU (Xeon 5600, Corei7 980X) support.
- New man pages with events list for core and uncore.
- Updated Corei7 events with Intel 253669-033US December 2009 doc.
There is some removed events in the documentation, they have been
kept in the code but documented in the man page as obsolete.
- Offcore response events can be setup with rsp token.

Sponsored by: NETASQ

If there is multiple PMCs for the same interrupt ignore new post.
This will indirectly fix a bug where the thread will be pinned
forever if the assert is not compiled.

MFC after: 3days

Handling SIGPIPE will cause deadlock/crash.
Return an error immediatly in case of hard shutdown.

MFC after: 3days

Change the way shutdown is handled for log file.

pmc_flush_logfile is now non-blocking and just ask the kernel
to shutdown the file. From that point, no more data is
accepted by the log thread and when the last buffer is flushed
the file is closed.

This will remove a deadlock between pmcstat asking for
flush while it cannot flush the pipe itself.

MFC after: 3 days

Add support for hwpmc(4) on the MIPS 24K, 32 bit, embedded processor.

Add macros for properly accessing coprocessor 0 registers that
support performance counters.

Reviewed by: jkoshy rpaulo fabien imp
MFC after: 1 month

Use VFS_{LOCK,UNLOCK}_GIANT() around the call to vrele().

Reviewed by: kib

* Support the L1D_CACHE_LD event on Core2 processors.
* Correct a group of typos: for Core2 programmable events, check
user supplied umask values against the correct event descriptor
field.

Submitted by: Ryan Stone <rysto32 at gmail dot com>

Log process mappings for existing processes at PMC start time.

Submitted by: Marc Unangst <mju at panasas dot com> [original patch]
Tested by: fabient

Intel XScale hwpmc(4) support.

This brings hwpmc(4) support for 2nd and 3rd generation XScale cores.
Right now it's enabled by default to make sure we test this a bit.
When the time comes it can be disabled by default.
Tested on Gateworks boards.

A man page is coming.

Obtained from: //depot/user/rpaulo/xscalepmc/...

Recognize Intel CPUs with Family 0x6, Models 0x1E and 0x1F.

Submitted by: Marc Unangst <mju at panasas dot com>

Use a better check for a valid kernel stack address when capturing
kernel call chains.

Submitted by: Mark Unangst <mju at panasas.com>
Tested by: fabient

Fix parenthesis typo -- copy full frame pointer for userland callchain,
not just one byte.

Submitted by: Ryan Stone rysto32 at gmail dot com

Use switch out (SWO) instead of switch in (SWI) debug log mask in csw_out.

- fix a LOR between process lock and pmc thread mutex
- fix a system deadlock on process exit when the sample buffer
is full (pmclog_loop blocked in fo_write) and pmcstat exit.

Reviewed by: jkoshy
MFC after: 3 weeks

Only claim that the PMC_CLASS_IAF PMCs are supported by a CPU if
there are PMCs on the CPU that belong to the class.

Review and testing by: fabient

Handle the case where there is only one PMC in the system.

Approved by: jkoshy (mentor)
MFC after: 3 days

Fix KASSERT string to include the real module name.

Reserve events for XScale.

Reviewed by: jkoshy, gnn
MFC after: 1 week

Add counters for the i7 architecture which were accidentally left
out of the original commit of i7 support. These are all the counters
on pages A-32 and A-33 of the _Intel(R) 64 and IA32 Architectures
Software Developer's Manual Vol 3B_, June 2009. Almost all
of these counters relate to operations on the L2 cache.

Reviewed by: jkoshy
MFC after: 1 month

Adjust the handling of the local APIC PMC interrupt vector:
- Provide lapic_disable_pmc(), lapic_enable_pmc(), and lapic_reenable_pmc()
routines in the local APIC code that the hwpmc(4) driver can use to
manage the local APIC PMC interrupt vector.
- Do not enable the local APIC PMC interrupt vector by default when
HWPMC_HOOKS is enabled. Instead, the hwpmc(4) driver explicitly
enables the interrupt when it is succesfully initialized and disables
the interrupt when it is unloaded. This avoids enabling the interrupt
on unsupported CPUs which may result in spurious NMIs.

Reported by: rnoland
Reviewed by: jkoshy
Approved by: re (kib)
MFC after: 2 weeks

Fix a LOR between pmc_sx and proctree/allproc when creating a new thread
for the pmclog.

Reported by: Ryan Stone <rstone at sandvine dot com>
Tested by: Ryan Stone <rstone at sandvine dot com>
Sponsored by: Sandvine Incorporated

- Add support for nehalem/corei7 cpus. This supports all of the core
counters defined in the reference manual. It does not support the
'uncore' events.

Reviewed by: jkoshy
Sponsored by: Nokia

Bug fixes:
- Initialize variables before use.
- Remove a KASSERT() that could falsely trigger if there are other sources
of NMIs in the system.

Efficiency tweak:
- When checking PMCs that overflowed, ignore PMCs that were not configured for
sampling.

- Disambiguate a few panic messages.
- Style fixes: wrap long lines, parenthesize return values.

- Bug fix: prevent a thread from migrating between CPUs between the
time it is marked for user space callchain capture in the NMI
handler and the time the callchain capture callback runs.

- Improve code and control flow clarity by invoking hwpmc(4)'s user
space callchain capture callback directly from low-level code.

Reviewed by: jhb (kern/subr_trap.c)
Testing (various patch revisions): gnn,
Fabien Thomas <fabien dot thomas at netasq dot com>,
Artem Belevich <artemb at gmail dot com>

Fixes for Core2 Extreme support.

Submitted by: "Artem Belevich" <artemb at gmail dot com>

Add aliases that map architectural event names to fixed function counters.

- Efficiency tweak: when checking for PMC overflows, only go to
hardware for PMCs that have been configured for sampling.

- Bug fix: acknowledge PMC hardware overflows irrespective of the
the (software) PMC's state.

Improve a comment.

- Add support for PMCs in Intel CPUs of Family 6, model 0xE (Core Solo
and Core Duo), models 0xF (Core2), model 0x17 (Core2Extreme) and
model 0x1C (Atom).

In these CPUs, the actual numbers, kinds and widths of PMCs present
need to queried at run time. Support for specific "architectural"
events also needs to be queried at run time.

Model 0xE CPUs support programmable PMCs, subsequent CPUs
additionally support "fixed-function" counters.

- Use event names that are close to vendor documentation, taking in
account that:
- events with identical semantics on two or more CPUs in this family
can have differing names in vendor documentation,
- identical vendor event names may map to differing events across
CPUs,
- each type of CPU supports a different subset of measurable
events.

Fixed-function and programmable counters both use the same vendor
names for events. The use of a class name prefix ("iaf-" or
"iap-" respectively) permits these to be distinguished.

- In libpmc, refactor pmc_name_of_event() into a public interface
and an internal helper function, for use by log handling code.

- Minor code tweaks: staticize a global, freshen a few comments.

Tested by: gnn

Introduce cpu_vendor_id and replace a lot of strcmp(cpu_vendor, "...").

Reviewed by: jhb, peter (early amd64 version)

Unbreak LINT.

Print PMC widths in the initialization announcement.

Correct an oversight: call the MD finalize hook at module unload
time.

Fix assertions.

Reported by: keramida

Correct an indexing error (a change missed out in #184802).

- Separate PMC class dependent code from other kinds of machine
dependencies. A 'struct pmc_classdep' structure describes operations
on PMCs; 'struct pmc_mdep' contains one or more 'struct pmc_classdep'
structures depending on the CPU in question.

Inside PMC class dependent code, row indices are relative to the
PMCs supported by the PMC class; MI code in "hwpmc_mod.c" translates
global row indices before invoking class dependent operations.

- Augment the OP_GETCPUINFO request with the number of PMCs present
in a PMC class.

- Move code common to Intel CPUs to file "hwpmc_intel.c".

- Move TSC handling to file "hwpmc_tsc.c".

Style tweak.

Remove unnecessary locking around vn_fullpath(). The vnode lock for the
vnode in question does not need to be held. All the data structures used
during the name lookup are protected by the global name cache lock.
Instead, the caller merely needs to ensure a reference is held on the
vnode (such as vhold()) to keep it from being freed.

In the case of procfs' <pid>/file entry, grab the process lock while we
gain a new reference (via vhold()) on p_textvp to fully close races with
execve(2).

For the kern.proc.vmmap sysctl handler, use a shared vnode lock around
the call to VOP_GETATTR() rather than an exclusive lock.

MFC after: 1 month

Fix a number of style issues in the MALLOC / FREE commit. I've tried to
be careful not to fix anything that was already broken; the NFSv4 code is
particularly bad in this respect.

Retire the MALLOC and FREE macros. They are an abomination unto style(9).

MFC after: 3 months

- Sparsely number enumerations 'pmc_cputype' and 'pmc_event' in order to
reduce ABI disruptions when new cpu types and new PMC events are added
in the future.
- Support alternate spellings for PMC events. Derive the canonical
spelling of an event name from its enumeration name in 'enum pmc_event'.
- Provide a way for users to disambiguate between identically named events
supported by multiple classes of PMCs in a CPU.
- Change libpmc's machine-dependent event specifier parsing code to
better support CPUs containing two or more classes of PMC resources.

Rework pmc-dependent flag handling.

Correct a typo.

Fix a typo.

Correct misspellings.

Support sparsely numbered CPUs.

Requested by: obrien, alfred (long ago)

Correct a callchain capture bug on the i386.

On the i386 architecture, the processor only saves the current value
of `%esp' on stack if a privilege switch is necessary when entering
the interrupt handler. Thus, `frame->tf_esp' is only valid for
an entry from user mode. For interrupts taken in kernel mode, we
need to determine the top-of-stack for the interrupted kernel
procedure by adding the appropriate offset to the current frame
pointer.

Reported by: kris, Fabien Thomas
Tested by: Fabien Thomas <fabien.thomas at netasq dot com>

- Provide kernelname as the name for process with P_KTHREAD set as
otherwise their textvp is NULL.

Reviewed by: jkoshy
Sponsored by: Nokia

Sign-extend the 48-bit AMD PMC counter before treating it to a 64-bit
2's compliment.

The 2's compliment transform is done so a "count down" sampling interval
can be converted into a "count up" PMC value. a 2's complimented 'count down'
value is written to the PMC counter; then the read-back counter is reverted
via another 2's compliment.

PR: kern/121660
Reviewed by: jkoshy
Approved by: jkoshy
MFC after: 1 week

Fix the debugging output - the '0x' was duplicated from the %p option.

Correct a typo.

VOP_LOCK1() (and so VOP_LOCK()) and VOP_UNLOCK() are only used in
conjuction with 'thread' argument passing which is always curthread.
Remove the unuseful extra-argument and pass explicitly curthread to lower
layer functions, when necessary.

KPI results broken by this change, which should affect several ports, so
version bumping and manpage update will be further committed.

Tested by: kris, pho, Diego Sardina <siarodx at gmail dot com>

vn_lock() is currently only used with the 'curthread' passed as argument.
Remove this argument and pass curthread directly to underlying
VOP_LOCK1() VFS method. This modify makes the code cleaner and in
particular remove an annoying dependence helping next lockmgr() cleanup.
KPI results, obviously, changed.

Manpage and FreeBSD_version will be updated through further commits.

As a side note, would be valuable to say that next commits will address
a similar cleanup about VFS methods, in particular vop_lock1 and
vop_unlock.

Tested by: Diego Sardina <siarodx at gmail dot com>,
Andrea Di Pasquale <whyx dot it at gmail dot com>

Add stub functions to unbreak LINT.

Kernel and hwpmc(4) support for callchain capture.

Sponsored by: FreeBSD Foundation and Google Inc.

Revert revision 1.4.

Intel CPUs with family 0x6, model 0xE and later (i.e., Intel Core(TM))
have a PMC architecture that differs somewhat from previous CPUs in
family 0x6. Even though the basic programming model is similar, the
documented set of legal values that may be loaded into their PMC MSRs
differs from that of the previous PMCs in family 0x6 and reusing bit
values valid for the older PMCs could result in undefined behaviour in
the general case.

Rename the kthread_xxx (e.g. kthread_create()) calls
to kproc_xxx as they actually make whole processes.
Thos makes way for us to add REAL kthread_create() and friends
that actually make theads. it turns out that most of these
calls actually end up being moved back to the thread version
when it's added. but we need to make this cosmetic change first.

I'd LOVE to do this rename in 7.0 so that we can eventually MFC the
new kthread_xxx() calls.

Commit 14/14 of sched_lock decomposition.
- Use thread_lock() rather than sched_lock for per-thread scheduling
sychronization.
- Use the per-process spinlock rather than the sched_lock for per-process
scheduling synchronization.

Tested by: kris, current@
Tested on: i386, amd64, ULE, 4BSD, libthr, libkse, PREEMPTION, etc.
Discussed with: kris, attilio, kmacy, jhb, julian, bde (small parts each)

Fix witness(4) warnings about mutex use.

Group mutexes used in hwpmc(4) into 3 "types" in the sense of
witness(4):

- leaf spin mutexes---only one of these should be held at a time,
so these mutexes are specified as belonging to a single witness
type "pmc-leaf".

- `struct pmc_owner' descriptors are protected by a spin mutex of
witness type "pmc-owner-proc". Since we call wakeup_one() while
holding these mutexes, the witness type of these mutexes needs
to dominate that of "sleepq chain" mutexes.

- logger threads use a sleep mutex, of type "pmc-sleep".

Submitted by: wkoszek (earlier patch)

Use pause() rather than tsleep() on stack variables and function pointers.

Sweep kernel replacing suser(9) calls with priv(9) calls, assigning
specific privilege names to a broad range of privileges. These may
require some future tweaking.

Sponsored by: nCircle Network Security, Inc.
Obtained from: TrustedBSD Project
Discussed on: arch@
Reviewed (at least in part) by: mlaier, jmg, pjd, bde, ceri,
Alex Lyashkov <umka at sevcity dot net>,
Skip Ford <skip dot ford at verizon dot net>,
Antoine Brodin <antoine dot brodin at laposte dot net>

Declare security and security.bsd sysctl hierarchies in sysctl.h along
with other commonly used sysctl name spaces, rather than declaring them
all over the place.

MFC after: 1 month
Sponsored by: nCircle Network Security, Inc.

First pass at removing Alpha kernel support.

Change msleep() and tsleep() to not alter the calling thread's priority
if the specified priority is zero. This avoids a race where the calling
thread could read a snapshot of it's current priority, then a different
thread could change the first thread's priority, then the original thread
would call sched_prio() inside msleep() undoing the change made by the
second thread. I used a priority of zero as no thread that calls msleep()
or tsleep() should be specifying a priority of zero anyway.

The various places that passed 'curthread->td_priority' or some variant
as the priority now pass 0.

Fix a cut-n-paste bug that crept in.

Reported by: "Pawel Worach" pawel.worach at gmail.com

Add support for Intel cpu model's 5 & 6.

Approved by: jkoshy

Forcibly turn off all PMCs at module unload time.

MFC after: 1 week

MFP4: Support for profiling dynamically loaded objects.

Kernel changes:

Inform hwpmc of executable objects brought into the system by
kldload() and mmap(), and of their removal by kldunload() and
munmap(). A helper function linker_hwpmc_list_objects() has been
added to "sys/kern/kern_linker.c" and is used by hwpmc to retrieve
the list of currently loaded kernel modules.

The unused `MAPPINGCHANGE' event has been deprecated in favour
of separate `MAP_IN' and `MAP_OUT' events; this change reduces
space wastage in the log.

Bump the hwpmc's ABI version to "2.0.00". Teach hwpmc(4) to
handle the map change callbacks.

Change the default per-cpu sample buffer size to hold
32 samples (up from 16).

Increment __FreeBSD_version.

libpmc(3) changes:

Update libpmc(3) to deal with the new events in the log file; bring
the pmclog(3) manual page in sync with the code.

pmcstat(8) changes:

Introduce new options to pmcstat(8): "-r" (root fs path), "-M"
(mapfile name), "-q"/"-v" (verbosity control). Option "-k" now
takes a kernel directory as its argument but will also work with
the older invocation syntax.

Rework string handling in pmcstat(8) to use an opaque type for
interned strings. Clean up ELF parsing code and add support for
tracking dynamic object mappings reported by a v2.0.00 hwpmc(4).

Report statistics at the end of a log conversion run depending
on the requested verbosity level.

Reviewed by: jhb, dds (kernel parts of an earlier patch)
Tested by: gallatin (earlier patch)

When deconfiguring a log, only stop PMCs that are in the RUNNING
state.

When compiled with -DDEBUG, only print the old value of a PMC in a
debugging message if the flag PMC_F_OLDVALUE was specified in the
PMC_OP_RW request being acted upon. This should fix Coverity bug
CID 671.

Found by: Coverity Prevent
MFC after: 3 weeks

When a process is de-configuring a log file, also stop all of its
PMCs that require a log file to operate. This change should fix
PR 90269.

PR: kern/90269
MFC after: 1 week

Fix a memory leak.

Found by: Coverity

- Plug a memory leak: free up per-cpu sample buffers at module unload time.
- Correct a few style nits.

Wrap comment lines to be under 80 characters wide.

MFC after: 3 days

Fix -Wundef warnings found when compiling i386 LINT, GENERIC and
custom kernels.

Add support for a new/unreleased Pentium-M.

Reviewed by: jkoshy

Bug fix initialization on multi-core HTT CPUs.

Reported by: ps
Tested by: ps

Process one NMI interrupt per handler invocation as the processor
'buffers' pending NMIs from multiple interrupting PMCs and delivers
them serially.

Reported by: Olivier Crameri <olivier.crameri@epfl.ch>
MFC after: 3 days

Re-enable sampling on the AMD64.

On x86 processors, turn off any 'INTERRUPT' capabilities on PMCs
if the CPU does not have its local APIC enabled.

MFC after: 3 days

Return EOPNOTSUPP instead of EINVAL if a PMC allocation request
specifies a PMC capability (e.g., sampling) that is not supported
by hardware. Return EINVAL early if the PMC class passed in is
not recognized.

MFC after: 3 days

Print PMC capabilities at module load time.

MFC after: 3 days

Turn off sampling modes on the AMD64 till the time I can track down
the reason for the double fault seen when sampling under load.

MFC after: 3 days

Fail the module loading process if the currently executing kernel
was not compiled with 'options HWPMC_HOOKS' or if the compiled-in
version numbers of the kernel and module are out of sync.

Reported by: cracauer
MFC after: 3 days

Use LK_CANRECURSE since when a PMC-owning process performs an exec,
the new text vnode is already locked by itself.

MFC after: 3 days

Convert the atomic_ptr() operations over to operating on uintptr_t
variables rather than void * variables. This makes it easier and simpler
to get asm constraints and volatile keywords correct.

MFC after: 3 days
Tested on: i386, alpha, sparc64
Compiled on: ia64, powerpc, amd64
Kernel toolchain busted on: arm

Fix breakage introduced in rev 1.7.

MFC after: 3 days

sys/dev/hwpmc/hwpmc_{amd,piv,ppro}.c:
- Update driver interrupt statistics correctly.

sys/sys/pmc.h, sys/dev/hwpmc/hwpmc_mod.c:
- Fix a bug affecting debug printfs.
- Move the 'stalled' flag from being in a bit in the
'pm_flags' field of a 'struct pmc' to a field of its own in the
same structure. This flag is updated from the NMI handler and
keeping it separate makes it easier to avoid races with other
parts of the code.

sys/dev/hwpmc/hwpmc_logging.c:
- Do arithmetic with 'uintptr_t' types rather that casting
to and from 'char *'.

Approved by: re (scottl)

- Update the CPU version check to recognize P4/EMT64 CPUs. [1]
- Allow libpmc(3) to support P4/EMT64 PMCs on the amd64 architecture
and AMD K8 PMCs on the i386. [2]

Submitted by: ps [1]
Pointy hat: myself [2]
Approved by: re (scottl)

MFP4:

- pmcstat(8) gprof output mode fixes:

lib/libpmc/pmclog.{c,h}, sys/sys/pmclog.h:
+ Add a 'is_usermode' field to the PMCLOG_PCSAMPLE event
+ Add an 'entryaddr' field to the PMCLOG_PROCEXEC event,
so that pmcstat(8) can determine where the runtime loader
/libexec/ld-elf.so.1 is getting loaded.

sys/kern/kern_exec.c:
+ Use a local struct to group the entry address of the image being
exec()'ed and the process credential changed flag to the exec
handling hook inside hwpmc(4).

usr.sbin/pmcstat/*:
+ Support "-k kernelpath", "-D sampledir".
+ Implement the ELF bits of 'gmon.out' profile generation in a new
file "pmcstat_log.c". Move all log related functions to this
file.
+ Move local definitions and prototypes to "pmcstat.h"

- Other bug fixes:
+ lib/libpmc/pmclog.c: correctly handle EOF in pmclog_read().
+ sys/dev/hwpmc_mod.c: unconditionally log a PROCEXIT event to all
attached PMCs when a process exits.
+ sys/sys/pmc.h: correct a function prototype.
+ Improve usage checks in pmcstat(8).

Approved by: re (blanket hwpmc)

Fix a -Wuninitialized warning reported by rwatson.

Approved by: re (blanket hwpmc)

MFP4:

- Implement sampling modes and logging support in hwpmc(4).

- Separate MI and MD parts of hwpmc(4) and allow sharing of
PMC implementations across different architectures.
Add support for P4 (EMT64) style PMCs to the amd64 code.

- New pmcstat(8) options: -E (exit time counts) -W (counts
every context switch), -R (print log file).

- pmc(3) API changes, improve our ability to keep ABI compatibility
in the future. Add more 'alias' names for commonly used events.

- bug fixes & documentation.

Kernel hooks to support PMC sampling modes.

Reviewed by: alc

Add convenience APIs pmc_width() and pmc_capabilities() to -lpmc.
Have pmcstat(8) and pmccontrol(8) use these APIs.

Return PMC class-related constants (PMC widths and capabilities)
with the OP GETCPUINFO call leaving OP PMCINFO to return only the
dynamic information associated with a PMC (i.e., whether enabled,
owner pid, reload count etc.).

Allow pmc_read() (i.e., OPS PMCRW) on active self-attached PMCs to
get upto-date values from hardware since we can guarantee that the
hardware is running the correct PMC at the time of the call.

Bug fixes:
- (x86 class processors) Fix a bug that prevented an RDPMC
instruction from being recognized as permitted till after the
attached process had context switched out and back in again after
a pmc_start() call.

Tighten the rules for using RDPMC class instructions: a GETMSR
OP is now allowed only after an OP ATTACH has been done by the
PMC's owner to itself. OP GETMSR is not allowed for PMCs that
track descendants, for PMCs attached to processes other than
their owner processes.

- (P4/HTT processors only) Fix a bug that caused the MI and MD
layers to get out of sync. Add a new MD operation 'get_config()'
as part of this fix.

- Allow multiple system-mode PMCs at the same row-index but on
different CPUs to be allocated.

- Reject allocation of an administratively disabled PMC.

Misc. code cleanups and refactoring. Improve a few comments.

Return the correct register number in the 'get_msr()' MD function.

Only allow a process to use the x86 RDPMC instruction if it has
allocated and attached a PMC to itself.

Inform the MD layer of the "pseudo context switch out" that needs
to be done when the last thread of a process is exiting.

Include <sys/pmc.h> instead of <machine/pmc_mdep.h>. The MI header
includes the MD header for us. Do not include <machine/specialreg.h>
as it is not a header file that can be included from MI files. It
is included from <machine/pmc_mdep.h> if so needed and possible.

Ok'd: jkoshy@

Remove dead variable.

Remove unused variable that was horking up the LINT build

Minimal changes to get this to compile with -DDEBUG defined as well
as hack a couple used before set warnings for LINT happiness.

Bring a working snapshot of hwpmc(4), its associated libraries, userland utilities
and documentation into -CURRENT.

Bump FreeBSD_version.

Reviewed by: alc, jhb (kernel changes)