ktrace: Record namei violations with KTR_CAPFAIL

Report namei path lookups while Capsicum violation tracing with
CAPFAIL_NAMEI. vfs caching is also ignored when tracing to mimic
capability mode behavior.

Reviewed by: markj
Approved by: markj (mentor)
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D40680

file: Remove the fd parameter to fgetvp_lookup() and fgetvp_lookup_smr()

The fd is always obtained from nameidata, so just fetch it from there
instead. No functional change intended.

Reviewed by: kib
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D43257

sys: Remove ancient SCCS tags.

Remove ancient SCCS tags from the tree, automated scripting, with two
minor fixup to keep things compiling. All the common forms in the tree
were removed with a perl script.

Sponsored by: Netflix

vfs cache: Fallback to namei to resolve symlinks with leading / in target for non-native ABI

This is a temporary solution to fix PR before release.
During 15.0 it's necessary to refactor symlinks handling
between vfs & namecache.

PR: 273414
Reported by: Vincent Milum Jr, Dan Kotowski, glebius
Tested by: Dan Kotowski, glebius
Reviewed by:
Differential Revision: https://reviews.freebsd.org/D41806
MFC after: 3 days

vfs cache: add 2 more optimizaiton ideas

vfs cache: denote a known bug in cache_remove_cnp

vfs cache: plug a hypothetical corner case when freeing

cache_zap_unlocked_bucket is called with a bunch of addresses and
without any locks held, forcing it to revalidate everything from
scratch.

It did not account for a case where the entry is reallocated with
everything the same except for the target vnode.

Should the target use a different lock than the one expected, freeing
would proceed without being properly synchronized.

Note this is almost impossible to happen in practice.

vfs cache: sanitize debug counters

They are very rarely triggered, so no need for per-cpu distribution.

At the same time the non-cpu ones still should use atomics to not lose
any updates.

vfs cache: describe various optimization ideas

While here report a sample result from running on Sapphire Rapids:

An access(2) loop slapped into will-it-scale, like so:
while (1) {
int error = access(tmpfile, R_OK);
assert(error == 0);

(*iterations)++;
}

.. operating on /usr/obj/usr/src/amd64.amd64/sys/GENERIC/vnode_if.c

In operations per second:
lockless: 3462164
locked: 1362376

While the over 3.4 mln may seem like a big number, a critical look shows
it should be significantly higher.

A poor man's profiler, counting how many times given routine was sampled:
dtrace -w -n 'profile:::profile-4999 /execname == "a.out"/ {
@[sym(arg0)] = count(); } tick-5s { system("clear"); trunc(@, 40);
printa("%40a %@16d\n", @); clear(@); }'

[snip]
kernel`kern_accessat 231
kernel`cpu_fetch_syscall_args 324
kernel`cache_fplookup_cross_mount 340
kernel`namei 346
kernel`amd64_syscall 352
kernel`tmpfs_fplookup_vexec 388
kernel`vput 467
kernel`vget_finish 499
kernel`lockmgr_unlock 529
kernel`lockmgr_slock 558
kernel`vget_prep_smr 571
kernel`vput_final 578
kernel`vdropl 1070
kernel`memcmp 1174
kernel`0xffffffff80 2080
0x0 2231
kernel`copyinstr_smap 2492
kernel`cache_fplookup 9246

vfs cache: s/vfs.cache_fast_lookup/vfs.cache.param.fast_lookup

vfs cache: retire dothits and dotdothits counters

They demonstrate nothing, and in case of dotdot they are not even hits.
This is just a count of lookups with "..", which are not worth
mentioniong.

vfs cache: mark vfs.cache.param.size as read-only

It was not meant to be writable and writes don't work correctly as they
fail to resize the hash.

vfs cache: Drop known argument of internal cache_recalc_neg_min()

'ncnegminpct' is to be passed always, so just drop the unneeded parameter.

Sponsored by: The FreeBSD Foundation
Reviewed by: mjg

Differential Revision: https://reviews.freebsd.org/D41763

vfs cache: garbage collect the fullpathfail2 counter

The conditions it checks cannot legally be true (modulo races against
forced unmount), so assert on it instead.

vfs cache: fix a hang when bumping vnode limit too high

Overflow in cache_changesize would make the value flip to 0 and stay
there as 0 << 1 does not do anything.

Note callers limit the outcome to something below u_int.

Also note there entire vnode handling thing both in vfs layer as a whole
and this file can't decide whether to long, u_long or u_int.

sys: Remove $FreeBSD$: one-line .c pattern

Remove /^[\s*]*__FBSDID\("\$FreeBSD\$"\);?\s*\n/

vfs: Deleting a doubled inclusion of sys/capsicum.h

Reviewed by:
Differential Revision: https://reviews.freebsd.org/D41223
MFC after: 1 week

vfs: use __enum_uint8 for vtype and vstate

This whacks hackery around only reading v_type once.

Bump __FreeBSD_version to 1400093

vn_path_to_global_path_hardlink(): initialize len

before calling vn_fullpath_hardlink(). Otherwise we get random failures
when the len is automatically clipped.

Reported and tested by: pho
Sponsored by: The FreeBSD Foundation
MFC after: 1 week

vn_path_to_global_path_hardlink(): avoid freeing non-initialized pointer

Reported by: pho
Sponsored by: The FreeBSD Foundation
MFC after: 1 week

vfs cache: restore sorted order of CACHE_FPL_SUPPORTED_CN_FLAGS

namei: Add the abilty for the ABI to specify an alternate root path

For now a non-native ABI (i.e., Linux) uses the kern_alternate_path()
facility to dynamically reroot lookups. First, an attempt is made to
lookup the file in /compat/linux/original-path. If that fails, the
lookup is done in /original-path. Thats requires a bit of code in
every ABI syscall implementation where path name translation is needed.
Also our kern_alternate_path() does not properly lookups absolute symlinks
in second attempt, i.e., does not append /compat/linux part to the resolved
link.
The change is intended to avoid this by specifiyng the ABI root directory
for namei(), using one call to pwd_altroot() during exec-time into the ABI.
In that case namei() will dynamically reroot lookups as mentioned above.

PR: 72920
Reviewed by: kib
Differential revision: https://reviews.freebsd.org/D38933
MFC after: 2 month

vfs cache: fix vfs.cache.stats.* name typos

Two vfs.cache.stats names are fixed:
- s/.dotdothis/.dotdothits/
- s/.posszaps/.poszaps/

Signed-off-by: Igor Ostapenko <pm@igoro.pro>
[mjg: massaged the header a little bit]

vfs: more informative panic for missing fplookup ops

vfs: validate that vop vectors provide all or none fplookup vops

In order to prevent later susprises.

vfs cache: always assert on ndp->ni_resflags

vfs cache: return ENOTDIR for not_a_dir/{.,..} lookups

Reported by: Oliver Kiddle
PR: 270419
MFC: 3 days

vfs cache: whack set-but-not-used warn in cache_purgevfs

Reported by: kib
Sponsored by: Rubicon Communications, LLC ("Netgate")

Allow realpath to work for file mounts

For file mounts, the directory vnode is not available from namei and this
prevents the use of vn_fullpath_hardlink. In this case, we can use the
vnode which was covered by the file mount with vn_fullpath.

This also disallows file mounts over files with link counts greater than
one to ensure a deterministic path to the mount point.

Reviewed by: mjg, kib
Tested by: pho

Add vn_path_to_global_path_hardlink

This is similar to vn_path_to_global_path but allows for regular files
which may not be present in the cache.

Reviewed by: mjg, kib
Tested by: pho

vfs: retire the now unused SAVESTART flag

Bump __FreeBSD_version to 1400075

Tested by: pho

vfs: stop using NDFREE

It provides nothing but a branchfest and next to no consumers want it
anyway.

Tested by: pho

cache: add cache_assert_no_entries

vfs: always retain path buffer after lookup

This removes some of the complexity needed to maintain HASBUF and
allows for removing injecting SAVENAME by filesystems.

Reviewed by: kib (previous version)
Differential Revision: https://reviews.freebsd.org/D36542

cache: drop the vfs.cache_rename_add tunable

The functionality has been in use since Jan 2021 -- long enough(tm).

kern: Fix two typos in source code comments

- s/accomodate/accommodate/

MFC after: 3 days

vfs: fix memory leak on lookup with fds with ioctl caps

Reviewed by: markj
PR: 262515
Noted by: firk@cantconnect.ru
Differential Revision: https://reviews.freebsd.org/D34667

vfs: NDFREE(&nd, NDF_ONLY_PNBUF) -> NDFREE_PNBUF(&nd)

cache: add a comment about a realpath bug

cache: use flexible array member

... instead of 0-sizing the array

cache: hide hash stats behind DEBUG_CACHE

They take a long time to dump and hinder sysctl -a when used with
DIAGNOSTIC.

cache: improve vnode vs name assertion in cache_enter_time

cache: remove NOCACHE handling from cache_fplookup_noentry

It was copy-pasted from locked lookup. As LOOKUP operation cannot have
the flag set it was always ending up setting MAKEENTRY.

vfs: remove the unused thread argument from NDINIT*

See b4a58fbf640409a1 ("vfs: remove cn_thread")

Bump __FreeBSD_version to 1400043.

cache: whack "set but not used" warnings

Make vn_fullpath_hardlink() externally callable

Reviewed by: markj
Tested by: pho
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
Differential revision: https://reviews.freebsd.org/D32611

cache: only let non-dir descriptors through when doing EMPTYPATH lookups

Otherwise things like realpath against a file and '.' end up with an
illegal state of having a regular vnode for the parent.

Reported by: syzbot+9aa5439dd9c708aeb1a8@syzkaller.appspotmail.com

cache: only assert on flags when dealing with EMPTYPATH

Reported by: syzbot+bd48ee0843206a09e6b8@syzkaller.appspotmail.com
Fixes: 7dd419cabc6bb9e0 ("cache: add empty path support")

cache: add empty path support

This avoids spurious drop offs as EMPTY is passed regardless of the
actual path name.

Pushign the work inside the lookup instead of just ignorign the flag
allows avoid checking for empty pathname for all other lookups.

vfs: remove cn_thread

It is always curthread.

Reviewed by: kib
Differential Revision: https://reviews.freebsd.org/D32453

cache: count vnodes in cache_purgevfs

cache: retire cache_fast_revlookup sysctl

Sponsored by: Rubicon Communications, LLC ("Netgate")

cache: add OPENREAD and OPENWRITE to fast path lookup

cache: add cache_enter_time_flags

cache: fix lockless absolute symlink traversal to non-fp mounts

Said lookups would incorrectly fail with EOPNOTSUP.

Reported by: kib

vfs: Fix error handling in vn_fullpath_hardlink()

vn_fullpath_any_smr() will return a positive error number if the
caller-supplied buffer isn't big enough. In this case the error must be
propagated up, otherwise we may copy out uninitialized bytes.

Reported by: syzkaller+KMSAN
Reviewed by: mjg, kib
MFC aftr: 3 days
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D30198

cache: remove incomplete lockless lockout support during resize

This is already properly handled thanks to 2 step hash replacement.

cache: extend mismatch vnode assert print to include the name

vfs: replace vfs_smr_quiesce with vfs_smr_synchronize

This ends up using a smr specific method.

Suggested by: markj
Tested by: pho

cache: update an assert on CACHE_FPL_STATUS_ABORTED

Since symlink support it can get upgraded to CACHE_FPL_STATUS_DESTROYED.

Reported by: bdrewery

cache: add high level overview

Differential Revision: https://reviews.freebsd.org/D28675

cache: fix resizing in face of lockless lookup

Reported by: pho
Tested by: pho

cache: temporarily drop the assert that dvp != vp when adding an entry

Historically it was allowed for any names, but arguably should never be
even attempted. Allow it again since there is a release pending and
allowing it is bug-compatible with previous behavior.

Reported by: otis

cache: assorted comment fixups

cache: remove the largely obsolete general description

Examples of inconsistencies with the current state:
- references LRU of all entries, removed years ago
- references a non-existent lock (neglist)
- claims negative entries have a NULL target

It will be replaced with a more accurate and more informative
description.

In the meantime take it out so it stops misleading.

cache: fix vfs:namecache:lookup:miss probe call sites

cache: drop spurious arg from panic in cache_validate

vp is already reported when noting mismatch

cache: comment on FNV

cache: fix trailing slash support in face of permission problems

Reported by: Johan Hendriks <joh.hendriks gmail.com>
Tested by: kevans

cache: add delayed degenerate path handling

cache: move hash computation into the parsing loop

cache: add trailing slash support

Tested by: pho

cache: handle NOFOLLOW requests for symlinks

Tested by: pho

cache: add missing MNT_NOSYMFOLLOW check to symlink traversal

cache: fallback when encountering a mount point during .. lookup

The current abort is overzealous.

cache: tidy up handling of foo/bar lookups where foo is not a directory

The code was performing an avoidable check for doomed state to account
for foo being a VDIR but turning VBAD. Now that dooming puts a vnode
in a permanent "modify" state this is no longer necessary as the final
status check will catch it.

cache: stop referring to removing entries as invalidating them

Said use is a remnant from the old code and clashes with the NCF_INVALID
flag.

cache: convert cache_fplookup_parse to void now that it always succeeds

cache: change ->v_cache_dd synchronisation rules

Instead of resorting to seqc modification take advantage of immutability
of entries and check if the entry still matches after everything got
prepared.

cache: make ->v_cache_dd accesses atomic-clean for lockless usage

cache: make ->nc_flag accesses atomic-clean for lockless usage

cache: store vnodes in local vars in cache_zap_locked

cache: assorted cleanups

cache: track calls to cache_symlink_alloc with unsupported size

While here assert on size passed to free.

cache: add back target entry on rename

cache: add symlink support to lockless lookup

Reviewed by: kib (previous version)
Tested by: pho (previous version)
Differential Revision: https://reviews.freebsd.org/D27488

cache: augment sdt probe in cache_fplookup_dot

Same as 6d386b4c ("cache: save a branch in cache_fplookup_next")

cache: whitespace nit in cache_fplookup_modifying

cache: fix some typos

cache: drop-write only var from cache_fplookup_preparse

cache: save a branch in cache_fplookup_next

Previously the code would branch on top find out whether it should
branch on SDT probe and bumping the numposhits counter, depending
on cache_fplookup_cross_mount.

Arguably it should be done regardless of what said function returns.

vfs: extend vfs:namei:lookup:return probe with nameidata

cache: just assign ni_resflags = NIRES_ABS

It is guaranteed to be 0 on entry.

cache: stop checkpointing cn_flags

They are only modified, if ever, for the last component.

cache: stop checkpointing cn_nameptr

For aborts cn_nameptr is the same as cn_pnbuf. For partial results
the same cn_nameptr is to be used.

cache: stop manipulating pathlen

It is a copy-pasto from regular lookup. Add debug to ensure the result
is the same.

cache: unengrish the comment in previous commit

Reported by: rpokala, brd

cache: stop pre-checking seqc when starting the lookup

Tested by: pho

cache: skip checking for spurious slashes if possible

Tested by: pho

cache: combine fast path enabled status into one flag

Tested by: pho

cache: split handling of 0 and non-0 error codes

Tested by: pho

cache: deinline state handling

The intent is to reduce branchfest when finishing the lookup.

Tested by: pho

cache: stop setting cn_nameptr on entry as matches cn_pnbuf already

While here tidy up other asserts.

cache: drop the now spurious doomed check when crossing a mount point

vfs: denote vnode being a mount point with VIRF_MOUNTPOINT

Reviewed by: kib (previous version)
Differential Revision: https://reviews.freebsd.org/D27794

cache: stop checkpointing cn_namelen

The variable is recomputed by regular lookup from the get go.

cache: predict on no spurious slashes in cache_fpl_handle_root

This is a step towards speculatively not handling them.

cache: postpone NAME_MAX check as it may be unnecessary

cache: remove spurious null check in sdt probe

cache: fold NCF_WHITE check into the rest

Tested by: pho

cache: call cache_fplookup_modifying in neg

Tested by: pho

cache: refactor cache_fpl_handle_root to fit the rest of the code better

Tested by: pho

cache: refactor dot handling

Tested by: pho

cache: remove a branch from mount point checking

Tested by: pho

cache: support lockless lookup of degenerate paths

Tested by: pho

cache: save on branching when parsing the path by inserting a sentinel

Tested by: pho

cache: hoist trailing slash and degenerate path handling out of the loop

Tested by: pho

cache: work around corner case of dvp == tvp in cache_fplookup_final_modifying

Fixes a panic where the kernel would unlock an unheld lock coming from
rename looking up "foo/." as the source.

Reported by: markj (syzkaller)

cache: reduce engrish in previous commit

cache: save on some branching in common case mount point traversal

vfs: add FAILIFEXISTS flag

Both FreeBSD and Linux mkdir -p walk the tree up ignoring any EEXIST on
the way and both are used a lot when building respective kernels.

This poses a problem as spurious locking avoidably interferes with
concurrent operations like getdirentries on affected directories.

Work around the problem by adding FAILIFEXISTS flag. In case of lockless
lookup this manages to avoid any work to begin with, there is no speed
up for the locked case but perhaps this can be augmented later on.

For simplicity the only supported semantics are as used by mkdir.

Reviewed by: kib (previous version)
Differential Revision: https://reviews.freebsd.org/D27789

cache: simplify lockless dot lookups

cache: modification and last entry filling support in lockless lookup v2

The previous patch failed to set the ISDOTDOT flag when appropriate,
which in turn fail to properly handle degenerate lookups.

While here sprinkle some extra assertions.

Tested by: pho (previous version)

cache: assert internal flags are not passed by namei

Revert "cache: modification and last entry filling support in lockless lookup"

This reverts commit 6dbb07ed6872ae7988b9b705e322c94658eba6d1.

Some ports unreliably fail to build with rmdir getting ENOTEMPTY.

cache: modification and last entry filling support in lockless lookup

Tested by: pho (previous version)

cache: fix up cache_hold_vnode comment

cache: fix ups bad predicts

- last level fallback normally sees CREATE; the code should be optimized to not
get there for said case
- fast path commonly fails with ENOENT

cache_fplookup: quiet gcc -Wreturn-type

Reviewed by: markj, mjg
Sponsored by: Dell EMC Isilon
Differential Revision: https://reviews.freebsd.org/D27555

vfs: group mount per-cpu vars into one struct

While here move frequently read stuff into the same cacheline.

This shrinks struct mount by 64 bytes.

Tested by: pho

cache: add cache_vop_mkdir and rename cache_rename to cache_vop_rename

cache: add missing NIRES_ABS handling

cache: assorted typo fixes

cache: add the missing NC_NOMAKEENTRY and NC_KEEPPOSENTRY to lockless lookup

They are de facto ignored.

cache: batch updates to numcache in case of mass removal

cache: refactor alloc/free

This in particular centralizes manipulation of numcache.

cache: fold branch prediction into cache_ncp_canuse

cache: fix some typos

cache: drop write-only vars

cache: reduce memory waste in struct namecache

The previous scheme for calculating the total size was doing sizeof
on the struct and then adding the wanted space for the buffer.

nc_name is at offset 58 while sizeof(struct namecache) is 64.
With CACHE_PATH_CUTOFF of 39 bytes and 1 byte of padding we were
allocating 104 bytes for the entry and never accounting for the 6
byte padding, wasting that space.

vfs: prevent avoidable evictions on mkdir of existing directories

mkdir -p /foo/bar/baz will mkdir each path component and ignore EEXIST.

The NOCACHE lookup will make the namecache unnecessarily evict the existing entry,
and then fallback to the fs lookup routine eventually leading namei to return an
error as the directory is already there.

For invocations like mkdir -p /usr/obj/usr/src/sys/GENERIC/modules this triggers
fallbacks to the slowpath for concurrently executing lookups.

Tested by: pho
Discussed with: kib

cache: assert the created entry does not point to itself

cache: drop the spurious slash_prefixed argument

vfs: drop spurious cred argument from VOP_VPTOCNP

cache: promote negative entries based on more than one hit

During tinderbox and similar workloads negative entries get at least one
hit before they get evicted. In the current scheme this avoidably promotes
them.

Be conservative and stick to 2 hits for now.

cache: refactor negative promotion/demotion handling

This will simplify policy changes.

cache: shorten names of debug stats

cache: don't automatically evict negative entries if usage is low

The previous scheme only looked at negative entry count in relation to the
total count, leading to tons of spurious evictions if the cache is not
significantly populated.

Instead, only try the above if negative entry count goes beyond namecache
capacity.

cache: erwork sysctl vfs.cache tree

Split everything into neg, debug, param and stat categories.

The legacy nchstats sysctl (queried e.g., by systat) remains untouched.

While here rename some vars to be easier on the eye.

cache: factor negative lookup out of cache_fplookup_next

cache: avoid smr in cache_neg_evict in favoro of the already held bucket lock

cache: rework parts of negative entry management

- declutter sysctl vfs.cache by moving relevant entries into
vfs.cache.neg
- add a little more parallelism to eviction by replacing the
global lock with an atomically modified counter
- track more statistics

The code needs further effort.

cache: remove entries before trying to add new ones, not after

Should allow positive entries to replace negative ones in case
the cache is full.

cache: add a probe reporting addition of duplicate entries

cache: flip inverted condition in previous

It happened to not affect correctness in that the fallback code would
simply neglect to promote the entry.

cache: support negative entry promotion in slowpath smr

cache: elide vhold/vdrop around promoting negative entry

cache: dedup code for negative promotion

cache: neglist -> nl; negstate -> ns

No functional changes.

cache: split hotlist between existing negative lists

This simplifies the code while allowing for concurrent negative eviction
down the road.

Cache misses increased slightly due to higher rate of evictions allowed by
the change.

The current algorithm remains too aggressive.

cache: make neglist an array given the static size

vfs: support lockless dirfd lookups

cache: fix vexec panic when racing against vgone

Use of dead_vnodeops would result in a panic instead of returning the intended
EOPNOTSUPP error.

While here make sure to abort, not just try to return a partial result.
The former allows the regular lookup to restart from scratch, while the latter
makes it stuck with an unusable vnode.

Reported by: kevans

cache: fix pwd use-after-free in setting up fallback

Since the code exits smr section prior to calling pwd_hold, the used
pwd can be freed and a new one allocated with the same address, making
the comparison erroneously true.

Note it is very unlikely anyone ran into it.

cache: update the commentary for path parsing

cache: properly report ENOTDIR on foo/bar lookups where foo is a file

Reported by: fernape

cache: push the lock into cache_purge_impl

cache: use cache_has_entries where appropriate instead of opencoding it

cache: eliminate cache_zap_locked_vnode

It is only ever called for negative entries and for those it is
just a wrapper around cache_zap_negative_locked_vnode_kl which
always succeeds.

This also fixes a bug where cache_lookup_fallback should have been
calling cache_zap_locked_bucket instead. Note that in order to trigger
the bug NOCACHE must not be set, which currently only happens when
creating a new coredump (and then the coredump-to-be has to have a
negative entry).

cache: drop the force flag from purgevfs

The optional scan is wasteful, thus it is removed altogether from unmount.

Callers which always want it anyway remain unaffected.

cache: reimplement purgevfs to iterate vnodes instead of the entire hash

The entire cache scan was a leftover from the old implementation.

It is incredibly wasteful in presence of several mount points and does not
win much even for single ones.

cache: clean up atomic ops on numneg and numcache

- use subtract instead of adding -1
- drop the useless _rel fence

Note this should be converted to a scalable scheme.

cache: drop write-only tvp_seqc vars

cache: don't update timestmaps on found entry

cache: assorted clean ups

In particular remove spurious comments, duplicate assertions and the
inconsistently done KTR support.

cache: ncp = NULL early to account for sdt probes in ailure path

CID: 1432106

cache: relock on failure in cache_zap_locked_vnode

This gets rid of bogus scheme of yielding in hopes the blocking thread will
make progress.

cache: stop null checking in cache_free

cache: make it mandatory to request both timestamps or neither

cache: convert bucketlocks to a mutex

By now bucket locks are almost never taken for anything but writing and
converting to mutex simplifies the code.

cache: only evict negative entries on CREATE when ISLASTCN is set

cache: decouple smr and locked lookup in the slowpath

Tested by: pho

cache: factor dotdot lookup out of cache_lookup

Tested by: pho

cache: remove leftover assert in vn_fullpath_any_smr

It is only valid when !slash_prefixed. For slash_prefixed the length
is properly accounted for later.

Reported by: markj (syzkaller)

cache: lockless reverse lookup

This enables fully scalable operation for getcwd and significantly improves
realpath.

For example:
PATH_CUSTOM=/usr/src ./getcwd_processes -t 104
before: 1550851
after: 380135380

Tested by: pho

cache: drop the always curthread argument from reverse lookup routines

Note VOP_VPTOCNP keeps getting it as temporary compatibility for zfs.

Tested by: pho

cache: perform reverse lookup using v_cache_dd if possible

Tested by: pho

cache: populate v_cache_dd for non-VDIR entries

It makes v_cache_dd into a little bit of a misnomer and it may be addressed later.

Tested by: pho

cache: stronger vnode asserts in cache_enter_time

vfs: add a work around for vp_crossmp bug to realpath

The actual bug is not yet addressed as it will get much easier after other
problems are addressed (most notably rename contract).

The only affected in-tree consumer is realpath. Everyone else happens to be
performing lookups within a mount point, having a side effect of ni_dvp being
set to mount point's root vnode in the worst case.

Reported by: pho

cache: don't use cache_purge_negative when renaming

It avoidably scans (and evicts) unrelated entries. Instead take
advantage of passed componentname and perform a hash lookup
for the exact one.

Sample data from buildworld probed on cache_purge_negative extended
to count both scanned and evicted entries on each call are below.
At most it has to evict 1.

evicted
value ------------- Distribution ------------- count
-1 | 0
0 |@@@@@@@@@@@@@@@ 19506
1 |@@@@@ 5820
2 |@@@@@@ 7751
4 |@@@@@ 6506
8 |@@@@@ 5996
16 |@@@ 4029
32 |@ 1489
64 | 193
128 | 109
256 | 56
512 | 16
1024 | 7
2048 | 3
4096 | 1
8192 | 1
16384 | 0

scanned
value ------------- Distribution ------------- count
-1 | 0
0 |@@ 2456
1 |@ 1496
2 |@@ 2728
4 |@@@ 4171
8 |@@@@ 5122
16 |@@@@ 5335
32 |@@@@@ 6279
64 |@@@@ 5671
128 |@@@@ 4558
256 |@@ 3123
512 |@@ 2790
1024 |@@ 2449
2048 |@@ 3021
4096 |@ 1398
8192 |@ 886
16384 | 0

cache: add cache_rename, a dedicated helper to use for renames

While here make both tmpfs and ufs use it.

No fuctional changes.

cache: reimplement cache_lookup_nomakeentry as cache_remove_cnp

This in particular removes unused arguments.

cache: when adding an already existing entry assert on a complete match

cache: tidy up the comment above cache_prehash

cache: add NOCAPCHECK to the list of supported flags for lockless lookup

It is de facto supported in that lockless lookup does not do any capability
checks.

vfs: use namei_zone for getcwd allocations

instead of malloc.

Note that this should probably be wrapped with a dedicated API and other
vn_getcwd callers did not get converted.

cache: let SAVESTART passthrough

The flag is only passed for non-LOOKUP ops and those fallback to the slowpath.

cache: resize struct namecache to a multiply of alignment

For example struct namecache on amd64 is 100 bytes, but it has to occupies
104. Use the extra bytes to support longer names.

cache: remove unused variables from cache_fplookup_parse

vfs: clean MNTK_FPLOOKUP if MNT_UNION is set

Elides checking it during lookup.

cache: strlcpy -> memcpy

vfs: partially support file create/delete/rename in lockless lookup

Perform the lookup until the last 2 elements and fallback to slowpath.

Tested by: pho
Sponsored by: The FreeBSD Foundation

vfs: drop the thread argumemnt from vfs_fplookup_vexec

It is guaranteed curthread.

Tested by: pho
Sponsored by: The FreeBSD Foundation

cache: add more predicts for failing conditions

cache: plug unititalized variable use

CID: 1431128

cache: don't ignore size passed to nchinittbl

cache: convert the hash from LIST to SLIST

This reduces struct namecache by sizeof(void *).

Negative side is that we have to find the previous element (if any) when
removing an entry, but since we normally don't expect collisions it should be
fine.

Note this adds cache_get_hash calls which can be eliminated.

cache: reduce zone alignment to 8 bytes

It used to be sizeof of the given struct to accomodate for 32 bit mips
doing 64 bit loads, but the same can be achieved with requireing just
64 bit alignment.

While here reorder struct namecache so that most commonly used fields
are closer.

cache: convert ncnegnash into a macro

It is a read-only var with value known at compilation time.

cache: cleanup lockless entry point

- remove spurious bzero
- assert ni_lcf, it has to be set by namei by this point

cache: stop messing with cn_lkflags

See r363882.

cache: stop messing with cn_flags

This removes flag setting/unsetting carried over from regular lookup.
Flags still get for compatibility when falling back.

Note .. and . handling can get partially folded together.

vfs: support lockless dotdot lookup

Tested by: pho

cache: add NCF_WIP flag

This allows making half-constructed entries visible to the lockless lookup,
which now can check for either "not yet fully constructed" and "no longer valid"
state.

This will be used for .. lookup.

cache: add cache_purge_vgone

cache_purge locklessly checks whether the vnode at hand has any namecache
entries. This can race with a concurrent purge which managed to remove
the last entry, but may not be done touching the vnode.

Make sure we observe the relevant vnode lock as not taken before proceeding
with vgone.

Paired with the fact that doomed vnodes cannnot receive entries this restores
the invariant that there are no namecache-related writing users past cache_purge
in vgone.

Reported by: pho

cache: factor away failed vexec handling

cache: assorted tidy ups

cache: factor away lockless dot lookup and add missing stat + sdt probe

vfs: add vfs_op_thread_enter/exit _crit variants

and employ them in the namecache. Eliminates all spurious checks for preemption.

cache: add missing numcache detrement on insertion failure

vfs: store precomputed namecache hash in the vnode

This significantly speeds up path lookup, Cascade Lake doing access(2) on ufs
on /usr/obj/usr/src/amd64.amd64/sys/GENERIC/vnode_if.c, ops/s:
before: 2535298
after: 2797621

Over +10%.

The reversed order of computation here does not seem to matter for hash
distribution.

Reviewed by: kib
Differential Revision: https://reviews.freebsd.org/D25921

vfs: move namecache initialisation into cache_vnode_init

cache: reshuffle struct cache_fpl and nameidata_saved

Shaves 16 bytes.

cache: mark climb_mount as __noinline

cache: drop the useless numchecks counter

vfs: add support for WANTPARENT and LOCKPARENT to lockless lookup

This makes the realpath syscall operational with the new lookup. Note that the
walk to obtain the full path name still takes locks.

Tested by: pho
Differential Revision: https://reviews.freebsd.org/D23917

vfs: support negative entry promotion in lockless lookup

Tested by: pho

vfs: add NOMACCHECK and AUDITVNODE2 to lockless lookup

They are both nops since lookup does not progress with either mac or audit enabled.

Tested by: pho

vfs: add support for !LOCKLEAF to lockless lookup

Tested by: pho (in a patchset)
Differential Revision: https://reviews.freebsd.org/D23916

vfs: lockless lookup

Provides full scalability as long as all visited filesystems support the
lookup and terminal vnodes are different.

Inner workings are explained in the comment above cache_fplookup.

Capabilities and fd-relative lookups are not supported and will result in
immediate fallback to regular code.

Symlinks, ".." in the path, mount points without support for lockless lookup
and mismatched counters will result in an attempt to get a reference to the
directory vnode and continue in regular lookup. If this fails, the entire
operation is aborted and regular lookup starts from scratch. However, care is
taken that data is not copied again from userspace.

Sample benchmark:
incremental -j 104 bzImage on tmpfs:
before: 142.96s user 1025.63s system 4924% cpu 23.731 total
after: 147.36s user 313.40s system 3216% cpu 14.326 total

Sample microbenchmark: access calls to separate files in /tmpfs, 104 workers, ops/s:
before: 2165816
after: 151216530

Reviewed by: kib
Tested by: pho (in a patchset)
Differential Revision: https://reviews.freebsd.org/D25578

cache: make negative shrinker round robin on all lists every time

Previously it would check 4, 3, 2, 1 lists. In practice by the time
it is getting called all lists have some elements and consequently
this does not result in new evictions.

Nonetheless, the code is clearer.

Tested by: pho

cache: remove numcalls

The counter is not very useful and if necessary the value can be
found by summing up other counters.

cache: count dropped entries

cache: remove neg_locked argument from cache_zap_locked

Tested by: pho

cache: remove a useless argument from cache_negative_insert

cache: create a dedicate struct for negative entries

.. and stuff if into the unused target vnode field

This gets rid of concurrent nc_flag modifications racing with the
shrinker and consequently fixes a bug where such a change could have
been missed when cache_ncp_invalidate was being issued..

Reported by: zeising
Tested by: pho, zeising
Fixes: r362828 ("cache: lockless forward lookup with smr")

cache: add missing call to cache_ncp_invalid for negative hits

Note the dtrace probe can fire even the entry is gone, but I don't think that's
worth fixing.

cache: fix misplaced fence in cache_ncp_invalidate

The intent was to mark the entry as invalid before cache_zap starts messing
with it.

While here add some comments.

cache: lockless forward lookup with smr

This eliminates the need to take bucket locks in the common case.

Concurrent lookup utilizng the same vnodes is still bottlenecked on referencing
and locking path components, this will be taken care of separately.

Reviewed by: kib
Tested by: pho
Differential Revision: https://reviews.freebsd.org/D23913

Use COUNTER_U64_DEFINE_EARLY() in places where it simplifies things.

Reviewed by: kib
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D23978

fd: move vnodes out of filedesc into a dedicated structure

The new structure is copy-on-write. With the assumption that path lookups are
significantly more frequent than chdirs and chrooting this is a win.

This provides stable root and jail root vnodes without the need to reference
them on lookup, which in turn means less work on globally shared structures.
Note this also happens to fix a bug where jail vnode was never referenced,
meaning subsequent access on lookup could run into use-after-free.

Reviewed by: kib
Differential Revision: https://reviews.freebsd.org/D23884

Mark more nodes as CTLFLAG_MPSAFE or CTLFLAG_NEEDGIANT (17 of many)

r357614 added CTLFLAG_NEEDGIANT to make it easier to find nodes that are
still not MPSAFE (or already are but aren’t properly marked).
Use it in preparation for a general review of all nodes.

This is non-functional change that adds annotations to SYSCTL_NODE and
SYSCTL_PROC nodes using one of the soon-to-be-required flags.

Mark all obvious cases as MPSAFE. All entries that haven't been marked
as MPSAFE before are by default marked as NEEDGIANT

Approved by: kib (mentor, blanket)
Commented by: kib, gallatin, melifaro
Differential Revision: https://reviews.freebsd.org/D23718

vfs: add realpathat syscall

realpath(3) is used a lot e.g., by clang and is a major source of getcwd
and fstatat calls. This can be done more efficiently in the kernel.

This works by performing a regular lookup while saving the name and found
parent directory. If the terminal vnode is a directory we can resolve it using
usual means. Otherwise we can use the name saved by lookup and resolve the
parent.

See the review for sample syscall counts.

Reviewed by: kib
Differential Revision: https://reviews.freebsd.org/D23574

Provide O_SEARCH

O_SEARCH is defined by POSIX [0] to open a directory for searching, skipping
permissions checks on the directory itself after the initial open(). This is
close to the semantics we've historically applied for O_EXEC on a directory,
which is UB according to POSIX. Conveniently, O_SEARCH on a file is also
explicitly undefined behavior according to POSIX, so O_EXEC would be a fine
choice. The spec goes on to state that O_SEARCH and O_EXEC need not be
distinct values, but they're not defined to be the same value.

This was pointed out as an incompatibility with other systems that had made
its way into libarchive, which had assumed that O_EXEC was an alias for
O_SEARCH.

This defines compatibility O_SEARCH/FSEARCH (equivalent to O_EXEC and FEXEC
respectively) and expands our UB for O_EXEC on a directory. O_EXEC on a
directory is checked in vn_open_vnode already, so for completeness we add a
NOEXECCHECK when O_SEARCH has been specified on the top-level fd and do not
re-check that when descending in namei.

[0] https://pubs.opengroup.org/onlinepubs/9699919799/

Reviewed by: kib
Differential Revision: https://reviews.freebsd.org/D23247

cache: replace kern___getcwd with vn_getcwd

The previous routine was resulting in extra data copies most notably in
linux_getcwd.

cache: return the total length from vn_fullpath1

This removes strlen from getcwd.

cache: remove vnode -> path lookup disablement

It seems to be of little to no use even when debugging.

Interested parties can resurrect it and gate compilation with a macro.

vfs: consistently use size_t for buflen around VOP_VPTOCNP

cache: revert r352613 now that vhold does not take locks

cache: make numcachehv use counter(9) on all archs

Requested by: kib

cache: counter_u64_add_protected -> counter_u64_add

Fixes booting on RISC-V where it does happen to not be equivalent.

Reported by: lwhsu

cache: convert numcachehv to counter(9) on 64-bit platforms

vfs: incomplete pass at converting more ints to u_long

Most notably numvnodes and freevnodes were u_long, but parameters used to
govern them remained as ints.

vfs: drop the mostly unused flags argument from VOP_UNLOCK

Filesystems which want to use it in limited capacity can employ the
VOP_UNLOCK_FLAGS macro.

Reviewed by: kib (previous version)
Differential Revision: https://reviews.freebsd.org/D21427

vfs: introduce v_irflag and make v_type smaller

The current vnode layout is not smp-friendly by having frequently read data
avoidably sharing cachelines with very frequently modified fields. In
particular v_iflag inspected for VI_DOOMED can be found in the same line with
v_usecount. Instead make it available in the same cacheline as the v_op, v_data
and v_type which all get read all the time.

v_type is avoidably 4 bytes while the necessary data will easily fit in 1.
Shrinking it frees up 3 bytes, 2 of which get used here to introduce a new
flag field with a new value: VIRF_DOOMED.

Reviewed by: kib, jeff
Differential Revision: https://reviews.freebsd.org/D22715

cache: stop reusing .. entries on enter

It almost never happens in practice anyway. With this eliminated ->nc_vp
cannot change vnodes, removing an obstacle on the road to lockless
lookup.

cache: fix numcache accounting on entry

. entries are never created and .. can reuse existing entries,
meaning the early count bump is both spurious and leading to
overcounting in certain cases.

cache: hide "doingcache" behind DEBUG_CACHE

cache: minor stat cleanup

Remove duplicated stats and move numcachehv from debug to vfs.cache.

cache: decrease ncnegfactor to 5

The current mechanism is bogus in several ways:
- the limit is a percentage of total entries added, which means negative
entries get evicted all the time even if there are plenty of resources
- evicting code is almost not concurrent, which makes it unable to
remove entries fast enough when doing something as simple as -j 104
buildworld
- there is no support for performing mass removal if necessary

Vast majority of negative entries never get any hits. Only evicting
them when the filesystem demands it results in a significant growth of
the namecache with almost no improvement in the hit ratio.

Sample result about afer 90 minutes of poudriere -j 104:

current no evict % of the original
numneg 219737 2013157 916
numneghits 266711906 263544562 98 [1]

[1] this may look funny but there is a certain dose of variation to the
build

The number was chosen as something which mostly eliminates spurious
evictions during lighter workloads but still keeps the total at bay.

Sponsored by: The FreeBSD Foundation

cache: stop requeuing negative entries on the hot list

Turns out it does not improve hit ratio, but it does come with a cost
induces stemming from dirtying hit entries.

Sample result: hit counts of evicted entries after 2 buildworlds

before:

value ------------- Distribution ------------- count
-1 | 0
0 |@@@@@@@@@@@@@@@@@@@@@@@@@ 180865
1 |@@@@@@@ 49150
2 |@@@ 19067
4 |@ 9825
8 |@ 7340
16 |@ 5952
32 |@ 5243
64 |@ 4446
128 | 3556
256 | 3035
512 | 1705
1024 | 1078
2048 | 365
4096 | 95
8192 | 34
16384 | 26
32768 | 23
65536 | 8
131072 | 6
262144 | 0

after:
value ------------- Distribution ------------- count
-1 | 0
0 |@@@@@@@@@@@@@@@@@@@@@@@@@ 184004
1 |@@@@@@ 47577
2 |@@@ 19446
4 |@ 10093
8 |@ 7470
16 |@ 5544
32 |@ 5475
64 |@ 5011
128 | 3451
256 | 3002
512 | 1729
1024 | 1086
2048 | 363
4096 | 86
8192 | 26
16384 | 25
32768 | 24
65536 | 7
131072 | 5
262144 | 0

Sponsored by: The FreeBSD Foundation

cache: make negative list shrinking a little bit concurrent

Continue protecting demotion from the hotlist and selection of the
target list with the ncneg_shrink_lock lock, but drop it before
relocking to zap the node.

While here count how many times we skipped shrinking due to the lock
being already taken.

Sponsored by: The FreeBSD Foundation

cache: stop recalculating upper limit each time a new entry is added

Sponsored by: The FreeBSD Foundation

cache: tidy up handling of negative entries

- track the total count of hot entries
- pre-read the lock when shrinking since it is typically already taken
- place the lock in its own cacheline
- shorten the hold time of hot lock list when zapping

Sponsored by: The FreeBSD Foundation

cache: count evictions of negatve entries

Sponsored by: The FreeBSD Foundation

cache: try to avoid vhold if locks held

Sponsored by: The FreeBSD Foundation

cache: jump in negative success instead of positive

Sponsored by: The FreeBSD Foundation

cache: avoid excessive relocking on entry removal during lookup

Due to lock ordering issues (bucket lock held, vnode locks wanted) the code
starts with trylocking which in face of contention often fails. Prior to
the change it would loop back with a possible yield.

Instead note we know what locks are needed and can take them in the right
order, avoiding retries. Then we can safely re-lookup and see if the entry
we are looking for is still there.

On a 104-way box poudriere would result in constant retries during an 11h
run as seen in the vfs.cache.zap_and_exit_bucket_fail counter.

before: 408866592
after : 0

However, a new stat reports:
vfs.cache.zap_and_exit_bucket_relock_success: 32638

Note this is only a bandaid over current design issues.

Tested by: pho
Sponsored by: The FreeBSD Foundation

cache: change the formula for calculating lock array sizes

It used to be mp_ncpus * 64, but this gives unnecessarily big values for small
machines and at the same time constraints bigger ones. In particular this helps
on a 104-way box for which the count is now doubled.

While here make cache_purgevfs less likely. Currently it is not efficient in
face of contention due to lock ordering issues. These are fixable but not worth
it at the moment.

Sponsored by: The FreeBSD Foundation

cache: assorted cleanups

Sponsored by: The FreeBSD Foundation

vfs: implement usecount implying holdcnt

vnodes have 2 reference counts - holdcnt to keep the vnode itself from getting
freed and usecount to denote it is actively used.

Previously all operations bumping usecount would also bump holdcnt, which is
not necessary. We can detect if usecount is already > 1 (in which case holdcnt
is also > 1) and utilize it to avoid bumping holdcnt on our own. This saves
on atomic ops.

Reviewed by: kib
Tested by: pho (previous version)
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D21471

[skip ci] Fix the comment for cache_purge(9)

This is a merge of r348738 from projects/fuse2

Reviewed by: kib
MFC after: 2 weeks
Sponsored by: The FreeBSD Foundation

[skip ci] Fix the comment for cache_purge(9)

Sponsored by: The FreeBSD Foundation

Include ktr.h in more compilation units

Similar to r348026, exhaustive search for uses of CTRn() and cross reference
ktr.h includes. Where it was obvious that an OS compat header of some kind
included ktr.h indirectly, .c files were left alone. Some of these files
clearly got ktr.h via header pollution in some scenarios, or tinderbox would
not be passing prior to this revision, but go ahead and explicitly include it
in files using it anyway.

Like r348026, these CUs did not show up in tinderbox as missing the include.

Reported by: peterj (arm64/mp_machdep.c)
X-MFC-With: r347984
Sponsored by: Dell EMC Isilon

cache: fix a brainfart in r347505

If bumping over the counter goes over the limit we have to decrement it back.

Previous code would only bump the counter after adding the entry (thus allowing
the cache to go over the limit).

Sponsored by: The FreeBSD Foundation

cache: bump numcache on entry, while here fix lnumcache type

Sponsored by: The FreeBSD Foundation

cache: push sdt probes in cache_zap_locked to code doing the work

Avoids branching to check which probe to evaluate. Very same check was
being done later to do the actual work.

Sponsored by: The FreeBSD Foundation

fix cache_lookup's documentation

cache_lookup's documentation got dislocated by r324378. Relocate and expand
it.

Reviewed by: jhb, kib
MFC after: 2 weeks
Sponsored by: The FreeBSD Foundation

cache: retire cache_enter compat schim

It was added over 6 years ago for binary compat. cache_enter macro remains
as it expands to cache_enter_time.

Sponsored by: The FreeBSD Foundation

Sometimes it is helpful to get the path for a vnode.
Implement a ddb function walking the namecache to do this.

Reviewed by: jhb, mjg
Inspired by: gdb macro from jhb (old version)
Sponsored by: iXsystems, Inc.
Differential Revision: https://reviews.freebsd.org/D14898

cache_lookup remove unused variable and initialize used

Plug a name cache lock leak.

Reviewed by: mjg
MFC after: 1 week
Sponsored by: Dell EMC Isilon

sys: further adoption of SPDX licensing ID tags.

Mainly focus on files that use BSD 3-Clause license.

The Software Package Data Exchange (SPDX) group provides a specification
to make it easier for automated tools to detect and summarize well known
opensource licenses. We are gradually adopting the specification, noting
that the tags are considered only advisory and do not, in any way,
superceed or replace the license texts.

Special thanks to Wind River for providing access to "The Duke of
Highlander" tool: an older (2014) run over FreeBSD tree was useful as a
starting point.

namecache: bump numcache after dropping all locks

This makes no difference correctness-wise, but shortens total hold time.

namecache: wlock buckets in cache_lookup_nomakeentry

Since the case of an empty chain was already covered, it si very likely
that the existing entry is matching. Skipping readlocking saves on lock
upgrade.

namecache: skip locking in cache_lookup_nomakeentry if there is no entry

namecache: skip locking in cache_purge_negative if there are no entries

namecache: fix .. check broken after r324378

wtf by: mjg
Diagnosed by: avg

namecache: ncnegfactor 16 -> 12

It is used on each new entry addition to decide whether to whack an existing
negative entry in order to prevent a blow out in size, but the parameter was
set years ago and never revisited.

Building with poudriere results in about 400 evictions per second which
unnecessarily grab entries from the hot list.

With the new parameter there are next to no evictions of the sort.

namecache: factor out ~MAKEENTRY lookups from the common path

Lookups of the sort are rare compared to regular ones and succesfull ones
result in removing entries from the cache.

In the current code buckets are rlocked and a trylock dance is performed,
which can fail and cause a restart. Fixing it will require a little bit
of surgery and in order to keep the code maintaineable the 2 cases have
to split.

MFC after: 1 week

Use UMA_ALIGNOF() for name cache UMA zones.

This fixes kernel crashes due to misaligned accesses to the 64-bit
time_t embedded in struct namecache_ts in MIPS n32 kernels.

MFC after: 1 week
Sponsored by: DARPA / AFRL

namecache: clean up struct namecache_ts handling

namecache_ts differs from mere namecache by few fields placed mid struct.
The access to the last element (the name) is thus special-cased.

The standard solution is to put new fields at the very beginning anad
embedd the original struct. The pointer shuffled around points to the
embedded part. If needed, access to new fields can be gained through
__containerof.

MFC after: 1 week

namecache: fold the unlock label into the only consumer

No functional changes.

MFC after: 1 week

namecache: factor out dot lookup into a dedicated function

The intent is to move uncommon cases out of the way.

MFC after: 1 week

cache: stop holding the ncneg_hot lock across purging

Only non-hot entries are purged so the lock is not needed in the first place.
This saves one lock/unlock pair.

MFC after: 1 week

Regen after r316594.

cache: use vrefact for '.' lookups and refing the rdir in fullpath

cache: annotate with __read_mostly and __exclusive_cache_line

MFC after: 1 month

cache: sprinkle __predict_false

cache: move shrink lock init to nchinit

This gets rid of unnecesary sysinit usage.

While here also rename the lock to be consistent with the rest.

cache: depessimize hashing macros/inlines

All hash sizes are power-of-2, but the compiler does not know that for sure
and 'foo % size' forces doing a division.

Store the size - 1 and use 'foo & hash' instead which allows mere shift.

cache: drop the NULL check from VP2VNODELOCK

Now that negative entries are annotated with a dedicated flag, NULL vnodes
are no longer passed.

vfs: use vrefact in getcwd and fchdir

cache: ensure that the number of bucket locks does not exceed hash size

The size can be changed by side effect of modifying kern.maxvnodes.

Since numbucketlocks was not modified, setting a sufficiently low value
would give more locks than actual buckets, which would then lead to
corruption.

Force the number of buckets to be not smaller.

Note this should not matter for real world cases.

Reported and tested by: pho

cache: plug a write-only variable in cache_negative_zap_one

cache: fix a race between entry removal and demotion

The negative list shrinker can demote an entry with only hotlist + neglist
locks held. On the other hand entry removal possibly sets the NCF_DVDROP
without aformentioned locks held prior to detaching it from the respective
netlist., which can lose the update made by the shrinker.

Reported and tested by: truckman

vn_fullpath1() checked VV_ROOT and then unreferenced
vp->v_mount->mnt_vnodecovered unlocked. This allowed unmount to race.
Lock vnode after we noticed the VV_ROOT flag. See comments for
explanation why unlocked check for the flag is considered safe.

Reported and tested by: avg
Sponsored by: The FreeBSD Foundation
MFC after: 1 week

cache: fix up a corner case in r307650

If no negative entry is found on the last list, the ncp pointer will be
left uninitialized and a non-null value will make the function assume an
entry was found.

Fix the problem by initializing to NULL on entry.

Reported by: glebius

cache: split negative entry LRU into multiple lists

This splits the ncneg_mtx lock while preserving the hit ratio at least
during buildworld.

Create N dedicated lists for new negative entries.

Entries with at least one hit get promoted to the hot list, where they
get requeued every M hits.

Shrinking demotes one hot entry and performs a round-robin shrinking of
regular lists.

Reviewed by: kib

Limit scope of the optimization in r306608 to dounmount() caller only.
Other uses of cache_purgevfs() do rely on the cache purge for correct
operations, when paths are invalidated without unmount.

Reported and tested by: jkim
Discussed with: mjg
Sponsored by: The FreeBSD Foundation

cache: ignore purgevfs requests for filesystems with few vnodes

purgevfs is purely optional and induces lock contention in workloads
which frequently mount and unmount filesystems.

In particular, poudriere will do this for filesystems with 4 vnodes or
less. Full cache scan is clearly wasteful.

Since there is no explicit counter for namecache entries, the number of
vnodes used by the target fs is checked.

The default limit is the number of bucket locks.

Reviewed by: kib

cache: get rid of the global lock

Add a table of vnode locks and use them along with bucketlocks to provide
concurrent modification support. The approach taken is to preserve the
current behaviour of the namecache and just lock all relevant parts before
any changes are made.

Lookups still require the relevant bucket to be locked.

Discussed with: kib
Tested by: pho

Renumber license clauses in sys/kern to avoid skipping #3

cache: improve scalability by introducing bucket locks

An array of bucket locks is added.

All modifications still require the global cache_lock to be held for
writing. However, most readers only need the relevant bucket lock and in
effect can run concurrently to the writer as long as they use a
different lock. See the added comment for more details.

This is an intermediate step towards removal of the global lock.

Reviewed by: kib
Tested by: pho

cache: defer freeing entries until after the global lock is dropped

This also defers vdrop for held vnodes.

Glanced at by: kib

cache: manage negative entry list with a dedicated lock

Since negative entries are managed with a LRU list, a hit requires a
modificaton.

Currently the code tries to upgrade the global lock if needed and is
forced to retry the lookup if it fails.

Provide a dedicated lock for use when the cache is only shared-locked.

Reviewed by: kib
MFC after: 1 week

cache: put all negative entry management code into dedicated functions

Reviewed by: kib
MFC after: 1 week

sys/kern: spelling fixes in comments.

No functional change.

Provide more correct sizing of the KVA consumed by a vnode, used by
the virtvnodes calculation. Include the size of fs-specific v_data as
the nfs nclnode inline, the NFS nclnode is bigger than either ZFS
znode or UFS inode. Include the size of namecache_ts and short cache
path element, multiplied by the name cache population factor, again
inline.

Inline defines are used to avoid pollution of the vnode.h with the
subsystem-private objects. Non-significant unsynchronized changes of
the definitions are fine, we do not care about that precision, and
e.g. ZFS consumes much malloced memory per vnode for reasons
unaccounted in the formula.

Lower the partition of kmem dedicated to vnodes, from 1/7 to 1/10.

The measures reduce vnode cache pressure on kmem and bring the vnode
cache memory use below some apparent thresholds that were exceeded by
r291244 due to more robust vnode reuse.

Reported and tested by: marius (i386, previous version)
Reviewed by: bde
Sponsored by: The FreeBSD Foundation
MFC after: 2 weeks

cache: minor changes

1. vhold and zap immediately instead of postponing few lines later
2. increment numneg after new entry is added

No functional changes.

No objections: kib

cache: perform . lockup without the namecache lock

Reviewed by: kib

cache: provide a helper for computing the hash

Reviewed by: kib

cache: use counter(9) API to maintain statistics

Previously the code would just increment statistics while only holding a
shared lock, in effect losing updates.

Separate tracking for nchstats is removed as values can be obtained from
existing counters. Note that some fields are updated by external
consumers and are left unfixed. This should not be a serious issue as
this structure looks quite obsolete.

No strong objections: kib

cache: ansify functions and fix some style issues

No functional changes.

Fix style issues around existing SDT probes.

- Use SDT_PROBE<N>() instead of SDT_PROBE(). This has no functional effect
at the moment, but will be needed for some future changes.
- Don't hardcode the module component of the probe identifier. This is
set automatically by the SDT framework.

MFC after: 1 week

save some bytes by using more concise SDT_PROBE<n> instead of SDT_PROBE

SDT_PROBE requires 5 parameters whereas SDT_PROBE<n> requires n parameters
where n is typically smaller than 5.

Perhaps SDT_PROBE should be made a private implementation detail.

MFC after: 20 days

Track changes to kern.maxvnodes and appropriately increase or decrease
the size of the name cache hash table (mapping file names to vnodes)
and the vnode hash table (mapping mount point and inode number to vnode).
An appropriate locking strategy is the key to changing hash table sizes
while they are in active use.

Reviewed by: kib
Tested by: Peter Holm
Differential Revision: https://reviews.freebsd.org/D2265
MFC after: 2 weeks

vfs: implement v_holdcnt/v_usecount manipulation using atomic ops

Transitions 0->1 and 1->0 (which decide e.g. on putting the vnode on the free
list) of either counter are still guarded with vnode interlock.

Reviewed by: kib (earlier version)
Tested by: pho

Modify kern___getcwd() to take max pathlen limit as an additional
argument. This will be used for the Linux emulation layer - for Linux,
PATH_MAX is 4096 and not 1024.

Differential Revision: https://reviews.freebsd.org/D2335
Reviewed by: kib@
MFC after: 1 month
Sponsored by: The FreeBSD Foundation

More accurately collect name-cache statistics in sysctl functions
sysctl_debug_hashstat_nchash() and sysctl_debug_hashstat_rawnchash().
These changes are in preparation for allowing changes in the size
of the vnode hash tables driven by increases and decreases in the
maximum number of vnodes in the system.

Reviewed by: kib@
Phabric: D2265

Indeed, instead of hiding the kern___getcwd() bug by bogus cast
in r276564, change path type to char * (pathnames are always char *).
And remove bogus casts of malloc().
kern___getcwd() internally doesn't actually use or support u_char *
paths, except to copy them to a normal char * path.

These changes are not visible to libc as libc/gen/getcwd.c misdeclares
__getcwd() as taking a plain char * path.

While here remove _SYS_SYSPROTO_H_ for __getcwd() syscall as
we always have sysproto.h.

Pointed out by: bde

MFC after: 1 week

Fix multiple incorrect SYSCTL arguments in the kernel:

- Wrong integer type was specified.

- Wrong or missing "access" specifier. The "access" specifier
sometimes included the SYSCTL type, which it should not, except for
procedural SYSCTL nodes.

- Logical OR where binary OR was expected.

- Properly assert the "access" argument passed to all SYSCTL macros,
using the CTASSERT macro. This applies to both static- and dynamically
created SYSCTLs.

- Properly assert the the data type for both static and dynamic
SYSCTLs. In the case of static SYSCTLs we only assert that the data
pointed to by the SYSCTL data pointer has the correct size, hence
there is no easy way to assert types in the C language outside a
C-function.

- Rewrote some code which doesn't pass a constant "access" specifier
when creating dynamic SYSCTL nodes, which is now a requirement.

- Updated "EXAMPLES" section in SYSCTL manual page.

MFC after: 3 days
Sponsored by: Mellanox Technologies

vn_path_to_global_path: update comment.

Fix accounting for the negative cache entries when reusing v_cache_dd.
Having ncneg diverge with the actual length of the ncneg tailq causes
NULL dereference.

Add assertion that an entry taken from ncneg queue is indeed negative.

Reported by and discussed with: avg
Sponsored by: The FreeBSD Foundation
MFC after: 1 week

dtrace sdt: remove the ugly sname parameter of SDT_PROBE_DEFINE

In its stead use the Solaris / illumos approach of emulating '-' (dash)
in probe names with '__' (two consecutive underscores).

Reviewed by: markj
MFC after: 3 weeks

- For kernel compiled only with KDTRACE_HOOKS and not any lock debugging
option, unbreak the lock tracing release semantic by embedding
calls to LOCKSTAT_PROFILE_RELEASE_LOCK() direclty in the inlined
version of the releasing functions for mutex, rwlock and sxlock.
Failing to do so skips the lockstat_probe_func invokation for
unlocking.
- As part of the LOCKSTAT support is inlined in mutex operation, for
kernel compiled without lock debugging options, potentially every
consumer must be compiled including opt_kdtrace.h.
Fix this by moving KDTRACE_HOOKS into opt_global.h and remove the
dependency by opt_kdtrace.h for all files, as now only KDTRACE_FRAMES
is linked there and it is only used as a compile-time stub [0].

[0] immediately shows some new bug as DTRACE-derived support for debug
in sfxge is broken and it was never really tested. As it was not
including correctly opt_kdtrace.h before it was never enabled so it
was kept broken for a while. Fix this by using a protection stub,
leaving sfxge driver authors the responsibility for fixing it
appropriately [1].

Sponsored by: EMC / Isilon storage division
Discussed with: rstone
[0] Reported by: rstone
[1] Discussed with: philip

namecache sdt: freebsd doesn't support structured characters yet

:-)

MFC after: 7 days

When renaming a directory from one parent directory to another,
we need to call ufs_checkpath() to walk from our new location to
the root of the filesystem to ensure that we do not encounter
ourselves along the way. Until now, we accomplished this by reading
the ".." entries of each directory in our path until we reached
the root (or encountered an error). This change tries to avoid the
I/O of reading the ".." entries by first looking them up in the
name cache and only doing the I/O when the name cache lookup fails.

Reviewed by: kib
Tested by: Peter Holm
MFC after: 4 weeks

Remove the support for using non-mpsafe filesystem modules.

In particular, do not lock Giant conditionally when calling into the
filesystem module, remove the VFS_LOCK_GIANT() and related
macros. Stop handling buffers belonging to non-mpsafe filesystems.

The VFS_VERSION is bumped to indicate the interface change which does
not result in the interface signatures changes.

Conducted and reviewed by: attilio
Tested by: pho

Post r230394, the Lookup RPC counts for both NFS clients increased
significantly. Upon investigation this was caused by name cache
misses for lookups of "..". For name cache entries for non-".."
directories, the cache entry serves double duty. It maps both the
named directory plus ".." for the parent of the directory. As such,
two ctime values (one for each of the directory and its parent) need
to be saved in the name cache entry.
This patch adds an entry for ctime of the parent directory to the
name cache. It also adds an additional uma zone for large entries
with this time value, in order to minimize memory wastage.
As well, it fixes a couple of cases where the mtime of the parent
directory was being saved instead of ctime for positive name cache
entries. With this patch, Lookup RPC counts return to values similar
to pre-r230394 kernels.

Reported by: bde
Discussed with: kib
Reviewed by: jhb
MFC after: 2 weeks

o Reduce chances for integer overflow.
o More verbose sysctl description added.

MFC after: 2 weeks
Sponsored by: Nginx, Inc.

Rename cache_lookup_times() to cache_lookup() and retire the old API and
ABI stub for cache_lookup().

Fix remaining calls to cache_enter() in both NFS clients to provide
appropriate timestamps. Restore the assertions which verify that
NCF_TS is set when timestamp is asked for.

Reviewed by: jhb (previous version)
MFC after: 2 weeks

Apparently, both nfs clients do not use cache_enter_time()
consistently, creating some namecache entries without NCF_TS flag.
This causes panic due to failed assertion.

As a temporal relief, remove the assert. Return epoch timestamp for
the entries without timestamp if asked.

While there, consolidate the code which returns timestamps, into a
helper cache_out_ts().

Discussed with: jhb
MFC after: 2 weeks

Remove the nc_time and nc_ticks elements from struct namecache, and
provide struct namecache_ts which is the old struct namecache. Only
allocate struct namecache_ts if non-null struct timespec *tsp was
passed to cache_enter_time, otherwise use struct namecache.

Change struct namecache allocation and deallocation macros into static
functions, since logic becomes somewhat twisty. Provide accessor for
the nc_name member of struct namecache to hide difference between
struct namecache and namecache_ts.

The aim of the change is to not waste 20 bytes per small namecache
entry.

Reviewed by: jhb
MFC after: 2 weeks
X-MFC-note: after r230394

Close a race in NFS lookup processing that could result in stale name cache
entries on one client when a directory was renamed on another client. The
root cause for the stale entry being trusted is that each per-vnode nfsnode
structure has a single 'n_ctime' timestamp used to validate positive name
cache entries. However, if there are multiple entries for a single vnode,
they all share a single timestamp. To fix this, extend the name cache
to allow filesystems to optionally store a timestamp value in each name
cache entry. The NFS clients now fetch the timestamp associated with
each name cache entry and use that to validate cache hits instead of the
timestamps previously stored in the nfsnode. Another part of the fix is
that the NFS clients now use timestamps from the post-op attributes of
RPCs when adding name cache entries rather than pulling the timestamps out
of the file's attribute cache. The latter is subject to races with other
lookups updating the attribute cache concurrently. Some more details:
- Add a variant of nfsm_postop_attr() to the old NFS client that can return
a vattr structure with a copy of the post-op attributes.
- Handle lookups of "." as a special case in the NFS clients since the name
cache does not store name cache entries for ".", so we cannot get a
useful timestamp. It didn't really make much sense to recheck the
attributes on the the directory to validate the namecache hit for "."
anyway.
- ABI compat shims for the name cache routines are present in this commit
so that it is safe to MFC.

MFC after: 2 weeks

Fix missing in r230129:

kern_jail.c: initialize fullpath_disabled to zero
vfs_cache.c: add missing dot in comment

Reported by: kib
MFC after: 1 month

Introduce vn_path_to_global_path()

This function updates path string to vnode's full global path and checks
the size of the new path string against the pathlen argument.

In vfs_domount(), sys_unmount() and kern_jail_set() this new function
is used to update the supplied path argument to the respective global path.

Unbreaks jailed zfs(8) with enforce_statfs set to 1.

Reviewed by: kib
MFC after: 1 month

put sys/systm.h at its proper place or add it if missing

Reported by: lstewart, tinderbox
Pointyhat to: avg, attilio
MFC after: 1 week
MFC with: r228430

Existing VOP_VPTOCNP() interface has a fatal flow that is critical for
nullfs. The problem is that resulting vnode is only required to be
held on return from the successfull call to vop, instead of being
referenced.

Nullfs VOP_INACTIVE() method reclaims the vnode, which in combination
with the VOP_VPTOCNP() interface means that the directory vnode
returned from VOP_VPTOCNP() is reclaimed in advance, causing
vn_fullpath() to error with EBADF or like.

Change the interface for VOP_VPTOCNP(), now the dvp must be
referenced. Convert all in-tree implementations of VOP_VPTOCNP(),
which is trivial, because vhold(9) and vref(9) are similar in the
locking prerequisites. Out-of-tree fs implementation of VOP_VPTOCNP(),
if any, should have no trouble with the fix.

Tested by: pho
Reviewed by: mckusick
MFC after: 3 weeks (subject of re approval)

Mark all SYSCTL_NODEs static that have no corresponding SYSCTL_DECLs.

The SYSCTL_NODE macro defines a list that stores all child-elements of
that node. If there's no SYSCTL_DECL macro anywhere else, there's no
reason why it shouldn't be static.

In order to maximize the re-usability of kernel code in user space this
patch modifies makesyscalls.sh to prefix all of the non-compatibility
calls (e.g. not linux_, freebsd32_) with sys_ and updates the kernel
entry points and all places in the code that use them. It also
fixes an additional name space collision between the kernel function
psignal and the libc function of the same name by renaming the kernel
psignal kern_psignal(). By introducing this change now we will ease future
MFCs that change syscalls.

Reviewed by: rwatson
Approved by: re (bz)

Fix some more style(9) issues.

Fix style(9) issues from r215281 and r215282.

MFC after: 1 week

Add some descriptions to sys/kern sysctls.

PR: kern/148710
Tested by: Chip Camden <sterling at camdensoftware.com>
MFC after: 1 week

Remove sysctl debug.ncnegfactor, it is renamed to vfs.ncnegfactor.

MFC: do not

A new jail(8) with a configuration file, to replace the work currently done
by /etc/rc.d/jail.

Provide vfs.ncsizefactor instead of hard-coding namecache ratio.
Move debug.ncnegfactor to vfs.ncnegfactor [1].
Provide some descriptions for the namecache related sysctls [1].

Based on the submission by: Rogier R. Mulhuijzen <drwilco drwilco net> [1]
MFC after: 2 weeks
X-MFC-note: remove debug.ncnegfactor in HEAD after MFC

Add an extra comment to the SDT probes definition. This allows us to get
use '-' in probe names, matching the probe names in Solaris.[1]

Add userland SDT probes definitions to sys/sdt.h.

Sponsored by: The FreeBSD Foundation
Discussed with: rwaston [1]

Use ISO C99 integer types in sys/kern where possible.

There are only about 100 occurences of the BSD-specific u_int*_t
datatypes in sys/kern. The ISO C99 integer types are used here more
often.

MFC r206894:
The cache_enter(9) function shall not be called for doomed dvp.
Assert this.

Verify that dvp is not reclaimed before calling cache_enter().

The cache_enter(9) function shall not be called for doomed dvp.
Assert this.

In the reported panic, vdestroy() fired the assertion "vp has namecache
for ..", because pseudofs may end up doing cache_enter() with reclaimed
dvp, after dotdot lookup temporary unlocked dvp.
Similar problem exists in ufs_lookup() for "." lookup, when vnode
lock needs to be upgraded.

Verify that dvp is not reclaimed before calling cache_enter().

Reported and tested by: pho
Reviewed by: kan
MFC after: 2 weeks

MFC r206671:
Fix typo.

Fix typo.

MFC after: 3 days

MFC r196203:
Correctly handle unlock for !MAKEENTRY case.

Approved by: re (rwatson)

Correctly handle unlock for !MAKEENTRY case, after successfull attempt of
lock upgrade cache shall be unlocked from write.

Reported by: Lucius Windschuh <lwindschuh googlemail com>
Reviewed by: kan
Approved by: re (rwatson)

Add explicit struct ucred * argument for VOP_VPTOCNP, to be used by
vn_open_cred in default implementation. Valid struct ucred is needed for
audit and MAC, and curthread credentials may be wrong.

This further requires modifying the interface of vn_fullpath(9), but it
is out of scope of this change.

Reviewed by: rwatson

Unlock the cache lock before returning when we run out of buffer space
trying to fill in the full path name.

Reported by: David Naylor <naylor.b.david@gmail.com>
Approved by: kib

Unbreak the build. Add missed probes.

Reviewed by: rwatson
Pointy hat to: me

Eliminate code duplication in vn_fullpath1() around the cache lookups
and calls to vn_vptocnp() by moving more of the common code to
vn_vptocnp(). Rename vn_vptocnp() to vn_vptocnp_locked() to signify that
cache is locked around the call.

Do not track buffer position by both the pointer and offset, use only
buflen to record the start of the free space.

Export vn_vptocnp() for external consumers as a wrapper around
vn_vptocnp_locked() that locks the cache and handles hold counts.

Tested by: pho

More fallout from negative dotdot caching. Negative entries should
be removed from and reinserted to proper ncneg list.

Reported by: pho
Submitted by: kib

Redo previous change using simpler patch that happens to be also
more correct.

Submitted by: tor

Fix yet another negative dotodot entry fallout.

Reported by: pho

Fix v_cache_dd handling for negative entries. v_cache_dd pointer was
not populated in parent directory if negative entry was being
created, yet entry itself was added to the nc_neg list. It was
possible for parent vnode to get discarded later, leaving negative
entry pointing to now unused memory block.

Reported by: dho
Revewed by: kib

When zapping v_cache_dd for !MAKEENTRY case in cache_lookup(), we shall
lock cache as writer.

Reviewed by: kan

Cache_lookup() for DOTDOT drops dvp vnode lock, allowing dvp to be reclaimed.
Check the condition and return ENOENT then.

In nfs_lookup(), respect ENOENT return from cache_lookup() when it is caused
by dvp reclaim.

Reported and tested by: pho

Nul-terminate strings in the VFS name cache, which negligibly change
the size and cost of name cache entries, but make adding debugging
and tracing easier.

Add SDT DTrace probes for various namecache events:

vfs:namecache:enter:done - new entry in the name cache, passed parent
directory vnode pointer, name added to the cache, and child vnode
pointer.

vfs:namecache:enter_negative:done - new negative entry in the name cache,
passed parent vnode pointer, name added to the cache.

vfs:namecache:fullpath:enter - call to vn_fullpath1() is made, passed
the vnode to resolve to a name.

vfs:namecache:fullpath:hit - vn_fullpath1() successfully resolved a
search for the parent of an object using the namecache, passed the
discovered parent directory vnode pointer, name, and child vnode
pointer.

vfs:namecache:fullpath:miss - vn_fullpath1() failed to resolve a search
for the parent of an object using the namecache, passed the child
vnode pointer.

vfs:namecache:fullpath:return - vn_fullpath1() has completed, passed the
error number, and if that is zero, the vnode to resolve, and the
returned path.

vfs:namecache:lookup:hit - postive name cache entry hit, passed the
parent directory vnode pointer, name, and child vnode pointer.

vfs:namecache:lookup:hit_negative - negative name cache entry hit,
passed the parent directory vnode pointer and name.

vfs:namecache:lookup:miss - name cache miss, passed the parent directory
pointer and the full remaining component name (not terminated after the
cache miss component).

vfs:namecache:purge:done - name cache purge for a vnode, passed the vnode
pointer to purge.

vfs:namecache:purge_negative:done - name cache purge of negative entries
for children of a vnode, passed the vnode pointer to purge.

vfs:namecache:purgevfs - name cache purge for a mountpoint, passed the
mount pointer. Separate probes will also be invoked for each cache
entry zapped.

vfs:namecache:zap:done - name cache entry zapped, passed the parent
directory vnode pointer, name, and child vnode pointer.

vfs:namecache:zap_negative:done - negative name cache entry zapped,
passed the parent directory vnode pointer and name.

For any probes involving an extant name cache entry (enter, hit, zapp),
we use the nul-terminated string for the name component. For misses,
the remainder of the path, including later components, is provided as
an argument instead since there is no handy nul-terminated version of
the string around. This is arguably a bug.

MFC after: 1 month
Sponsored by: Google, Inc.
Reviewed by: jhb, kan, kib (earlier version)

Revert change 190655 temporarily. It breaks many setups where nullfs is
used and needs to be revisited.

vn_vptocnp() unlocks the name cache and forgets to re-lock it before
returning in one error case, and mistakenly unlocks it for the
umount -f case.

Replace v_dd vnode pointer with v_cache_dd pointer to struct namecache
in directory vnodes. Allow namecache dotdot entry to be created pointing
from child vnode to parent vnode if no existing links in opposite
direction exist. Use direct link from parent to child for dotdot lookups
otherwise.

This restores more efficient dotdot caching in NFS filesystems which
was lost when vnodes stoppped being type stable.

Reviewed by: kib

When a file lookup fails due to encountering a doomed vnode from a forced
unmount, consistently return ENOENT rather than EBADF.

Reviewed by: kib
MFC after: 1 month

Do not underflow the buffer and then report the problem. Check for the
condition before the buffer write.
Also, since buflen is unsigned, previous check was ignored.

Reviewed by: marcus
Tested by: pho

Remove unneeded braces to reduce used vertical screen space.
The location was missed in r190140.

Do not forget to adjust buflen for the first resolution of the path
from namecache.
While there, compare pointers for equiality.

Reviewed by: marcus
Tested by: pho

The nc_nlen member of the struct namecache contains the length of the cached
name, not the length + 1.

PR: 132620, 132542
Reported by: bf2006a yahoo com
Tested by: bf2006a, pho
Reviewed by: marcus

When ktracing namei operations, log a result of the __getcwd().

MFC after: 1 week

Remove unneeded braces to reduce used vertical screen space.

Move the debug.hashstat sysctl tree under DIAGNOSTIC. I measured the
debug.hashstat.rawnchash sysctl in particular as taking 7 milliseconds on
a 3GHz Intel Xeon (4x2) running 7.1. It accounted for almost a quarter of
the total runtime of 'sysctl -a'. It also performs lots of copyout's while
holding the namecache lock (this does not attempt to fix that).

MFC after: 2 weeks

Enable caching of negative pathname lookups in the NFS client. To avoid
stale entries, we save a copy of the directory's modification time when
the first negative cache entry was added in the directory's NFS node.
When a negative cache entry is hit during a pathname lookup, the parent
directory's modification time is checked. If it has changed, all of the
negative cache entries for that parent are purged and the lookup falls
back to using the RPC. This required adding a new cache_purge_negative()
method to the name cache to purge only negative cache entries for a given
directory.

Submitted by: mohans, Rick Macklem, Ricardo Labiaga @ NetApp
Reviewed by: mohans

Convert the global mutex protecting the directory lookup name cache from a
mutex to a reader/writer lock. Lookup operations first grab a read lock and
perform the lookup. If the operation results in a need to modify the cache,
then it tries to do an upgrade. If that fails, it drops the read lock,
obtains a write lock, and redoes the lookup.

- Mark all standalone INT/LONG/QUAD sysctl's MPSAFE. This is done
inside the SYSCTL() macros and thus does not need to be done for
all of the nodes scattered across the source tree.
- Mark the name-cache related sysctl's (including debug.hashstat.*) MPSAFE.
- Mark vm.loadavg MPSAFE.
- Remove GIANT_REQUIRED from vmtotal() (everything in this routine already
has sufficient locking) and mark vm.vmtotal MPSAFE.
- Mark the vm.stats.(sys|vm).* sysctls MPSAFE.

Add a limit on namecache entries.

In normal operation, the number of cache entries is roughly equal to the
number of active vnodes. However, when most of the recently accessed
vnodes have many hard links, the number of cache entries can be 32000
times as large, exhausting kernel memory and provoking a panic in
kmem_malloc().

MFC after: 2 weeks

In r185557, the check for existing negative entry for the given name
did not compared nc_dvp with supplied parent directory vnode pointer.
Add the check and note that now branches for vp != NULL and vp == NULL
are the same, thus can be merged.

Reported and reviewed by: kan
Tested by: pho
MFC after: 2 weeks

Do not KASSERT when vp->v_dd is NULL. Only directories which have had ".."
looked up would have v_dd set to a non-NULL value. This fixes a panic
seen when running installworld on a diskless system with a separate /usr
file system.

Submitted by: cracauer
Approved by: kib

Keep the hold on the vnode during VOP_VPTOCNP() call, allowing the vop
implementation to drop vnode lock, if needed.

Reported and tested by: pho

Add a new VOP, VOP_VPTOCNP, which translates a vnode to its component name
on a best-effort basis. Teach vn_fullpath to use this new VOP if a
regular VFS cache lookup fails. This VOP is designed to supplement the
VFS cache to provide a better chance that a vnode-to-name lookup will
succeed.

Currently, an implementation for devfs is being committed. The default
implementation is to return ENOENT.

A big thanks to kib for the mentorship on this, and to pho for running it
through his stress test suite.

Reviewed by: arch
Approved by: kib

Shared lookup makes it possible to create several negative cache
entries for one name. Then, creating inode with that name would remove
one entry, leaving others dormant. Reclaiming the vnode would uncover
negative entries, causing false return of ENOENT from the calls like
stat, that do not create inode.

Prevent creation of the duplicated negative entries.

Reported and debugged with: pho
Reviewed by: jhb
X-MFC: after shared lookup changes

Move vn_fullpath1() outside of FILEDESC locking. This is being done in
advance of teaching vn_fullpath1() how to query file systems for
vnode-to-name mappings when cache lookups fail.

Thanks to kib for guidance and patience on this process.

Reviewed by: kib
Approved by: kib