Copy stable/10@r272459 to releng/10.1 as part of
the 10.1-RELEASE process.

Approved by: re (implicit)
Sponsored by: The FreeBSD Foundation

Copy head (r256279) to stable/10 as part of the 10.0-RELEASE cycle.

Approved by: re (implicit)
Sponsored by: The FreeBSD Foundation

Add currently unused flag argument to the cluster_read(),
cluster_write() and cluster_wbuild() functions. The flags to be
allowed are a subset of the GB_* flags for getblk().

Sponsored by: The FreeBSD Foundation
Tested by: pho

Remove fifo.h. The only used function declaration from the header is
migrated to sys/vnode.h.

Submitted by: gianni

Fix found places where uio_resid is truncated to int.

Add the sysctl debug.iosize_max_clamp, enabled by default. Setting the
sysctl to zero allows to perform the SSIZE_MAX-sized i/o requests from
the usermode.

Discussed with: bde, das (previous versions)
MFC after: 1 month

Mark MALLOC_DEFINEs static that have no corresponding MALLOC_DECLAREs.

This means that their use is restricted to a single C file.

udf_vnops: cosmetic followup to r208671 - better looking code

Suggested by: jhb
MFC after: 3 days

udf_readlink: fix malloc call with uninitialized size parameter

Found by: clang static analyzer
MFC after: 4 days

udf: use truly unique directory cookie

'off' is an offset within current block, so there is a good chance
it can be non-unique, so use complete offset.

Submitted by: bde
Approved by: jhb

udf_strategy: remove redundant comment

We fail mapping for any udf_bmap_internal error and there can be
different reasons for it, so no need to (over-)emphasize files with
data in fentry.

Submitted by: bde
Approved by: jhb

udf_readdir: do not advance offset if entry can not be uio-ed

Previosly readdir missed some directory entries because there was
no space for them in current uio but directory stream offset
was advanced nevertheless.
jhb has discoved the issue and provided a test-case.

Reviewed by: bde
Approved by: jhb (mentor)

udf_readatoffset: return correct size and data pointer for data in fentry

This should help correct reading of directories with data located
in fentry.

Submitted by: bde
Approved by: jhb (mentor)

udf_readatoffset: read through directory vnode, do not read > MAXBSIZE

Currently bread()-ing through device vnode with
(1) VMIO enabled,
(2) bo_bsize != DEV_BSIZE
(3) more than 1 block
results in data being incorrectly cached.
So instead a more common approach of using a vnode belonging to fs is now
employed.
Also, prevent attempt to bread more than MAXBSIZE bytes because of
adjustments made to account for offset that doesn't start on block
boundary.
Add expanded comments to explain the calculations.
Also drop unused inline function while here.

PR: kern/120967
PR: kern/129084

Reviewed by: scottl, kib
Approved by: jhb (mentor)

udf: add read-ahead support modeled after cd9660

Reviewed by: scottl
Approved by: jhb (mentor)

udf_map: return proper error code instead of leaking an internal one

Incidentally this also allows for small files with data embedded into
fentry to be mmap-ed.

Approved by: jhb (mentor)

udf_read: correctly read data from files with data embedded into fentry,

... as opposed to files with data in extents.
Some UDF authoring tools produce this type of file for sufficiently small
data files.

Approved by: jhb (mentor)

udf_strategy: tiny optimization of logic, calculations; extra diagnostics

Use bit-shift instead of division/multiplication.
Act on error as soon as it is detected.
Report attempt to read data embedded in file entry via regular way.
While there, fix lblktosize macro and make use of it.

No functionality should change as a result.

Approved by: jhb (mentor)

style nit in r188815

Pointed out by: jhb, rpaulo
Approved by: jhb (mentor)

fs/udf: fix incorrect error return (-1) when reading a large dir

Not enough space in user-land buffer is not an error, userland
will read further until eof is reached. So instead of propagating
-1 to caller we convert it to zero/success.

cd9660 code works exactly the same way.

PR: kern/78987
Reviewed by: jhb (mentor)
Approved by: jhb (mentor)

Mark udf(4) MPSAFE and add support for shared vnode locks during pathname
lookups:
- Honor the caller's locking flags in udf_root() and udf_vget().
- Set VV_ROOT for the root vnode in udf_vget() instead of only doing it in
udf_root().
- Honor the requested locking flags during pathname lookups in udf_lookup().
- Release the buffer holding the directory data before looking up the vnode
for a given file to avoid a LOR between the "udf" vnode locks and
"bufwait".
- Use vn_vget_ino() to handle ".." lookups.
- Special case "." lookups instead of calling udf_vget(). We have to do
extra checking for the vnode lock for "." lookups.

Add rudimentary support for symbolic links on UDF. Links are stored as a
sequence of pathname components. We walk the list building a string in
the caller's passed in buffer. Currently this only handles path names
in CS8 (character set 8) as that is what mkisofs generates for UDF images.

MFC after: 1 month

Add support for fifos to UDF:
- Add a separate set of vnode operations that inherits from the fifo ops
and use it for fifo nodes.
- Add a VOP_SETATTR() method that allows setting the size (by silently
ignoring the requests) of fifos. This is to allow O_TRUNC opens of
fifo devices (e.g. I/O redirection in shells using ">").
- Add a VOP_PRINT() handler while I'm here.

Remove unused local variables.

Submitted by: Christoph Mallon christoph.mallon@gmx.de
Reviewed by: kib
MFC after: 2 weeks

Introduce accmode_t. This is required for NFSv4 ACLs - it will be neccessary
to add more V* constants, and the variables changed by this patch were often
being assigned to mode_t variables, which is 16 bit.

Approved by: rwatson (mentor)

Retire the MALLOC and FREE macros. They are an abomination unto style(9).

MFC after: 3 months

Initialize va_rdev to NODEV instead of 0 or VNOVAL in VOP_GETATTR().
NODEV is more appropriate when va_rdev doesn't have a meaningful value.

Submitted by: Jaakko Heinonen <jh saunalahti fi>
Suggested by: bde
Discussed on: freebsd-fs
MFC after: 1 month

Fix and speedup timestamp calculations which is roughly based on the patch in
the mentioned PR:

- bounds check time->month as it is used as an array index
- fix usage of time->month as array index (month is 1-12)
- fix calculation based on time->day (day is 1-31)
- fix the speedup code as it doesn't calculate correct timestamps before
the year 2000 and reduce the number of calculation in the year-by-year code
- speedup month calculations by replacing the array content with cumulative
values
- add microseconds calculation
- fix an endian problem

PR: kern/97786
Submitted by: Andriy Gapon <avg@topspin.kiev.ua>
Reviewed by: scottl (earlier version)
Approved by: emax (mentor)
MFC after: 1 week

VOP_LOCK1() (and so VOP_LOCK()) and VOP_UNLOCK() are only used in
conjuction with 'thread' argument passing which is always curthread.
Remove the unuseful extra-argument and pass explicitly curthread to lower
layer functions, when necessary.

KPI results broken by this change, which should affect several ports, so
version bumping and manpage update will be further committed.

Tested by: kris, pho, Diego Sardina <siarodx at gmail dot com>

vn_lock() is currently only used with the 'curthread' passed as argument.
Remove this argument and pass curthread directly to underlying
VOP_LOCK1() VFS method. This modify makes the code cleaner and in
particular remove an annoying dependence helping next lockmgr() cleanup.
KPI results, obviously, changed.

Manpage and FreeBSD_version will be updated through further commits.

As a side note, would be valuable to say that next commits will address
a similar cleanup about VFS methods, in particular vop_lock1 and
vop_unlock.

Tested by: Diego Sardina <siarodx at gmail dot com>,
Andrea Di Pasquale <whyx dot it at gmail dot com>

Correct corrupt read when the read starts at a non-aligned offset.

PR: kern/77234
MFC After: 1 week
Approved by: imp (mentor)
Requested by: many many people
Submitted by: Andriy Gapon <avg at icyb dot net dot ua>

o cd9660 code repo-copied, update a comment.

Move vnode-to-file-handle translation from vfs_vptofh to vop_vptofh method.
This way we may support multiple structures in v_data vnode field within
one file system without using black magic.

Vnode-to-file-handle should be VOP in the first place, but was made VFS
operation to keep interface as compatible as possible with SUN's VFS.
BTW. Now Solaris also implements vnode-to-file-handle as VOP operation.

VFS_VPTOFH() was left for API backward compatibility, but is marked for
removal before 8.0-RELEASE.

Approved by: mckusick
Discussed with: many (on IRC)
Tested with: ufs, msdosfs, cd9660, nullfs and zfs

Rewrite the udf_read() routine to use a file vnode instead of the devvp vnode.
The code is modelled after cd9660, including support for simple read-ahead
courtesy of clustered read.

Fix udf_strategy to DTRT.

This change fixes sendfile(2) not to send out garbage.

Reviewed by: scottl
MFC after: 1 month

Call vnode_create_vobject() in VOP_OPEN. Makes mmap work on UDF filesystem.

PR: kern/92040
Approved by: scottl
MFC after: 1 week

Make UDF endian-safe.

Submitted by: Pedro Martelletto <pedro@ambientworks.net> (via scottl)
Tested on: sparc64

I ran into an nfs client panic a couple of times in a row over the
last few days. I tracked it down to the fact that nfs_reclaim()
is setting vp->v_data to NULL _before_ calling vnode_destroy_object().
After silence from the mailing list I checked further and discovered
that ufs_reclaim() is unique among FreeBSD filesystems for calling
vnode_destroy_object() early, long before tossing v_data or much
of anything else, for that matter. The rest, including NFS, appear
to be identical, as if they were just clones of one original routine.

The enclosed patch fixes all file systems in essentially the same
way, by moving the call to vnode_destroy_object() to early in the
routine (before the call to vfs_hash_remove(), if any). I have
only tested NFS, but I've now run for over eighteen hours with the
patch where I wouldn't get past four or five without it.

Submitted by: Frank Mayhar
Requested by: Mohan Srinivasan
MFC After: 1 week

Normalize a significant number of kernel malloc type names:

- Prefer '_' to ' ', as it results in more easily parsed results in
memory monitoring tools such as vmstat.

- Remove punctuation that is incompatible with using memory type names
as file names, such as '/' characters.

- Disambiguate some collisions by adding subsystem prefixes to some
memory types.

- Generally prefer lower case to upper case.

- If the same type is defined in multiple architecture directories,
attempt to use the same name in additional cases.

Not all instances were caught in this change, so more work is required to
finish this conversion. Similar changes are required for UMA zone names.

Regrab dvp only when ISDOTDOT.

Approved by: re (scottl)

Avoid casting from (int *) to (size_t *) in order to fix udf_iconv on amd64.

Reviewed by: scottl
MFC after: 2 weeks

Fix three typos in comments. Two of them obtained from OpenBSD.

MFC after: 3 days

- Change all filesystems and vfs_cache to relock the dvp once the child is
locked in the ISDOTDOT case. Se vfs_lookup.c r1.79 for details.

Sponsored by: Isilon Systems, Inc.

- We no longer have to bother with PDIRUNLOCK, lookup() handles it for us.
- In the ISDOTDOT case we have to unlock the dvp before locking the child,
if this fails we must relock dvp before returning an error. This was
missing before.

Sponsored by: Isilon Systems, Inc.

Also remember to set the fsid here.

Don't hold a reference to the disk vnode for each inode.

Eliminate cdev and vnode pointer to the disk from the inodes,
the mount holds everything we need.

Use vfs_hash instead of home-rolled.

Correct locking around g_vfs_close()

Make filesystems get rid of their own vnodes vnode_pager object in
VOP_RECLAIM().

Fix an incorrect cast.

Submitted by: Andriy Gapon
MFC-after: 3 days.

NULL-terminate the . and .. directory entries. Apparently some tools ignore
d_namlen and assume that d_name is null-terminated.

Submitted by: Andriy Gapon

Replace the min() macro with a test that doesn't truncate the 64-bit values
that are used. Thanks to Bruce Evans for pointing this out.

Whitespace in vop_vector{} initializations.

Use off_t when passing and calculating file offsets. While a single
extent in UDF is only 32 bits, multiple extents can exist in a file.
Also clean up some minor whitespace problems.

Submitted by: John Wehle

Don't allow reads past the end of a file.

Submitted by: John Wehle, Andriy Gapon
MFC After: 3 days

Wrap the bufobj operations in macros: BO_STRATEGY() and BO_WRITE()

Back when VOP_* was introduced, we did not have new-style struct
initializations but we did have lofty goals and big ideals.

Adjust to more contemporary circumstances and gain type checking.

Replace the entire vop_t frobbing thing with properly typed
structures. The only casualty is that we can not add a new
VOP_ method with a loadable module. History has not given
us reason to belive this would ever be feasible in the the
first place.

Eliminate in toto VOCALL(), vop_t, VNODEOP_SET() etc.

Give coda correct prototypes and function definitions for
all vop_()s.

Generate a bit more data from the vnode_if.src file: a
struct vop_vector and protype typedefs for all vop methods.

Add a new vop_bypass() and make vop_default be a pointer
to another struct vop_vector.

Remove a lot of vfs_init since vop_vector is ready to use
from the compiler.

Cast various vop_mumble() to void * with uppercase name,
for instance VOP_PANIC, VOP_NULL etc.

Implement VCALL() by making vdesc_offset the offsetof() the
relevant function pointer in vop_vector. This is disgusting
but since the code is generated by a script comparatively
safe. The alternative for nullfs etc. would be much worse.

Fix up all vnode method vectors to remove casts so they
become typesafe. (The bulk of this is generated by scripts)

Mechanically change prototypes for vnode operations to use the new typedefs.

Make VOP_BMAP return a struct bufobj for the underlying storage device
instead of a vnode for it.

The vnode_pager does not and should not have any interest in what
the filesystem uses for backend.

(vfs_cluster doesn't use the backing store argument.)

Move UDF to GEOM backing instead of DEVFS.

For details, please see src/sys/ufs/ffs/ffs_vfsops.c 1.250.

Eliminate unnecessary KASSERTs.

Don't use bp->b_vp in VOP_STRATEGY: the vnode is passed in as an argument.

Make the udf_vnops side endian clean.

First half of making UDF be endian-clean. This addresses the vfsops side.

Catch all cases where bread() returns an error and a valid *bp, and release
the *bp.

Obtained from: DragonFlyBSD

Change __FUNCTION__ to __func__

Submitted by: Stefan Farfeleder

Re-arrange and consolidate some random debugging stuff

Add hooks for translating directories entries using the iconv methods.

Submitted by: imura@ryu16.org

DuH!

bp->b_iooffset (the spot on the disk), not bp->b_offset (the offset in
the file)

Initialize b_offset before calling VOP_STRATEGY/VOP_SPECSTRATEGY.

Remove various comments of KASSERTS and comments about B_PHYS which
does not apply anymore.

- Don't cache_purge() in *_reclaim routines. vclean() does it for us so
this is redundant.

Add the same KASSERT to all VOP_STRATEGY and VOP_SPECSTRATEGY implementations
to check that the buffer points to the correct vnode.

Correctly calculate the size of the extent that should be read in
udf_readatoffset(). This should fixe problems with reading udf filesystems
created with mkisofs.

Implement the node cache as a hash table.

Make udf_allocv() return an unlocked vnode instead of a locked one
to avoid a "locking against myself" panic when udf_hashins() tries
to lock it again. Lock the vnode in udf_hashins() before adding it to
the hash bucket.

Finish cleanup of vprint() which was begun with changing v_tag to a string.
Remove extraneous uses of vop_null, instead defering to the default op.
Rename vnode type "vfs" to the more descriptive "syncer".
Fix formatting for various filesystems that use vop_print.

Clean up whitespace, s/register //, refrain from strong urge to ANSIfy.

uiomove-related caddr_t -> void * (just the low-hanging fruit)

Back out M_* changes, per decision of the TRB.

Approved by: trb

Remove M_TRYWAIT/M_WAITOK/M_WAIT. Callers should use 0.
Merge M_NOWAIT/M_DONTWAIT into a single flag M_NOWAIT.

Temporarily introduce a new VOP_SPECSTRATEGY operation while I try
to sort out disk-io from file-io in the vm/buffer/filesystem space.

The intent is to sort VOP_STRATEGY calls into those which operate
on "real" vnodes and those which operate on VCHR vnodes. For
the latter kind, the call will be changed to VOP_SPECSTRATEGY,
possibly conditionally for those places where dual-use happens.

Add a default VOP_SPECSTRATEGY method which will call the normal
VOP_STRATEGY. First time it is called it will print debugging
information. This will only happen if a normal vnode is passed
to VOP_SPECSTRATEGY by mistake.

Add a real VOP_SPECSTRATEGY in specfs, which does what VOP_STRATEGY
does on a VCHR vnode today.

Add a new VOP_STRATEGY method in specfs to catch instances where
the conversion to VOP_SPECSTRATEGY has not yet happened. Handle
the request just like we always did, but first time called print
debugging information.

Apart up to two instances of console messages per boot, this amounts
to a glorified no-op commit.

If you get any of the messages on your console I would very much
like a copy of them mailed to phk@freebsd.org

Since Jeffr made the std* functions the default in rev 1.63 of
kern/vfs_defaults.c it is wrong for the individual filesystems to use
the std* functions as that prevents override of the default.

Found by: src/tools/tools/vop_table

Regularize the vop_stdlock'ing protocol across all the filesystems
that use it. Specifically, vop_stdlock uses the lock pointed to by
vp->v_vnlock. By default, getnewvnode sets up vp->v_vnlock to
reference vp->v_lock. Filesystems that wish to use the default
do not need to allocate a lock at the front of their node structure
(as some still did) or do a lockinit. They can simply start using
vn_lock/VOP_UNLOCK. Filesystems that wish to manage their own locks,
but still use the vop_stdlock functions (such as nullfs) can simply
replace vp->v_vnlock with a pointer to the lock that they wish to
have used for the vnode. Such filesystems are responsible for
setting the vp->v_vnlock back to the default in their vop_reclaim
routine (e.g., vp->v_vnlock = &vp->v_lock).

In theory, this set of changes cleans up the existing filesystem
lock interface and should have no function change to the existing
locking scheme.

Sponsored by: DARPA & NAI Labs.

Return ENOTTY on unrecognized ioctls.

Remove all use of vnode->v_tag, replacing with appropriate substitutes.
v_tag is now const char * and should only be used for debugging.

Additionally:
1. All users of VT_NTS now check vfsconf->vf_type VFCF_NETWORK
2. The user of VT_PROCFS now checks for the new flag VV_PROCDEP, which
is propagated by pseudofs to all child vnodes if the fs sets PFS_PROCDEP.

Suggested by: phk
Reviewed by: bde, rwatson (earlier version)

Remove the possibility of a race condition when reading the . and ..
entries.

Don't abuse the stack when translating names.

Clean up comments that are no longer relevant.

Factor out some ugle code that's shared by udf_readdir and udf_lookup.
Significantly de-obfuscate udf_lookup

Inspired By: tes@sgi.com

Simplify the handling of a fragmented file_id descriptor. Also
de-obfuscate the file_char flags.

Calculate the correct physical block number for files that are
embedded into their file_entry descriptor. This is more for
correctness, since these files cannot be bmap'ed/mmap'ed anyways.
Enforce this restriction.

Submitted by: tes@sgi.com

Check for deleted files in udf_lookup(), not just udf_readdir().

Submitted by: tes@sgi.com

In VOP_LOOKUP, don't assume that the final pathname component
will be in the same filesystem than the one where the current
component is.

Approved by: scottl

Make daddr_t and u_daddr_t 64bits wide.
Retire daddr64_t and use daddr_t instead.

Sponsored by: DARPA & NAI Labs.

Don't reference vop_std* since they are already implicitly
referenced through the VOP_DEFAULT vector

Submitted by: phk

In udf_bmap(), return the physical block number, not the logical
block number. This fixes things like cp (ouch!) which use mmap.

Fix udf_read(). Honor the uio_resid when determining the size of
the block to read and copy out. This removes the hack in
udf_readatoffset() for only reading one block at a time. WooHoo!
Remove a redundant test for fragmented fids in both udf_readdir()
and udf_lookup(). Add comment to both as to why the test is
written the way it is. Add a few more safety checks for brelse().

Thanks to Timothy Shimmin <tes@boing.melbourne.sgi.com> for pointing
out these problems.

Sync with UDF p4 tree: Use POSIX integer types instead of BSD types.

Actually add the UDF files!