Copy stable/10@r272459 to releng/10.1 as part of
the 10.1-RELEASE process.

Approved by: re (implicit)
Sponsored by: The FreeBSD Foundation

MFC r262340, r262345, r262347, find and load an appropriate dtb file.

The search order for a usable dtb in fdt_setup_fdtp() is now

- A dtb loaded with an explicit "load -t dtb" command.
- A dtb already loaded into memory somehow[*] and pointed to by fdt_to_load.
- A dtb in the memory pointed to by the u-boot env vars fdtaddr or fdt_addr.
- A file named by the u-boot env vars fdtfile or fdt_file.
- A static dtb compiled into the kernel.

* Presumably by some arch-specific command or code.

Copy head (r256279) to stable/10 as part of the 10.0-RELEASE cycle.

Approved by: re (implicit)
Sponsored by: The FreeBSD Foundation

Fix the bug I introduced in r247045.
After digging through more carefully, it looks like there's
no real need to have the DTB in the module directory.
So we can simplify a lot: Just copy DTB into local heap
for "fdt addr" and U-Boot integration, drop all the extra
COPYIN() calls.

I've left one final COPYIN() to update the in-kernel DTB
for consistency with how this code used to work, but I'm
no longer convinced it's appropriate here.

I've also remove the mem_load_raw() utility that I added
to boot/common/module.c with r247045 since it's no longer
necessary.

Add mem_load_raw() for loading data from another location in memory.

This will be used by some upcoming changes to loader(8) FDT
handling to allow it to use an FDT provided by an earlier
boot stage the same as an FDT loaded from disk.

boot/console: handle consoles that fail to probe

- clarify meaning of console flags
- perform i/o via a console only if both of the following conditions are met:
o console is active (selected by user or config)
o console flags that it can perform the operation
- warn if a chosen console can not work (the warning may go nowhere without
working and active console, though)

Reviewed by: jhb
Tested by: Uffe Jakobsen <uffe@uffe.org>,
Olivier Cochard-Labbe' <olivier@cochard.me>
MFC after: 26 days

zfsboot/zfsloader: support accessing filesystems within a pool

In zfs loader zfs device name format now is "zfs:pool/fs",
fully qualified file path is "zfs:pool/fs:/path/to/file"
loader allows accessing files from various pools and filesystems as well
as changing currdev to a different pool/filesystem.

zfsboot accepts kernel/loader name in a format pool:fs:path/to/file or,
as before, pool:path/to/file; in the latter case a default filesystem
is used (pool root or bootfs). zfsboot passes guids of the selected
pool and dataset to zfsloader to be used as its defaults.

zfs support should be architecture independent and is provided
in a separate library, but architectures wishing to use this zfs support
still have to provide some glue code and their devdesc should be
compatible with zfs_devdesc.
arch_zfs_probe method is used to discover all disk devices that may
be part of ZFS pool(s).

libi386 unconditionally includes zfs support, but some zfs-specific
functions are stubbed out as weak symbols. The strong definitions
are provided in libzfsboot.
This change mean that the size of i386_devspec becomes larger
to match zfs_devspec.

Backward-compatibility shims are provided for recently added sparc64
zfs boot support. Currently that architecture still works the old
way and does not support the new features.

TODO:
- clear up pool root filesystem vs pool bootfs filesystem distinction
- update sparc64 support
- set vfs.root.mountfrom based on currdev (for zfs)

Mid-future TODO:
- loader sub-menu for selecting alternative boot environment

Distant future TODO:
- support accessing snapshots, using a snapshot as readonly root

Reviewed by: marius (sparc64),
Gavin Mu <gavin.mu@gmail.com> (sparc64)
Tested by: Florian Wagner <florian@wagner-flo.net> (x86),
marius (sparc64)
No objections: fs@, hackers@
MFC after: 1 month

sys/boot: add common CTASSERT definition

Add multiple inclusion protection.

PR: 165025
Submitted by: Gavin Mu
MFC after: 1 week

Add 2 new archsw interfaces:
1. arch_loadaddr - used by platform code to adjust the address at which
the object gets loaded. Implement PC98 using this new interface instead
of using conditional compilation. For ELF objects the ELF header is
passed as the data pointer. For raw files it's the filename. Note that
ELF objects are first considered as raw files.
2. arch_loadseg - used by platform code to keep track of actual segments,
so that (instruction) caches can be flushed or translations can be
created. Both the ELF header as well as the program header are passed
to allow platform code to treat the kernel proper differently from any
additional modules and to have all the relevant details of the loaded
segment (e.g. protection).

Revert rev 165325. The arch_maphint interface hasn't been in use for
more than 4 years.

Remove file system support based on the simple file system protocol
as this only allows us to access file systems that EFI knows about.
With a loader that can only use EFI-supported file systems, we're
forced to put /boot on the EFI system partition. This is suboptimal
in the following ways:
1. With /boot a symlink to /efi/boot, mergemaster complains about
the mismatch and there's no quick solution.
2. The EFI loader can only boot a single version of FreeBSD. There's
no way to install multiple versions of FreeBSD and select one
at the loader prompt.
3. ZFS maintains /boot/zfs/zpool.cache and with /boot a symlink we
end up with the file on a MSDOS file system. ZFS does not have
proper handling of file systems that are under Giant.

Implement a disk device based on the block I/O protocol instead and
pull in file system code from libstand. The disk devices are really
the partitions that EFI knows about.

This change is backward compatible.

MFC after: 1 week

Update ZFS from version 6 to 13 and bring some FreeBSD-specific changes.

This bring huge amount of changes, I'll enumerate only user-visible changes:

- Delegated Administration

Allows regular users to perform ZFS operations, like file system
creation, snapshot creation, etc.

- L2ARC

Level 2 cache for ZFS - allows to use additional disks for cache.
Huge performance improvements mostly for random read of mostly
static content.

- slog

Allow to use additional disks for ZFS Intent Log to speed up
operations like fsync(2).

- vfs.zfs.super_owner

Allows regular users to perform privileged operations on files stored
on ZFS file systems owned by him. Very careful with this one.

- chflags(2)

Not all the flags are supported. This still needs work.

- ZFSBoot

Support to boot off of ZFS pool. Not finished, AFAIK.

Submitted by: dfr

- Snapshot properties

- New failure modes

Before if write requested failed, system paniced. Now one
can select from one of three failure modes:
- panic - panic on write error
- wait - wait for disk to reappear
- continue - serve read requests if possible, block write requests

- Refquota, refreservation properties

Just quota and reservation properties, but don't count space consumed
by children file systems, clones and snapshots.

- Sparse volumes

ZVOLs that don't reserve space in the pool.

- External attributes

Compatible with extattr(2).

- NFSv4-ACLs

Not sure about the status, might not be complete yet.

Submitted by: trasz

- Creation-time properties

- Regression tests for zpool(8) command.

Obtained from: OpenSolaris

Add __elfN(relocation_offset). It holds the offset between the virtual
(link) address and the physical (load) address. Ideally, the mapping
between link and load addresses should be abstracted by the copyin(),
copyout() and readin() functions, so that we don't have to add kluges
in __elfN(loadimage)(). Then, we could also have paged virtual memory
for the kernel. This can be important under EFI, where you need to
allocate physical memory form the firmware if you want to work in all
scenarios.

add an interface for passing the entire kernel size up front to the
loader so that it can memory can be allocated aligned at the beginning of
the desired large page

Extend struct devdesc with a unit field, called d_unit. Promote the
device (kind) specific unit field to the common field. This change
allows a future version of libefi to work without requiring anything
more than what is defined in struct devdesc and as such makes it
possible to compile said version of libefi for different platforms
without requiring that those platforms have identical derivatives
of struct devdesc.

Make our ELF64 type definitions match standards. In particular this
means:
o Remove Elf64_Quarter,
o Redefine Elf64_Half to be 16-bit,
o Redefine Elf64_Word to be 32-bit,
o Add Elf64_Xword and Elf64_Sxword for 64-bit entities,
o Use Elf_Size in MI code to abstract the difference between
Elf32_Word and Elf64_Word.
o Add Elf_Ssize as the signed counterpart of Elf_Size.

MFC after: 2 weeks

Add the loader side of support for preloading ELF relocatable object
format modules, which are currently only used on the amd64 platform.
This initial implementation just parses enough of the module to
allow it to extract dependencies and load all the bits into the
right place in memory, so the kernel must still do the full relocation
and linking. The details of the loaded sections are passed to the
kernel by supplying a copy of the ELF section header table as module
metadata with the MODINFOMD_SHDR tag.

Separate out the ELF relocation code from the ELF loader, and add
better relocation support for the amd64 and i386 platforms. This
should not result in any change in functionality, but moves a step
towards supporting the relocatable object file modules on amd64.

The same hack/trick as load_elf*.c uses is used here to simultaneously
support both elf32 and elf64 on amd64 and i386.

Add a few helper functions for zeroing kernel space and reading
from specified file offsets. Make use of these in load_elf.c.

Enable the i386 loader to load and run an amd64 kernel. If this puts
things over floppy size limits, I can exclude it for release builds or
something like that. Most of the changes are to get the load_elf.c file
into a seperate elf32_ or elf64_ namespace so that you can have two
ELF loaders present at once. Note that for 64 bit kernels, it actually
starts up the kernel already in 64 bit mode with paging enabled. This
is really easy because we have a known minimum feature set.

Of note is that for amd64, we have to pass in the bios int 15 0xe821
memory map because once in long mode, you absolutely cannot make VM86
calls. amd64 does not use 'struct bootinfo' at all. It is a pure loader
metadata startup, just like sparc64 and powerpc. Much of the
infrastructure to support this was adapted from sparc64.

Zap some a.out leftovers

CG superfluous prototype.

Declare time(not3) instead of depending on namespace pollution 3 layers
deep in <stand.h> to eventually include <time.h> to declare the user
version.

This is not quite the right place to declare it, but <stand.h> would
be worse because time() is very MD so it isn't in libstand.

Many places in the boot sources still get the user version using only
1 layer of pollution (#include <sys/time.h>. Some pollute themselves
directly (#include <time.h>). But the boot Makefiles are too broken
to enable warnings for redeclarations.

Add a DEVT_CD type for CD drivers.

Implement the long-awaited module->file cache database. A userland
tool (kldxref(8)) keeps a cache of what modules and versions are inside
what .ko files. I have tested this on both Alpha and i386.

Submitted by: bp

Nuke old gensetdefs based linker sets with extreme prejudice

Make the pnp structure a global variable, so it can be used elsewhere.

Cleanup warnings. Most of these are signed/unsigned warnings, as well as
some added const's.

Delay calling the device cleanup routines until the absolute last
moment. We were cleaning up after PXE too early and the module
dependancy code would not be able to load any files if it needed
too.

Back out the previous change to the queue(3) interface.
It was not discussed and should probably not happen.

Requested by: msmith and others

Change the way that the queue(3) structures are declared; don't assume that
the type argument to *_HEAD and *_ENTRY is a struct.

Suggested by: phk
Reviewed by: phk
Approved by: mdodd

Update loader logic to distinguish modules vs. files.
Add support for module metadata. The old way of dependancy
handling will be supported for a while.

Reviewed by: peter

bcache_strategy() now receives an unit number, and keep track of what
was the last unit number received. If it changes, it flushes the cache.
Add bcache_flash().

The actual fix is sligthly different from the one in the PR.

PR: 17098
Submitted by: John Hood <jhood@sitaranetworks.com>

$Id$ -> $FreeBSD$

Walk around the end of all the silly guessing of device types and unit
numbers that we have been doing in the past, and read /etc/fstab off the
proposed root filesystem to determine the actual device name and vfs
type for the root filesystem. These are then exported to the kernel
via the environment variable vfs.root.mountfrom.

ABS_SET() and BSS_SET() are no longer used. The sole remaining user of
TEXT_SET() is DDB's command tables.

Bring the 'new-bus' to the i386. This extensively changes the way the
i386 platform boots, it is no longer ISA-centric, and is fully dynamic.
Most old drivers compile and run without modification via 'compatability
shims' to enable a smoother transition. eisa, isapnp and pccard* are
not yet using the new resource manager. Once fully converted, all drivers
will be loadable, including PCI and ISA.

(Some other changes appear to have snuck in, including a port of Soren's
ATA driver to the Alpha. Soren, back this out if you need to.)

This is a checkpoint of work-in-progress, but is quite functional.

The bulk of the work was done over the last few years by Doug Rabson and
Garrett Wollman.

Approved by: core

Make use of prototypes to silence warnings.

Change include() so it will be able to load files with forth code,
instead of just builtins. Remove #@- from the include section of the
help file, since they don't work in the new version of include, unless
BOOT_FORTH is not defined.

Change bf_run() so it will return the result returned by ficlExec(). Also,
make bf_run() push "interpret" to be executed by ficlExec(), since ficlExec()
doesn't do it by itself. (Things worked previously because nothing
recursed through ficlExec() by the way of bf_run()).

Change/extend comments on builtin behavior.

Search for "interpret" at the end of bf_init(), so /boot/boot.4th can
provide it's own version.

Remove dead code.

bootstrap.h
help.common
interp.c
Rename the 'source' command to 'include' in order to avoid conflict
with the ANS Forth command of the same name. (kern/9473)

interp_forth.c:
Changes from kern/9412 (EXCEPTION word), kern/9442 (TIB buffer
sizing) and an improved version of kern/9460 (set
version numbers).

load_aout.c:
Trim some obsolete #if 0'ed cruft.

pnp.c:
Tidy the pnpscan output, turn off the module scanning until we
sort out how to do it right.

PR: kern/9412 kern/9442 kern/9460 kern/9473
Submitted by: PRs from Daniel Sobral <dcs@newsguy.com>

Replace duplicated "old" linker set definitions with an include of
<sys/linker_set.h>. Note, this isn't used on the i386.

Fall back to /boot/boot.conf, if /boot/loader.rc was not fount. This is
to cover the transition period.

Suggested by: msmith

Implement a simple LRU block cache. By default this is initialised to 16k,
and will bypass transfers for more than 8k. Blocks are invalidated after
2 seconds, so removable media should not confuse the cache.

The 8k threshold is a compromise; all UFS transfers performed by
libstand are 8k or less, so large file reads thrash the cache.
However many filesystem metadata operations are also performed using
8k blocks, so using a lower threshold gives poor performance.

Those of you with an eye for cache algorithms are welcome to tell me
how badly this one sucks; you can start with the 'bcachestats' command
which will print the contents of the cache and access statistics.

Export an interface for presetting the ISA PnP read port address.
Make the EISA ID formatter generally available

Make ISA PnP work. This successfully enumerates as many ISA devices as I
can fit into my test machine.

- Move to using STAILQs rather than ad-hoc singly-linked lists.
- Use a mostly procedural interface to the PnP information. This
improves data-hiding.

Implement a new linker-set technique (currently on i386 only but should work
on Alpha as well). This is a good candidate for replacing the current
gensetdefs cruft completely.

loader <-> kernel preload module interface constants moved to common file.
Remove Mike's explicit data structures for dependency info. This is
done via DT_NEEDED etc in the dynamic section for now. This may need
to be revisited later on.

Sync the MODINFO constants with <sys/linker.h>
Remove debugging in command_read().
Correctly strip leading controls on script commands.
Make 'ls' more DWIM in regard to pathnames. We can still do better.

- VERBOSE_LS is obsolete, as the heap is much better behaved now.
- Don't whine about nodes we can't stat(); these are usually
symlinks that lead out of the filesystem.
- Autoboot is now controlled by $autoboot_delay, which is a value
in seconds or NO to disable autoboot.
- Don't autoboot at the end of boot.conf if we have already tried.
- Add a 'read' command to complement 'echo'. Both are still hidden.
- Improve the 'source' command/function so that it is possible to
source scripts off removable media. The entire script is read and
saved before beginning execution. Script lines beginning with
'@' will not be echoed when being executed. Script execution will
normally terminate at the first error, however if the script line
begins with '-' this behaviour is overriden for that command.

First shot at loading elf symbols. Things are a bit strange because
of the ..umm.. "wierd" way binutils lays out the file. The section
headers are nearly at the end of the file and this is a problem when
loading from a .gz file which can't seek backwards (or has a limited
reverse seek, ~2K from memory).

This is intended to be compatable with the ddb/db_elf.c code and the
alpha/libalpha/elf_freebsd.c layout. I've studied these (which are NetBSD
derived) but did it a bit differently. Naturally the process is similar
since it's supposed to end up with the same result.

Uncomment prototype for elf_loadmodule

* Add old UFS compatibility code to alpha/boot1.
* Fix a raft of warnings, printf and otherwise.
* Allocate the correct amount in mod_searchmodule to prevent an overflow.
* Fix the makefiles so they work outside my home directory (oops).

Make the alpha bootstrap build again, fix some warning and change sdboot to daboot.

Add a generic hexdump tool for debugging purposes.

Resynch with working sources before BTX integration.

- Use format-independant module allocator.
- Conditionalise ISA PnP support.
- Simplify PnP enumerator interface.
- Improve module/object searching.
- Add missing depend/install targets in BTX makefiles.
- Pass the kernel environment and module data in extended bootinfo fields.
- Add a pointer to the end of the kernel + modules in bootinfo.
- Fix parsing of old-style kernel arguments.

Generic plug-and-play enumerator infrastructure. Query supplied
enumerators, crossreference returned identifiers with a text-format
database and automatically load corresponding modules and dependancies.

Bootstrap updates.

- Move some startup code from MD to MI sections
- Add a 'copyout' and some copyout-related functions. These will be
obsoleted when BTX is available for the 386 and the kernel load
area becomes directly addressable.
- Add the ability load an arbitrary file as a module, associating
and arbitrary type string with it. This can be used eg. for loading
splash-screen images etc.
- Add KLD module dependancy infrastructure. We know how to look for
dependancies inside KLD modules, how to resolve these dependancies
and what to do if things go wrong. Only works for a.out at the
moment, due to lack of an MI ELF loader. Attach KLD module information
to loaded modules as metadata, but don't pass it to the kernel (it
can find it itself).
- Load a.out KLD modules on a page boundary. Only pad the a.out BSS
for the kernel, as it may want to throw symbols away. (We might want
to do this for KLD modules too.)
- Allow commands to be hidden from the '?' display, to avoid cluttering
it with things like 'echo'. Add 'echo'.
- Bring the 'prompt' command into line with the parser syntax.
- Fix the verbose 'ls'; it was using an uninitialised stack variable.
- Add a '-v' flag to 'lsmod' to have it display module metadata as well
(not terribly useful for the average user)
- Support a 'module searchpath' for required modules.
- The bootstrap file on i386 is now called 'loader' to permit the
/boot directory to use that name.
- Discard the old i386 pread() function, as it's replaced by
arch_readin()

Bootloader update.

- Implement a new copyin/readin interface for loading modules.
This allows the module loaders to become MI, reducing code duplication.
- Simplify the search for an image activator for the loaded kernel.
- Use the common module management code for all module metadata.
- Add an 'unload' command that throws everything away.
- Move the a.out module loader to MI code, add support for a.out
kld modules.

Submitted by: Alpha changes fixed by Doug Rabson <dfr@freebsd.org>

This commit was generated by cvs2svn to compensate for changes in r38465,
which included commits to RCS files with non-trunk default branches.

This is the new unified bootstrap, sometimes known previously as the
'three-stage' bootstrap.
There are a number of caveats with the code in its current state:
- The i386 bootstrap only supports booting from a floppy.
- The kernel and kld do not yet know how to deal with the extended
information and module summary passed in.
- PnP-based autodetection and demand loading of modules is not implemented.
- i386 ELF kernel loading is not ready yet.
- The i386 bootstrap is loaded via an ugly blockmap.

On the alpha, both net- and disk-booting (SRM console machines only) is
supported. No blockmaps are used by this code.

Obtained from: Parts from the NetBSD/i386 standalone bootstrap.