Move MIPS N64 compat out of {base,debug}32 into {base,debug}64

martin requested separation by ABI and it doesn't make much sense
to have 64-bit binaries in a set called 32.

Add new sets: base32, debug32, manhtml

- base32 contains (when MKCOMPAT=yes) shared libraries for 32-bit
compatibility, previously included in base

- debug32 contains (when MKCOMPAT=yes) debug symbols and static libraries
containing debug symbols for 32-bit compatiblity, previously included
in debug

- manhtml contains (when MKHTML=yes) the HTML files previously included
in 'man', which are of limited utility without third-party software.

The motivation for this change is to be able to easily exclude sets
from CD-ROM images that go over the size limit without xz compression
(which many NetBSD platforms struggle to extract at acceptable speeds).

Add a https transfer method (sharing all host details with the http
transfer). Make this method the default for all downloads.

Try to make sure the binary pkg installation (which runs in a chroot
in the already installed system) can make use of SSL verification.

This does NOT fix the missing SSL verification in most install media
even if using https (due to not fully populated /etc/openssl/certs).

Correct a minor word usage error in a comment. NFC.

branches: 1.88.2;
Add an (expert) option to manually edit partitions, inspired by PR 57100.

Avoid C++-isms (stupid muscle memory) to fix the build.

Preparations to make the extended partitioning during normal installation
more usefull - pass an optional install_description to partman().

Check on-disk disklabel properly even on ports without raw BSD disklabel.

Fixes PR install/56890.

Centralize the "we can newfs this type of filesystem" predicate
and fix the list.

fix various small typos, mainly in comments.

When not invoked via some explicit "configure network" menu item,
auto-detect an already working network setup and offer to "just use" it
whenver we need connectivity.

Make default installations a bit more secure:
- push a bit harder for setup of a root password
- simplify/clarify manual entropy setup
- visit the entropy setup (on machines where it is needed) on the
default path through sysinst

x86: fix previous: in the UEFI case copy the bootloaders from install
media during initial installation, but use the (by then: updated)
files from the target disk for system upgrades.

Do not defer wedge deletions when we have nothing mounted.
Define and use a constant for maximum disk name length.

When upgrading, update the boot code post extraction from
the updated target file system.

Try to get rid of all wedges we created (after unmounting).

s/exisit/exist/ in comment.

Re-enable the entropy handling code in sysinst (only visible on machines
that do not have full entropy), but move it to the config menu
instead of enforcing it as mandatory step.

This menu is shown at the end of the setup, or if explicitly invoked
from the main menu.

Some of the input options are complex but useful in some situations
and code to support them is tiny. Most users will use the manual input
(first + default option) but some might prefer to connect a USB TRNG
or have easy setups to transfer entropy from another machine (while
copy & paste on a serial console sometimes is tricky).

Restore MKNOUVEAUFIRMWARE and MKRADEONFIRMWARE and make gpufw set unconditional

Simplifies logic.
(Second commit - first one was partial)

Restoring MK* requested by mrg on tech-kern discussion
https://mail-index.netbsd.org/tech-kern/2021/09/25/msg027695.html

Move DRM-driver firmware from base to its own set, gpufw

This set is only installed on amd64,i386,evbarm.
This set is installed on minimal installs and on install media, in
case someone needs it for basic driver functionality.

Comments:
Switched to a single MK tunable for it - that is probably unneeded.

An upcoming DRM update will include even fatter firmware, and we'd
like to minimize the impact of it.

PR 56354: all actions to set up swap space are not guaranteed to gain
us enough virtual memory anyway, so drop return codes from set_swap*.
The state for cleanup (which swap dev to unuse) has been made global
some time ago anyway.

Previously use of the return values was inconsistent. Error reporting
will only confuse users and sometimes the situation is hard to fix or
even impossible (like in miniroots copide to swap space for booting).

PR install/56303: when using xz files enable swap during set extraction
if the machine does not have more than 256MB of RAM.

sysinst: remove trailing whitespace from *.c *.h

In contrast to the messages files, this whitespace is not significant.

Add (experimental) entropy input support:
Early during new installs or after upgrades we check if entropy is
available. If not (no hardware random number generator available)
we inform the user and ask them to fix it.

PR 55769: avoid the module set if we do not build any

PR 55142: on popular demand bring back expert options to adjust the
number of free inodes, block size and fragment size for FFS and LFS.

PR 55536: when we find existing partition tables and have alternative
formats available, offer to delete partitions and create new ones from
scratch (in some other or the same on-disk format).

When MD code requests a reserved area (at the start of the disk)
make sure to properly account for the "lost" space during all calculations
and especially when expanding the main partition to fill the disk.

Avoid using the global "pm" or a passed device descriptons when we are
passing disk limits explicitly anyway.

Simplify/fix a few rounding calculations.

PR 55381: try to deal with moved (but not yet saved) partitions when
calculating free space

Keep a list of all partition tables from which we have deleted partitions
and make sure we update those partition tables when writing partitions,
even if no install related partition remains on them.

Separate devicetree .dts -> .dtb building from kernel builds. They are now
part of a separate set, "dtb.tgz", and only built when MKDTB=yes. This
defaults to yes for earmv[67]* and aarch64, and no everywhere else.

PR 55202: disable swapping if we started it due to low ram

Ooops, backout previous - twice the entropy saving is not needed.
Instead slightly improve the (slightly) hidden other code that already
did it.

At the end of a new installation save entropy.

Don't depend on common symbol definitions.

PR misc/54886: bump threshold for automatic/default creation of a tmpfs /tmp
up slightly (to 384 MB ram). This will make sure the default install has
a > 64 MB /tmp available (number pulled out of thin air, 64 MB is the minimum
required by the ZFS tests).

Remove unused "rel" global variable and make the -r option a no-op.

Do not use -P for tar extractions - the in tree tar has been fixed.

Fix support for non-512-byte/sector disks again after I broke it when
introducing the abstract partition backends.

Factor out all RAM size thresholds as defines to avoid magic numbers.
To work around PR misc/54886 bump the threshold for a tmpfs /tmp mount
up to 256 MB.

First try to bring evbarm installation closer to current reality.

Fix /tmp tmpfs handling and on machines with enough RAM default to
creating a /tmp tmpfs with 25% of ram size limit. Suggested by ad@.

Finish conversion of extended partitioning parts to new abstract
backend interface. XXX still could use a lot of polishing.

Get rid of the evbarm preliminary menu: do not bother to ask the user
whether this is a RPi - query the FDT instead.

Make tar extraction flags depend on our usage of pax-as-tar or bsdtar.

Make cloning support optional, so we can save some space on very small
install media.

Add options to the various partitioning stages that allow cloning of
alien partitions (optionally including data).

Split out /rescue to its own set and adapt installers/images to add it.

This is meant to make updates safer: if something goes wrong with updating
base, we still have the old, standalone /rescue to recover from.

Do not offer to upgared the "current system" if we are running off a CD
(i.e. / is mounted read-only)

Support upgrade of systems using NAME= syntax in /etc/fstab.

Make supported file system types dynamic - instead of hardcoding the
available types at compile time, check for available newfs_* helper
binaries in the actual install environment at runtime.

branches: 1.42.2;
Strip trailing / from last mounted strings.
No idea how they happen, but for cgd root (init.root = "/altroot")
they have been reported to exist.

For upgrade / re-install allow the currently running system as a target
in the "disk" selection.

Fix the "reinstall sets" action.

When searching for a partition that may be the root partition for upgrading,
allow "/", "/targetroot" and "/altroot" as potential last mount points.

We may call md_pre_mount() multiple times for the same install set,
so pass an additional index argument so MD code knows which part
of the install will be mounted next.

When adding non-disklabel boot partitions (or similar), distinguish between
ones we need to add to the outer (MBR) partitions, or copy over to the
inner (disklabel) partitions.

refactor disk ioctl stuff to make it smaller.

add more printflikes() (rin) and remove unused function.
cVS: ----------------------------------------------------------------------

Rework internal data structures and "interfaces to user interface" functions
to get rid of all disklabel assumptions.

Previously (even for GPT partitioning) struct disklabel was used, which
obviously breaks large disk setups. Also many MD parts and parts of the
user interface assumed (a) a struct disklabel is used internally to
store partitioning information and (b) partitions are named 'a' ... $MAXPART.

Get rid of this and replace it with a quite abstract interface that should
be able to deal with all variants in partition storage:

- partitions are stored in a (partly abstract) struct disk_partitions
and most parts of it are only accessed via accessor functions provided
by a "partitioning scheme".

- implement partitioning schemes for MBR, disklabel and GPT (with likely
RDB [amiga] and Apple Partition Map [mac*] to follow soon)

- partitioning schemes may be cascaded, e.g. on x86 when using MBR as
"outer partitions", we have disklabel as "inner partitions".

- all user interface goes via accessor functions in the partitioning scheme,
some of which return pointers to special user interface descriptors
(e.g. to allow editing partition flags, which are scheme specific)

Overall the user interface changes (in this initial step) are minimal but
noticable. A new Anita is needed for automatic test setups - many thanks
to Andreas Gustafsson for lots of early testing and a new Anita version,
and to Manuel Bouyer for cooperation and tests of the Anita release.

This work was sponsored by The NetBSD Foundation, Inc.

Backout previous snprintf nonsense, instead STFU gcc 7's most
useless warning ever.

Fix a few off by one (in both directions) in previous, pointed out by mrg.
Avoid open coding snprintf return value checking and introduce a
helper functions that always ensures string termination instead,
suggested by christos.

Remove extra #ifdef and duplicated set of #ifdef's.

Merge the [pgoyette-compat] branch

Separate binary sets and binary pkg prefix from the machine, and set
it explicitly for evbarm*, evbmips* and evbsh3 so we get the correct
64bit pkgs on aarch64 and also have working defaults on the other
variants.

Cosmetics: consistently use MEG and GIG defines for size calculations

Split positional argument string expansion from msg display, we can make
good use of it elsewhere too.

Add a new helper function to show messages with positional parameters

Slightly enhance previous: create two utility functions checking for
partitionability and boot code requirements, use a generic match helper
function for both and fully check the device name.

Skip floopies when selecting an install target, but do not add them to
the CD list when selecting an installation source device.

Do not allow installation onto floppy disk - can be overriden by
architectures where it would make sense (none currently), and also
works around a bug in QEMU for sparc that kills our -current anita tests.

Adapt other cd-device query functions to new world order, fix
default cd device (assume 'a' partition).

Unify handling of CDROM device names.
Skip CDs when looking for install targets (we do not support installing
onto a blueray with UDF).

Fix search for the default CDROM device and the CD we booted from
for ports with nonstandard CDROM device names.

Support sets in .tar.xz format

Add NOPARTMAN compile-time option, which drops extended partitioning
support provided by partman.c. It reduces, e.g., about 30KB for
crunched binary in atari install floppy.

OK christos

Now that the build cluster creates stable "latest" links, use them
as default for downloading sets.

Try to properly guess a path on the ftp server for pkg binaries
and use that as default.

Assume for $N.99 (current) we can use $N.0. Otherwise strip all
RC*/STABLE* annotations and just use the plain numbers.

Split the host name used for ftp transfers from the one used for http
transfers. This is slightly inconsistent, as directories are still
shared - but this allows us to default to cdn/nycdn for http (which don't
support ftp).

Fix editor accident in previous

Fix previous in case when BUILDID is passed - the subdirectory has the
same name as the BUILDID (which already includes the trailing "Z", no need
to append it again)

For local builds, at least use the right prefix on nyftp

Try to guess the proper directory on ftp or nyftp where the binary
sets for the current build will end up.

branches: 1.11.2;
Add an option to install onto a pre-configured wedge.
Greatly simmplifies (U)EFI setups (but does not fully automate them yet).

deconst -> __UNCONST, the former involves UB with NULL arithmetic.

branches: 1.9.8; 1.9.14;
Make "ask_yesno" and "ask_noyes" take a const char * as argument, moving
the deconst() dance into the utility functions and simplifying all callers.

Backout the previous incorrect fix for PR 49440 and redo it more globaly:
get rid of the global "yesno", introduce utility functions "ask_yesno()"
and "ask_noyes()" instead, greatly simplifying a lot of code.
Pass in a pointer to the return value to various "set source" menus.

Add a new utility function "update_wedges()", which triggers a scan
for wedges on the given disk. Call this after writing a disklabel.
This makes all auto-discovered wedges go away after we changed the
partitioning (and are not using GPT) and fixes PR 49665.

Rather than scatter set_swap() calls wrapped in ram size checks through
md_post_disklabel() and md_pre_update(), abstract them out to an MI
set_swap_if_low_ram() and call from the appropriate MI places.
Now all platforms add swap if the system has <= 32M of RAM.

Don't use asctime(localtime(time_t *)) because this is really ctime(time_t *)
and not checking the result of localtime can lead to tears.
Add a safectime() that always returns a good string, and add some debugging
so that we can see if there is indeed something wrong in the new libc time
code.

Only prompt for one nameserver, let the user decide if it's a
pre-determined google server other another one.

branches: 1.3.2; 1.3.4; 1.3.6;
Remove a few menu entries if raid/lvm/cgd/gpt are not available.

(Finally) merge a slightly modified version of the 2012 GSoC results
from Eugene Lozovoy: add extended partitioning options to sysinst.
Still needs some testing and polishing, but it now is possible to use GPT
or to create a RAID set from scratch and install onto it.

Move sysinst sources to usr.sbin.

This commit only physically moves the sources - there are no other
changes, to maximize the probability that this will be treated as a
rename if we ever do manage to migrate away from CVS.

Moving sysinst has been discussed on and off for years and has two
goals: making it easier to work on sysinst, and also making sysinst
available on running systems for use installing chroots and VM images
and other such things. None of the latter is possible yet, but as they
say, one thing at a time.

Doing this now was approved in an impromptu fashion by mrg, riz,
riastradh, me, and groo.

Add new sets: base32, debug32, manhtml

- base32 contains (when MKCOMPAT=yes) shared libraries for 32-bit
compatibility, previously included in base

- debug32 contains (when MKCOMPAT=yes) debug symbols and static libraries
containing debug symbols for 32-bit compatiblity, previously included
in debug

- manhtml contains (when MKHTML=yes) the HTML files previously included
in 'man', which are of limited utility without third-party software.

The motivation for this change is to be able to easily exclude sets
from CD-ROM images that go over the size limit without xz compression
(which many NetBSD platforms struggle to extract at acceptable speeds).

Add a https transfer method (sharing all host details with the http
transfer). Make this method the default for all downloads.

Try to make sure the binary pkg installation (which runs in a chroot
in the already installed system) can make use of SSL verification.

This does NOT fix the missing SSL verification in most install media
even if using https (due to not fully populated /etc/openssl/certs).

Correct a minor word usage error in a comment. NFC.

branches: 1.88.2;
Add an (expert) option to manually edit partitions, inspired by PR 57100.

Avoid C++-isms (stupid muscle memory) to fix the build.

Preparations to make the extended partitioning during normal installation
more usefull - pass an optional install_description to partman().

Check on-disk disklabel properly even on ports without raw BSD disklabel.

Fixes PR install/56890.

Centralize the "we can newfs this type of filesystem" predicate
and fix the list.

fix various small typos, mainly in comments.

When not invoked via some explicit "configure network" menu item,
auto-detect an already working network setup and offer to "just use" it
whenver we need connectivity.

Make default installations a bit more secure:
- push a bit harder for setup of a root password
- simplify/clarify manual entropy setup
- visit the entropy setup (on machines where it is needed) on the
default path through sysinst

x86: fix previous: in the UEFI case copy the bootloaders from install
media during initial installation, but use the (by then: updated)
files from the target disk for system upgrades.

Do not defer wedge deletions when we have nothing mounted.
Define and use a constant for maximum disk name length.

When upgrading, update the boot code post extraction from
the updated target file system.

Try to get rid of all wedges we created (after unmounting).

s/exisit/exist/ in comment.

Re-enable the entropy handling code in sysinst (only visible on machines
that do not have full entropy), but move it to the config menu
instead of enforcing it as mandatory step.

This menu is shown at the end of the setup, or if explicitly invoked
from the main menu.

Some of the input options are complex but useful in some situations
and code to support them is tiny. Most users will use the manual input
(first + default option) but some might prefer to connect a USB TRNG
or have easy setups to transfer entropy from another machine (while
copy & paste on a serial console sometimes is tricky).

Restore MKNOUVEAUFIRMWARE and MKRADEONFIRMWARE and make gpufw set unconditional

Simplifies logic.
(Second commit - first one was partial)

Restoring MK* requested by mrg on tech-kern discussion
https://mail-index.netbsd.org/tech-kern/2021/09/25/msg027695.html

Move DRM-driver firmware from base to its own set, gpufw

This set is only installed on amd64,i386,evbarm.
This set is installed on minimal installs and on install media, in
case someone needs it for basic driver functionality.

Comments:
Switched to a single MK tunable for it - that is probably unneeded.

An upcoming DRM update will include even fatter firmware, and we'd
like to minimize the impact of it.

PR 56354: all actions to set up swap space are not guaranteed to gain
us enough virtual memory anyway, so drop return codes from set_swap*.
The state for cleanup (which swap dev to unuse) has been made global
some time ago anyway.

Previously use of the return values was inconsistent. Error reporting
will only confuse users and sometimes the situation is hard to fix or
even impossible (like in miniroots copide to swap space for booting).

PR install/56303: when using xz files enable swap during set extraction
if the machine does not have more than 256MB of RAM.

sysinst: remove trailing whitespace from *.c *.h

In contrast to the messages files, this whitespace is not significant.

Add (experimental) entropy input support:
Early during new installs or after upgrades we check if entropy is
available. If not (no hardware random number generator available)
we inform the user and ask them to fix it.

PR 55769: avoid the module set if we do not build any

PR 55142: on popular demand bring back expert options to adjust the
number of free inodes, block size and fragment size for FFS and LFS.

PR 55536: when we find existing partition tables and have alternative
formats available, offer to delete partitions and create new ones from
scratch (in some other or the same on-disk format).

When MD code requests a reserved area (at the start of the disk)
make sure to properly account for the "lost" space during all calculations
and especially when expanding the main partition to fill the disk.

Avoid using the global "pm" or a passed device descriptons when we are
passing disk limits explicitly anyway.

Simplify/fix a few rounding calculations.

PR 55381: try to deal with moved (but not yet saved) partitions when
calculating free space

Keep a list of all partition tables from which we have deleted partitions
and make sure we update those partition tables when writing partitions,
even if no install related partition remains on them.

Separate devicetree .dts -> .dtb building from kernel builds. They are now
part of a separate set, "dtb.tgz", and only built when MKDTB=yes. This
defaults to yes for earmv[67]* and aarch64, and no everywhere else.

PR 55202: disable swapping if we started it due to low ram

Ooops, backout previous - twice the entropy saving is not needed.
Instead slightly improve the (slightly) hidden other code that already
did it.

At the end of a new installation save entropy.

Don't depend on common symbol definitions.

PR misc/54886: bump threshold for automatic/default creation of a tmpfs /tmp
up slightly (to 384 MB ram). This will make sure the default install has
a > 64 MB /tmp available (number pulled out of thin air, 64 MB is the minimum
required by the ZFS tests).

Remove unused "rel" global variable and make the -r option a no-op.

Do not use -P for tar extractions - the in tree tar has been fixed.

Fix support for non-512-byte/sector disks again after I broke it when
introducing the abstract partition backends.

Factor out all RAM size thresholds as defines to avoid magic numbers.
To work around PR misc/54886 bump the threshold for a tmpfs /tmp mount
up to 256 MB.

First try to bring evbarm installation closer to current reality.

Fix /tmp tmpfs handling and on machines with enough RAM default to
creating a /tmp tmpfs with 25% of ram size limit. Suggested by ad@.

Finish conversion of extended partitioning parts to new abstract
backend interface. XXX still could use a lot of polishing.

Get rid of the evbarm preliminary menu: do not bother to ask the user
whether this is a RPi - query the FDT instead.

Make tar extraction flags depend on our usage of pax-as-tar or bsdtar.

Make cloning support optional, so we can save some space on very small
install media.

Add options to the various partitioning stages that allow cloning of
alien partitions (optionally including data).

Split out /rescue to its own set and adapt installers/images to add it.

This is meant to make updates safer: if something goes wrong with updating
base, we still have the old, standalone /rescue to recover from.

Do not offer to upgared the "current system" if we are running off a CD
(i.e. / is mounted read-only)

Support upgrade of systems using NAME= syntax in /etc/fstab.

Make supported file system types dynamic - instead of hardcoding the
available types at compile time, check for available newfs_* helper
binaries in the actual install environment at runtime.

branches: 1.42.2;
Strip trailing / from last mounted strings.
No idea how they happen, but for cgd root (init.root = "/altroot")
they have been reported to exist.

For upgrade / re-install allow the currently running system as a target
in the "disk" selection.

Fix the "reinstall sets" action.

When searching for a partition that may be the root partition for upgrading,
allow "/", "/targetroot" and "/altroot" as potential last mount points.

We may call md_pre_mount() multiple times for the same install set,
so pass an additional index argument so MD code knows which part
of the install will be mounted next.

When adding non-disklabel boot partitions (or similar), distinguish between
ones we need to add to the outer (MBR) partitions, or copy over to the
inner (disklabel) partitions.

refactor disk ioctl stuff to make it smaller.

add more printflikes() (rin) and remove unused function.
cVS: ----------------------------------------------------------------------

Rework internal data structures and "interfaces to user interface" functions
to get rid of all disklabel assumptions.

Previously (even for GPT partitioning) struct disklabel was used, which
obviously breaks large disk setups. Also many MD parts and parts of the
user interface assumed (a) a struct disklabel is used internally to
store partitioning information and (b) partitions are named 'a' ... $MAXPART.

Get rid of this and replace it with a quite abstract interface that should
be able to deal with all variants in partition storage:

- partitions are stored in a (partly abstract) struct disk_partitions
and most parts of it are only accessed via accessor functions provided
by a "partitioning scheme".

- implement partitioning schemes for MBR, disklabel and GPT (with likely
RDB [amiga] and Apple Partition Map [mac*] to follow soon)

- partitioning schemes may be cascaded, e.g. on x86 when using MBR as
"outer partitions", we have disklabel as "inner partitions".

- all user interface goes via accessor functions in the partitioning scheme,
some of which return pointers to special user interface descriptors
(e.g. to allow editing partition flags, which are scheme specific)

Overall the user interface changes (in this initial step) are minimal but
noticable. A new Anita is needed for automatic test setups - many thanks
to Andreas Gustafsson for lots of early testing and a new Anita version,
and to Manuel Bouyer for cooperation and tests of the Anita release.

This work was sponsored by The NetBSD Foundation, Inc.

Backout previous snprintf nonsense, instead STFU gcc 7's most
useless warning ever.

Fix a few off by one (in both directions) in previous, pointed out by mrg.
Avoid open coding snprintf return value checking and introduce a
helper functions that always ensures string termination instead,
suggested by christos.

Remove extra #ifdef and duplicated set of #ifdef's.

Merge the [pgoyette-compat] branch

Separate binary sets and binary pkg prefix from the machine, and set
it explicitly for evbarm*, evbmips* and evbsh3 so we get the correct
64bit pkgs on aarch64 and also have working defaults on the other
variants.

Cosmetics: consistently use MEG and GIG defines for size calculations

Split positional argument string expansion from msg display, we can make
good use of it elsewhere too.

Add a new helper function to show messages with positional parameters

Slightly enhance previous: create two utility functions checking for
partitionability and boot code requirements, use a generic match helper
function for both and fully check the device name.

Skip floopies when selecting an install target, but do not add them to
the CD list when selecting an installation source device.

Do not allow installation onto floppy disk - can be overriden by
architectures where it would make sense (none currently), and also
works around a bug in QEMU for sparc that kills our -current anita tests.

Adapt other cd-device query functions to new world order, fix
default cd device (assume 'a' partition).

Unify handling of CDROM device names.
Skip CDs when looking for install targets (we do not support installing
onto a blueray with UDF).

Fix search for the default CDROM device and the CD we booted from
for ports with nonstandard CDROM device names.

Support sets in .tar.xz format

Add NOPARTMAN compile-time option, which drops extended partitioning
support provided by partman.c. It reduces, e.g., about 30KB for
crunched binary in atari install floppy.

OK christos

Now that the build cluster creates stable "latest" links, use them
as default for downloading sets.

Try to properly guess a path on the ftp server for pkg binaries
and use that as default.

Assume for $N.99 (current) we can use $N.0. Otherwise strip all
RC*/STABLE* annotations and just use the plain numbers.

Split the host name used for ftp transfers from the one used for http
transfers. This is slightly inconsistent, as directories are still
shared - but this allows us to default to cdn/nycdn for http (which don't
support ftp).

Fix editor accident in previous

Fix previous in case when BUILDID is passed - the subdirectory has the
same name as the BUILDID (which already includes the trailing "Z", no need
to append it again)

For local builds, at least use the right prefix on nyftp

Try to guess the proper directory on ftp or nyftp where the binary
sets for the current build will end up.

branches: 1.11.2;
Add an option to install onto a pre-configured wedge.
Greatly simmplifies (U)EFI setups (but does not fully automate them yet).

deconst -> __UNCONST, the former involves UB with NULL arithmetic.

branches: 1.9.8; 1.9.14;
Make "ask_yesno" and "ask_noyes" take a const char * as argument, moving
the deconst() dance into the utility functions and simplifying all callers.

Backout the previous incorrect fix for PR 49440 and redo it more globaly:
get rid of the global "yesno", introduce utility functions "ask_yesno()"
and "ask_noyes()" instead, greatly simplifying a lot of code.
Pass in a pointer to the return value to various "set source" menus.

Add a new utility function "update_wedges()", which triggers a scan
for wedges on the given disk. Call this after writing a disklabel.
This makes all auto-discovered wedges go away after we changed the
partitioning (and are not using GPT) and fixes PR 49665.

Rather than scatter set_swap() calls wrapped in ram size checks through
md_post_disklabel() and md_pre_update(), abstract them out to an MI
set_swap_if_low_ram() and call from the appropriate MI places.
Now all platforms add swap if the system has <= 32M of RAM.

Don't use asctime(localtime(time_t *)) because this is really ctime(time_t *)
and not checking the result of localtime can lead to tears.
Add a safectime() that always returns a good string, and add some debugging
so that we can see if there is indeed something wrong in the new libc time
code.

Only prompt for one nameserver, let the user decide if it's a
pre-determined google server other another one.

branches: 1.3.2; 1.3.4; 1.3.6;
Remove a few menu entries if raid/lvm/cgd/gpt are not available.

(Finally) merge a slightly modified version of the 2012 GSoC results
from Eugene Lozovoy: add extended partitioning options to sysinst.
Still needs some testing and polishing, but it now is possible to use GPT
or to create a RAID set from scratch and install onto it.

Move sysinst sources to usr.sbin.

This commit only physically moves the sources - there are no other
changes, to maximize the probability that this will be treated as a
rename if we ever do manage to migrate away from CVS.

Moving sysinst has been discussed on and off for years and has two
goals: making it easier to work on sysinst, and also making sysinst
available on running systems for use installing chroots and VM images
and other such things. None of the latter is possible yet, but as they
say, one thing at a time.

Doing this now was approved in an impromptu fashion by mrg, riz,
riastradh, me, and groo.

Add a https transfer method (sharing all host details with the http
transfer). Make this method the default for all downloads.

Try to make sure the binary pkg installation (which runs in a chroot
in the already installed system) can make use of SSL verification.

This does NOT fix the missing SSL verification in most install media
even if using https (due to not fully populated /etc/openssl/certs).

Correct a minor word usage error in a comment. NFC.

Add an (expert) option to manually edit partitions, inspired by PR 57100.

Avoid C++-isms (stupid muscle memory) to fix the build.

Preparations to make the extended partitioning during normal installation
more usefull - pass an optional install_description to partman().

Check on-disk disklabel properly even on ports without raw BSD disklabel.

Fixes PR install/56890.

Centralize the "we can newfs this type of filesystem" predicate
and fix the list.

fix various small typos, mainly in comments.

When not invoked via some explicit "configure network" menu item,
auto-detect an already working network setup and offer to "just use" it
whenver we need connectivity.

Make default installations a bit more secure:
- push a bit harder for setup of a root password
- simplify/clarify manual entropy setup
- visit the entropy setup (on machines where it is needed) on the
default path through sysinst

x86: fix previous: in the UEFI case copy the bootloaders from install
media during initial installation, but use the (by then: updated)
files from the target disk for system upgrades.

Do not defer wedge deletions when we have nothing mounted.
Define and use a constant for maximum disk name length.

When upgrading, update the boot code post extraction from
the updated target file system.

Try to get rid of all wedges we created (after unmounting).

s/exisit/exist/ in comment.

Re-enable the entropy handling code in sysinst (only visible on machines
that do not have full entropy), but move it to the config menu
instead of enforcing it as mandatory step.

This menu is shown at the end of the setup, or if explicitly invoked
from the main menu.

Some of the input options are complex but useful in some situations
and code to support them is tiny. Most users will use the manual input
(first + default option) but some might prefer to connect a USB TRNG
or have easy setups to transfer entropy from another machine (while
copy & paste on a serial console sometimes is tricky).

Restore MKNOUVEAUFIRMWARE and MKRADEONFIRMWARE and make gpufw set unconditional

Simplifies logic.
(Second commit - first one was partial)

Restoring MK* requested by mrg on tech-kern discussion
https://mail-index.netbsd.org/tech-kern/2021/09/25/msg027695.html

Move DRM-driver firmware from base to its own set, gpufw

This set is only installed on amd64,i386,evbarm.
This set is installed on minimal installs and on install media, in
case someone needs it for basic driver functionality.

Comments:
Switched to a single MK tunable for it - that is probably unneeded.

An upcoming DRM update will include even fatter firmware, and we'd
like to minimize the impact of it.

PR 56354: all actions to set up swap space are not guaranteed to gain
us enough virtual memory anyway, so drop return codes from set_swap*.
The state for cleanup (which swap dev to unuse) has been made global
some time ago anyway.

Previously use of the return values was inconsistent. Error reporting
will only confuse users and sometimes the situation is hard to fix or
even impossible (like in miniroots copide to swap space for booting).

PR install/56303: when using xz files enable swap during set extraction
if the machine does not have more than 256MB of RAM.

sysinst: remove trailing whitespace from *.c *.h

In contrast to the messages files, this whitespace is not significant.

Add (experimental) entropy input support:
Early during new installs or after upgrades we check if entropy is
available. If not (no hardware random number generator available)
we inform the user and ask them to fix it.

PR 55769: avoid the module set if we do not build any

PR 55142: on popular demand bring back expert options to adjust the
number of free inodes, block size and fragment size for FFS and LFS.

PR 55536: when we find existing partition tables and have alternative
formats available, offer to delete partitions and create new ones from
scratch (in some other or the same on-disk format).

When MD code requests a reserved area (at the start of the disk)
make sure to properly account for the "lost" space during all calculations
and especially when expanding the main partition to fill the disk.

Avoid using the global "pm" or a passed device descriptons when we are
passing disk limits explicitly anyway.

Simplify/fix a few rounding calculations.

PR 55381: try to deal with moved (but not yet saved) partitions when
calculating free space

Keep a list of all partition tables from which we have deleted partitions
and make sure we update those partition tables when writing partitions,
even if no install related partition remains on them.

Separate devicetree .dts -> .dtb building from kernel builds. They are now
part of a separate set, "dtb.tgz", and only built when MKDTB=yes. This
defaults to yes for earmv[67]* and aarch64, and no everywhere else.

PR 55202: disable swapping if we started it due to low ram

Ooops, backout previous - twice the entropy saving is not needed.
Instead slightly improve the (slightly) hidden other code that already
did it.

At the end of a new installation save entropy.

Don't depend on common symbol definitions.

PR misc/54886: bump threshold for automatic/default creation of a tmpfs /tmp
up slightly (to 384 MB ram). This will make sure the default install has
a > 64 MB /tmp available (number pulled out of thin air, 64 MB is the minimum
required by the ZFS tests).

Remove unused "rel" global variable and make the -r option a no-op.

Do not use -P for tar extractions - the in tree tar has been fixed.

Fix support for non-512-byte/sector disks again after I broke it when
introducing the abstract partition backends.

Factor out all RAM size thresholds as defines to avoid magic numbers.
To work around PR misc/54886 bump the threshold for a tmpfs /tmp mount
up to 256 MB.

First try to bring evbarm installation closer to current reality.

Fix /tmp tmpfs handling and on machines with enough RAM default to
creating a /tmp tmpfs with 25% of ram size limit. Suggested by ad@.

Finish conversion of extended partitioning parts to new abstract
backend interface. XXX still could use a lot of polishing.

Get rid of the evbarm preliminary menu: do not bother to ask the user
whether this is a RPi - query the FDT instead.

Make tar extraction flags depend on our usage of pax-as-tar or bsdtar.

Make cloning support optional, so we can save some space on very small
install media.

Add options to the various partitioning stages that allow cloning of
alien partitions (optionally including data).

Split out /rescue to its own set and adapt installers/images to add it.

This is meant to make updates safer: if something goes wrong with updating
base, we still have the old, standalone /rescue to recover from.

Do not offer to upgared the "current system" if we are running off a CD
(i.e. / is mounted read-only)

Support upgrade of systems using NAME= syntax in /etc/fstab.

Make supported file system types dynamic - instead of hardcoding the
available types at compile time, check for available newfs_* helper
binaries in the actual install environment at runtime.

branches: 1.42.2;
Strip trailing / from last mounted strings.
No idea how they happen, but for cgd root (init.root = "/altroot")
they have been reported to exist.

For upgrade / re-install allow the currently running system as a target
in the "disk" selection.

Fix the "reinstall sets" action.

When searching for a partition that may be the root partition for upgrading,
allow "/", "/targetroot" and "/altroot" as potential last mount points.

We may call md_pre_mount() multiple times for the same install set,
so pass an additional index argument so MD code knows which part
of the install will be mounted next.

When adding non-disklabel boot partitions (or similar), distinguish between
ones we need to add to the outer (MBR) partitions, or copy over to the
inner (disklabel) partitions.

refactor disk ioctl stuff to make it smaller.

add more printflikes() (rin) and remove unused function.
cVS: ----------------------------------------------------------------------

Rework internal data structures and "interfaces to user interface" functions
to get rid of all disklabel assumptions.

Previously (even for GPT partitioning) struct disklabel was used, which
obviously breaks large disk setups. Also many MD parts and parts of the
user interface assumed (a) a struct disklabel is used internally to
store partitioning information and (b) partitions are named 'a' ... $MAXPART.

Get rid of this and replace it with a quite abstract interface that should
be able to deal with all variants in partition storage:

- partitions are stored in a (partly abstract) struct disk_partitions
and most parts of it are only accessed via accessor functions provided
by a "partitioning scheme".

- implement partitioning schemes for MBR, disklabel and GPT (with likely
RDB [amiga] and Apple Partition Map [mac*] to follow soon)

- partitioning schemes may be cascaded, e.g. on x86 when using MBR as
"outer partitions", we have disklabel as "inner partitions".

- all user interface goes via accessor functions in the partitioning scheme,
some of which return pointers to special user interface descriptors
(e.g. to allow editing partition flags, which are scheme specific)

Overall the user interface changes (in this initial step) are minimal but
noticable. A new Anita is needed for automatic test setups - many thanks
to Andreas Gustafsson for lots of early testing and a new Anita version,
and to Manuel Bouyer for cooperation and tests of the Anita release.

This work was sponsored by The NetBSD Foundation, Inc.

Backout previous snprintf nonsense, instead STFU gcc 7's most
useless warning ever.

Fix a few off by one (in both directions) in previous, pointed out by mrg.
Avoid open coding snprintf return value checking and introduce a
helper functions that always ensures string termination instead,
suggested by christos.

Remove extra #ifdef and duplicated set of #ifdef's.

Merge the [pgoyette-compat] branch

Separate binary sets and binary pkg prefix from the machine, and set
it explicitly for evbarm*, evbmips* and evbsh3 so we get the correct
64bit pkgs on aarch64 and also have working defaults on the other
variants.

Cosmetics: consistently use MEG and GIG defines for size calculations

Split positional argument string expansion from msg display, we can make
good use of it elsewhere too.

Add a new helper function to show messages with positional parameters

Slightly enhance previous: create two utility functions checking for
partitionability and boot code requirements, use a generic match helper
function for both and fully check the device name.

Skip floopies when selecting an install target, but do not add them to
the CD list when selecting an installation source device.

Do not allow installation onto floppy disk - can be overriden by
architectures where it would make sense (none currently), and also
works around a bug in QEMU for sparc that kills our -current anita tests.

Adapt other cd-device query functions to new world order, fix
default cd device (assume 'a' partition).

Unify handling of CDROM device names.
Skip CDs when looking for install targets (we do not support installing
onto a blueray with UDF).

Fix search for the default CDROM device and the CD we booted from
for ports with nonstandard CDROM device names.

Support sets in .tar.xz format

Add NOPARTMAN compile-time option, which drops extended partitioning
support provided by partman.c. It reduces, e.g., about 30KB for
crunched binary in atari install floppy.

OK christos

Now that the build cluster creates stable "latest" links, use them
as default for downloading sets.

Try to properly guess a path on the ftp server for pkg binaries
and use that as default.

Assume for $N.99 (current) we can use $N.0. Otherwise strip all
RC*/STABLE* annotations and just use the plain numbers.

Split the host name used for ftp transfers from the one used for http
transfers. This is slightly inconsistent, as directories are still
shared - but this allows us to default to cdn/nycdn for http (which don't
support ftp).

Fix editor accident in previous

Fix previous in case when BUILDID is passed - the subdirectory has the
same name as the BUILDID (which already includes the trailing "Z", no need
to append it again)

For local builds, at least use the right prefix on nyftp

Try to guess the proper directory on ftp or nyftp where the binary
sets for the current build will end up.

branches: 1.11.2;
Add an option to install onto a pre-configured wedge.
Greatly simmplifies (U)EFI setups (but does not fully automate them yet).

deconst -> __UNCONST, the former involves UB with NULL arithmetic.

branches: 1.9.8; 1.9.14;
Make "ask_yesno" and "ask_noyes" take a const char * as argument, moving
the deconst() dance into the utility functions and simplifying all callers.

Backout the previous incorrect fix for PR 49440 and redo it more globaly:
get rid of the global "yesno", introduce utility functions "ask_yesno()"
and "ask_noyes()" instead, greatly simplifying a lot of code.
Pass in a pointer to the return value to various "set source" menus.

Add a new utility function "update_wedges()", which triggers a scan
for wedges on the given disk. Call this after writing a disklabel.
This makes all auto-discovered wedges go away after we changed the
partitioning (and are not using GPT) and fixes PR 49665.

Rather than scatter set_swap() calls wrapped in ram size checks through
md_post_disklabel() and md_pre_update(), abstract them out to an MI
set_swap_if_low_ram() and call from the appropriate MI places.
Now all platforms add swap if the system has <= 32M of RAM.

Don't use asctime(localtime(time_t *)) because this is really ctime(time_t *)
and not checking the result of localtime can lead to tears.
Add a safectime() that always returns a good string, and add some debugging
so that we can see if there is indeed something wrong in the new libc time
code.

Only prompt for one nameserver, let the user decide if it's a
pre-determined google server other another one.

branches: 1.3.2; 1.3.4; 1.3.6;
Remove a few menu entries if raid/lvm/cgd/gpt are not available.

(Finally) merge a slightly modified version of the 2012 GSoC results
from Eugene Lozovoy: add extended partitioning options to sysinst.
Still needs some testing and polishing, but it now is possible to use GPT
or to create a RAID set from scratch and install onto it.

Move sysinst sources to usr.sbin.

This commit only physically moves the sources - there are no other
changes, to maximize the probability that this will be treated as a
rename if we ever do manage to migrate away from CVS.

Moving sysinst has been discussed on and off for years and has two
goals: making it easier to work on sysinst, and also making sysinst
available on running systems for use installing chroots and VM images
and other such things. None of the latter is possible yet, but as they
say, one thing at a time.

Doing this now was approved in an impromptu fashion by mrg, riz,
riastradh, me, and groo.

Correct a minor word usage error in a comment. NFC.

Add an (expert) option to manually edit partitions, inspired by PR 57100.

Avoid C++-isms (stupid muscle memory) to fix the build.

Preparations to make the extended partitioning during normal installation
more usefull - pass an optional install_description to partman().

Check on-disk disklabel properly even on ports without raw BSD disklabel.

Fixes PR install/56890.

Centralize the "we can newfs this type of filesystem" predicate
and fix the list.

fix various small typos, mainly in comments.

When not invoked via some explicit "configure network" menu item,
auto-detect an already working network setup and offer to "just use" it
whenver we need connectivity.

Make default installations a bit more secure:
- push a bit harder for setup of a root password
- simplify/clarify manual entropy setup
- visit the entropy setup (on machines where it is needed) on the
default path through sysinst

x86: fix previous: in the UEFI case copy the bootloaders from install
media during initial installation, but use the (by then: updated)
files from the target disk for system upgrades.

Do not defer wedge deletions when we have nothing mounted.
Define and use a constant for maximum disk name length.

When upgrading, update the boot code post extraction from
the updated target file system.

Try to get rid of all wedges we created (after unmounting).

s/exisit/exist/ in comment.

Re-enable the entropy handling code in sysinst (only visible on machines
that do not have full entropy), but move it to the config menu
instead of enforcing it as mandatory step.

This menu is shown at the end of the setup, or if explicitly invoked
from the main menu.

Some of the input options are complex but useful in some situations
and code to support them is tiny. Most users will use the manual input
(first + default option) but some might prefer to connect a USB TRNG
or have easy setups to transfer entropy from another machine (while
copy & paste on a serial console sometimes is tricky).

Restore MKNOUVEAUFIRMWARE and MKRADEONFIRMWARE and make gpufw set unconditional

Simplifies logic.
(Second commit - first one was partial)

Restoring MK* requested by mrg on tech-kern discussion
https://mail-index.netbsd.org/tech-kern/2021/09/25/msg027695.html

Move DRM-driver firmware from base to its own set, gpufw

This set is only installed on amd64,i386,evbarm.
This set is installed on minimal installs and on install media, in
case someone needs it for basic driver functionality.

Comments:
Switched to a single MK tunable for it - that is probably unneeded.

An upcoming DRM update will include even fatter firmware, and we'd
like to minimize the impact of it.

PR 56354: all actions to set up swap space are not guaranteed to gain
us enough virtual memory anyway, so drop return codes from set_swap*.
The state for cleanup (which swap dev to unuse) has been made global
some time ago anyway.

Previously use of the return values was inconsistent. Error reporting
will only confuse users and sometimes the situation is hard to fix or
even impossible (like in miniroots copide to swap space for booting).

PR install/56303: when using xz files enable swap during set extraction
if the machine does not have more than 256MB of RAM.

sysinst: remove trailing whitespace from *.c *.h

In contrast to the messages files, this whitespace is not significant.

Add (experimental) entropy input support:
Early during new installs or after upgrades we check if entropy is
available. If not (no hardware random number generator available)
we inform the user and ask them to fix it.

PR 55769: avoid the module set if we do not build any

PR 55142: on popular demand bring back expert options to adjust the
number of free inodes, block size and fragment size for FFS and LFS.

PR 55536: when we find existing partition tables and have alternative
formats available, offer to delete partitions and create new ones from
scratch (in some other or the same on-disk format).

When MD code requests a reserved area (at the start of the disk)
make sure to properly account for the "lost" space during all calculations
and especially when expanding the main partition to fill the disk.

Avoid using the global "pm" or a passed device descriptons when we are
passing disk limits explicitly anyway.

Simplify/fix a few rounding calculations.

PR 55381: try to deal with moved (but not yet saved) partitions when
calculating free space

Keep a list of all partition tables from which we have deleted partitions
and make sure we update those partition tables when writing partitions,
even if no install related partition remains on them.

Separate devicetree .dts -> .dtb building from kernel builds. They are now
part of a separate set, "dtb.tgz", and only built when MKDTB=yes. This
defaults to yes for earmv[67]* and aarch64, and no everywhere else.

PR 55202: disable swapping if we started it due to low ram

Ooops, backout previous - twice the entropy saving is not needed.
Instead slightly improve the (slightly) hidden other code that already
did it.

At the end of a new installation save entropy.

Don't depend on common symbol definitions.

PR misc/54886: bump threshold for automatic/default creation of a tmpfs /tmp
up slightly (to 384 MB ram). This will make sure the default install has
a > 64 MB /tmp available (number pulled out of thin air, 64 MB is the minimum
required by the ZFS tests).

Remove unused "rel" global variable and make the -r option a no-op.

Do not use -P for tar extractions - the in tree tar has been fixed.

Fix support for non-512-byte/sector disks again after I broke it when
introducing the abstract partition backends.

Factor out all RAM size thresholds as defines to avoid magic numbers.
To work around PR misc/54886 bump the threshold for a tmpfs /tmp mount
up to 256 MB.

First try to bring evbarm installation closer to current reality.

Fix /tmp tmpfs handling and on machines with enough RAM default to
creating a /tmp tmpfs with 25% of ram size limit. Suggested by ad@.

Finish conversion of extended partitioning parts to new abstract
backend interface. XXX still could use a lot of polishing.

Get rid of the evbarm preliminary menu: do not bother to ask the user
whether this is a RPi - query the FDT instead.

Make tar extraction flags depend on our usage of pax-as-tar or bsdtar.

Make cloning support optional, so we can save some space on very small
install media.

Add options to the various partitioning stages that allow cloning of
alien partitions (optionally including data).

Split out /rescue to its own set and adapt installers/images to add it.

This is meant to make updates safer: if something goes wrong with updating
base, we still have the old, standalone /rescue to recover from.

Do not offer to upgared the "current system" if we are running off a CD
(i.e. / is mounted read-only)

Support upgrade of systems using NAME= syntax in /etc/fstab.

Make supported file system types dynamic - instead of hardcoding the
available types at compile time, check for available newfs_* helper
binaries in the actual install environment at runtime.

branches: 1.42.2;
Strip trailing / from last mounted strings.
No idea how they happen, but for cgd root (init.root = "/altroot")
they have been reported to exist.

For upgrade / re-install allow the currently running system as a target
in the "disk" selection.

Fix the "reinstall sets" action.

When searching for a partition that may be the root partition for upgrading,
allow "/", "/targetroot" and "/altroot" as potential last mount points.

We may call md_pre_mount() multiple times for the same install set,
so pass an additional index argument so MD code knows which part
of the install will be mounted next.

When adding non-disklabel boot partitions (or similar), distinguish between
ones we need to add to the outer (MBR) partitions, or copy over to the
inner (disklabel) partitions.

refactor disk ioctl stuff to make it smaller.

add more printflikes() (rin) and remove unused function.
cVS: ----------------------------------------------------------------------

Rework internal data structures and "interfaces to user interface" functions
to get rid of all disklabel assumptions.

Previously (even for GPT partitioning) struct disklabel was used, which
obviously breaks large disk setups. Also many MD parts and parts of the
user interface assumed (a) a struct disklabel is used internally to
store partitioning information and (b) partitions are named 'a' ... $MAXPART.

Get rid of this and replace it with a quite abstract interface that should
be able to deal with all variants in partition storage:

- partitions are stored in a (partly abstract) struct disk_partitions
and most parts of it are only accessed via accessor functions provided
by a "partitioning scheme".

- implement partitioning schemes for MBR, disklabel and GPT (with likely
RDB [amiga] and Apple Partition Map [mac*] to follow soon)

- partitioning schemes may be cascaded, e.g. on x86 when using MBR as
"outer partitions", we have disklabel as "inner partitions".

- all user interface goes via accessor functions in the partitioning scheme,
some of which return pointers to special user interface descriptors
(e.g. to allow editing partition flags, which are scheme specific)

Overall the user interface changes (in this initial step) are minimal but
noticable. A new Anita is needed for automatic test setups - many thanks
to Andreas Gustafsson for lots of early testing and a new Anita version,
and to Manuel Bouyer for cooperation and tests of the Anita release.

This work was sponsored by The NetBSD Foundation, Inc.

Backout previous snprintf nonsense, instead STFU gcc 7's most
useless warning ever.

Fix a few off by one (in both directions) in previous, pointed out by mrg.
Avoid open coding snprintf return value checking and introduce a
helper functions that always ensures string termination instead,
suggested by christos.

Remove extra #ifdef and duplicated set of #ifdef's.

Merge the [pgoyette-compat] branch

Separate binary sets and binary pkg prefix from the machine, and set
it explicitly for evbarm*, evbmips* and evbsh3 so we get the correct
64bit pkgs on aarch64 and also have working defaults on the other
variants.

Cosmetics: consistently use MEG and GIG defines for size calculations

Split positional argument string expansion from msg display, we can make
good use of it elsewhere too.

Add a new helper function to show messages with positional parameters

Slightly enhance previous: create two utility functions checking for
partitionability and boot code requirements, use a generic match helper
function for both and fully check the device name.

Skip floopies when selecting an install target, but do not add them to
the CD list when selecting an installation source device.

Do not allow installation onto floppy disk - can be overriden by
architectures where it would make sense (none currently), and also
works around a bug in QEMU for sparc that kills our -current anita tests.

Adapt other cd-device query functions to new world order, fix
default cd device (assume 'a' partition).

Unify handling of CDROM device names.
Skip CDs when looking for install targets (we do not support installing
onto a blueray with UDF).

Fix search for the default CDROM device and the CD we booted from
for ports with nonstandard CDROM device names.

Support sets in .tar.xz format

Add NOPARTMAN compile-time option, which drops extended partitioning
support provided by partman.c. It reduces, e.g., about 30KB for
crunched binary in atari install floppy.

OK christos

Now that the build cluster creates stable "latest" links, use them
as default for downloading sets.

Try to properly guess a path on the ftp server for pkg binaries
and use that as default.

Assume for $N.99 (current) we can use $N.0. Otherwise strip all
RC*/STABLE* annotations and just use the plain numbers.

Split the host name used for ftp transfers from the one used for http
transfers. This is slightly inconsistent, as directories are still
shared - but this allows us to default to cdn/nycdn for http (which don't
support ftp).

Fix editor accident in previous

Fix previous in case when BUILDID is passed - the subdirectory has the
same name as the BUILDID (which already includes the trailing "Z", no need
to append it again)

For local builds, at least use the right prefix on nyftp

Try to guess the proper directory on ftp or nyftp where the binary
sets for the current build will end up.

branches: 1.11.2;
Add an option to install onto a pre-configured wedge.
Greatly simmplifies (U)EFI setups (but does not fully automate them yet).

deconst -> __UNCONST, the former involves UB with NULL arithmetic.

branches: 1.9.8; 1.9.14;
Make "ask_yesno" and "ask_noyes" take a const char * as argument, moving
the deconst() dance into the utility functions and simplifying all callers.

Backout the previous incorrect fix for PR 49440 and redo it more globaly:
get rid of the global "yesno", introduce utility functions "ask_yesno()"
and "ask_noyes()" instead, greatly simplifying a lot of code.
Pass in a pointer to the return value to various "set source" menus.

Add a new utility function "update_wedges()", which triggers a scan
for wedges on the given disk. Call this after writing a disklabel.
This makes all auto-discovered wedges go away after we changed the
partitioning (and are not using GPT) and fixes PR 49665.

Rather than scatter set_swap() calls wrapped in ram size checks through
md_post_disklabel() and md_pre_update(), abstract them out to an MI
set_swap_if_low_ram() and call from the appropriate MI places.
Now all platforms add swap if the system has <= 32M of RAM.

Don't use asctime(localtime(time_t *)) because this is really ctime(time_t *)
and not checking the result of localtime can lead to tears.
Add a safectime() that always returns a good string, and add some debugging
so that we can see if there is indeed something wrong in the new libc time
code.

Only prompt for one nameserver, let the user decide if it's a
pre-determined google server other another one.

branches: 1.3.2; 1.3.4; 1.3.6;
Remove a few menu entries if raid/lvm/cgd/gpt are not available.

(Finally) merge a slightly modified version of the 2012 GSoC results
from Eugene Lozovoy: add extended partitioning options to sysinst.
Still needs some testing and polishing, but it now is possible to use GPT
or to create a RAID set from scratch and install onto it.

Move sysinst sources to usr.sbin.

This commit only physically moves the sources - there are no other
changes, to maximize the probability that this will be treated as a
rename if we ever do manage to migrate away from CVS.

Moving sysinst has been discussed on and off for years and has two
goals: making it easier to work on sysinst, and also making sysinst
available on running systems for use installing chroots and VM images
and other such things. None of the latter is possible yet, but as they
say, one thing at a time.

Doing this now was approved in an impromptu fashion by mrg, riz,
riastradh, me, and groo.

Add an (expert) option to manually edit partitions, inspired by PR 57100.

Avoid C++-isms (stupid muscle memory) to fix the build.

Preparations to make the extended partitioning during normal installation
more usefull - pass an optional install_description to partman().

Check on-disk disklabel properly even on ports without raw BSD disklabel.

Fixes PR install/56890.

Centralize the "we can newfs this type of filesystem" predicate
and fix the list.

fix various small typos, mainly in comments.

When not invoked via some explicit "configure network" menu item,
auto-detect an already working network setup and offer to "just use" it
whenver we need connectivity.

Make default installations a bit more secure:
- push a bit harder for setup of a root password
- simplify/clarify manual entropy setup
- visit the entropy setup (on machines where it is needed) on the
default path through sysinst

x86: fix previous: in the UEFI case copy the bootloaders from install
media during initial installation, but use the (by then: updated)
files from the target disk for system upgrades.

Do not defer wedge deletions when we have nothing mounted.
Define and use a constant for maximum disk name length.

When upgrading, update the boot code post extraction from
the updated target file system.

Try to get rid of all wedges we created (after unmounting).

s/exisit/exist/ in comment.

Re-enable the entropy handling code in sysinst (only visible on machines
that do not have full entropy), but move it to the config menu
instead of enforcing it as mandatory step.

This menu is shown at the end of the setup, or if explicitly invoked
from the main menu.

Some of the input options are complex but useful in some situations
and code to support them is tiny. Most users will use the manual input
(first + default option) but some might prefer to connect a USB TRNG
or have easy setups to transfer entropy from another machine (while
copy & paste on a serial console sometimes is tricky).

Restore MKNOUVEAUFIRMWARE and MKRADEONFIRMWARE and make gpufw set unconditional

Simplifies logic.
(Second commit - first one was partial)

Restoring MK* requested by mrg on tech-kern discussion
https://mail-index.netbsd.org/tech-kern/2021/09/25/msg027695.html

Move DRM-driver firmware from base to its own set, gpufw

This set is only installed on amd64,i386,evbarm.
This set is installed on minimal installs and on install media, in
case someone needs it for basic driver functionality.

Comments:
Switched to a single MK tunable for it - that is probably unneeded.

An upcoming DRM update will include even fatter firmware, and we'd
like to minimize the impact of it.

PR 56354: all actions to set up swap space are not guaranteed to gain
us enough virtual memory anyway, so drop return codes from set_swap*.
The state for cleanup (which swap dev to unuse) has been made global
some time ago anyway.

Previously use of the return values was inconsistent. Error reporting
will only confuse users and sometimes the situation is hard to fix or
even impossible (like in miniroots copide to swap space for booting).

PR install/56303: when using xz files enable swap during set extraction
if the machine does not have more than 256MB of RAM.

sysinst: remove trailing whitespace from *.c *.h

In contrast to the messages files, this whitespace is not significant.

Add (experimental) entropy input support:
Early during new installs or after upgrades we check if entropy is
available. If not (no hardware random number generator available)
we inform the user and ask them to fix it.

PR 55769: avoid the module set if we do not build any

PR 55142: on popular demand bring back expert options to adjust the
number of free inodes, block size and fragment size for FFS and LFS.

PR 55536: when we find existing partition tables and have alternative
formats available, offer to delete partitions and create new ones from
scratch (in some other or the same on-disk format).

When MD code requests a reserved area (at the start of the disk)
make sure to properly account for the "lost" space during all calculations
and especially when expanding the main partition to fill the disk.

Avoid using the global "pm" or a passed device descriptons when we are
passing disk limits explicitly anyway.

Simplify/fix a few rounding calculations.

PR 55381: try to deal with moved (but not yet saved) partitions when
calculating free space

Keep a list of all partition tables from which we have deleted partitions
and make sure we update those partition tables when writing partitions,
even if no install related partition remains on them.

Separate devicetree .dts -> .dtb building from kernel builds. They are now
part of a separate set, "dtb.tgz", and only built when MKDTB=yes. This
defaults to yes for earmv[67]* and aarch64, and no everywhere else.

PR 55202: disable swapping if we started it due to low ram

Ooops, backout previous - twice the entropy saving is not needed.
Instead slightly improve the (slightly) hidden other code that already
did it.

At the end of a new installation save entropy.

Don't depend on common symbol definitions.

PR misc/54886: bump threshold for automatic/default creation of a tmpfs /tmp
up slightly (to 384 MB ram). This will make sure the default install has
a > 64 MB /tmp available (number pulled out of thin air, 64 MB is the minimum
required by the ZFS tests).

Remove unused "rel" global variable and make the -r option a no-op.

Do not use -P for tar extractions - the in tree tar has been fixed.

Fix support for non-512-byte/sector disks again after I broke it when
introducing the abstract partition backends.

Factor out all RAM size thresholds as defines to avoid magic numbers.
To work around PR misc/54886 bump the threshold for a tmpfs /tmp mount
up to 256 MB.

First try to bring evbarm installation closer to current reality.

Fix /tmp tmpfs handling and on machines with enough RAM default to
creating a /tmp tmpfs with 25% of ram size limit. Suggested by ad@.

Finish conversion of extended partitioning parts to new abstract
backend interface. XXX still could use a lot of polishing.

Get rid of the evbarm preliminary menu: do not bother to ask the user
whether this is a RPi - query the FDT instead.

Make tar extraction flags depend on our usage of pax-as-tar or bsdtar.

Make cloning support optional, so we can save some space on very small
install media.

Add options to the various partitioning stages that allow cloning of
alien partitions (optionally including data).

Split out /rescue to its own set and adapt installers/images to add it.

This is meant to make updates safer: if something goes wrong with updating
base, we still have the old, standalone /rescue to recover from.

Do not offer to upgared the "current system" if we are running off a CD
(i.e. / is mounted read-only)

Support upgrade of systems using NAME= syntax in /etc/fstab.

Make supported file system types dynamic - instead of hardcoding the
available types at compile time, check for available newfs_* helper
binaries in the actual install environment at runtime.

branches: 1.42.2;
Strip trailing / from last mounted strings.
No idea how they happen, but for cgd root (init.root = "/altroot")
they have been reported to exist.

For upgrade / re-install allow the currently running system as a target
in the "disk" selection.

Fix the "reinstall sets" action.

When searching for a partition that may be the root partition for upgrading,
allow "/", "/targetroot" and "/altroot" as potential last mount points.

We may call md_pre_mount() multiple times for the same install set,
so pass an additional index argument so MD code knows which part
of the install will be mounted next.

When adding non-disklabel boot partitions (or similar), distinguish between
ones we need to add to the outer (MBR) partitions, or copy over to the
inner (disklabel) partitions.

refactor disk ioctl stuff to make it smaller.

add more printflikes() (rin) and remove unused function.
cVS: ----------------------------------------------------------------------

Rework internal data structures and "interfaces to user interface" functions
to get rid of all disklabel assumptions.

Previously (even for GPT partitioning) struct disklabel was used, which
obviously breaks large disk setups. Also many MD parts and parts of the
user interface assumed (a) a struct disklabel is used internally to
store partitioning information and (b) partitions are named 'a' ... $MAXPART.

Get rid of this and replace it with a quite abstract interface that should
be able to deal with all variants in partition storage:

- partitions are stored in a (partly abstract) struct disk_partitions
and most parts of it are only accessed via accessor functions provided
by a "partitioning scheme".

- implement partitioning schemes for MBR, disklabel and GPT (with likely
RDB [amiga] and Apple Partition Map [mac*] to follow soon)

- partitioning schemes may be cascaded, e.g. on x86 when using MBR as
"outer partitions", we have disklabel as "inner partitions".

- all user interface goes via accessor functions in the partitioning scheme,
some of which return pointers to special user interface descriptors
(e.g. to allow editing partition flags, which are scheme specific)

Overall the user interface changes (in this initial step) are minimal but
noticable. A new Anita is needed for automatic test setups - many thanks
to Andreas Gustafsson for lots of early testing and a new Anita version,
and to Manuel Bouyer for cooperation and tests of the Anita release.

This work was sponsored by The NetBSD Foundation, Inc.

Backout previous snprintf nonsense, instead STFU gcc 7's most
useless warning ever.

Fix a few off by one (in both directions) in previous, pointed out by mrg.
Avoid open coding snprintf return value checking and introduce a
helper functions that always ensures string termination instead,
suggested by christos.

Remove extra #ifdef and duplicated set of #ifdef's.

Merge the [pgoyette-compat] branch

Separate binary sets and binary pkg prefix from the machine, and set
it explicitly for evbarm*, evbmips* and evbsh3 so we get the correct
64bit pkgs on aarch64 and also have working defaults on the other
variants.

Cosmetics: consistently use MEG and GIG defines for size calculations

Split positional argument string expansion from msg display, we can make
good use of it elsewhere too.

Add a new helper function to show messages with positional parameters

Slightly enhance previous: create two utility functions checking for
partitionability and boot code requirements, use a generic match helper
function for both and fully check the device name.

Skip floopies when selecting an install target, but do not add them to
the CD list when selecting an installation source device.

Do not allow installation onto floppy disk - can be overriden by
architectures where it would make sense (none currently), and also
works around a bug in QEMU for sparc that kills our -current anita tests.

Adapt other cd-device query functions to new world order, fix
default cd device (assume 'a' partition).

Unify handling of CDROM device names.
Skip CDs when looking for install targets (we do not support installing
onto a blueray with UDF).

Fix search for the default CDROM device and the CD we booted from
for ports with nonstandard CDROM device names.

Support sets in .tar.xz format

Add NOPARTMAN compile-time option, which drops extended partitioning
support provided by partman.c. It reduces, e.g., about 30KB for
crunched binary in atari install floppy.

OK christos

Now that the build cluster creates stable "latest" links, use them
as default for downloading sets.

Try to properly guess a path on the ftp server for pkg binaries
and use that as default.

Assume for $N.99 (current) we can use $N.0. Otherwise strip all
RC*/STABLE* annotations and just use the plain numbers.

Split the host name used for ftp transfers from the one used for http
transfers. This is slightly inconsistent, as directories are still
shared - but this allows us to default to cdn/nycdn for http (which don't
support ftp).

Fix editor accident in previous

Fix previous in case when BUILDID is passed - the subdirectory has the
same name as the BUILDID (which already includes the trailing "Z", no need
to append it again)

For local builds, at least use the right prefix on nyftp

Try to guess the proper directory on ftp or nyftp where the binary
sets for the current build will end up.

branches: 1.11.2;
Add an option to install onto a pre-configured wedge.
Greatly simmplifies (U)EFI setups (but does not fully automate them yet).

deconst -> __UNCONST, the former involves UB with NULL arithmetic.