usr.sbin: Automated cleanup of cdefs and other formatting

Apply the following automated changes to try to eliminate
no-longer-needed sys/cdefs.h includes as well as now-empty
blank lines in a row.

Remove /^#if.*\n#endif.*\n#include\s+<sys/cdefs.h>.*\n/
Remove /\n+#include\s+<sys/cdefs.h>.*\n+#if.*\n#endif.*\n+/
Remove /\n+#if.*\n#endif.*\n+/
Remove /^#if.*\n#endif.*\n/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/types.h>/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/param.h>/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/capsicum.h>/

Sponsored by: Netflix

pkg: Allocate a suitably-sized string for the local ABI

Previously the local ABI string was written to an on-stack buffer and
the pointer to that buffer was saved in a global before the function
returned. This had two issues: c[ABI].val pointed to a
no-longer-valid on-stack buffer after config_init returned, and the
string could potentially be truncated. Fix both of those by changing
pkg_get_myabi to return a pointer to a string allocated by asprintf.

Note that the allocated string is left in the global config array
until it is implicitly freed on process exit.

Reported by: GCC 13 -Wdangling-pointer
Reviewed by: emaste
Differential Revision: https://reviews.freebsd.org/D42623

Remove $FreeBSD$: one-line .c pattern

Remove /^[\s*]*__FBSDID\("\$FreeBSD\$"\);?\s*\n/

spdx: The BSD-2-Clause-FreeBSD identifier is obsolete, drop -FreeBSD

The SPDX folks have obsoleted the BSD-2-Clause-FreeBSD identifier. Catch
up to that fact and revert to their recommended match of BSD-2-Clause.

Discussed with: pfg
MFC After: 3 days
Sponsored by: Netflix

Revert "grep: remove tautological condition"

This reverts commit f6d6c66889001208aee7b0c46efe1c8ddffda57c.

Gremlins snuck into my tree and injected some WIP.

grep: remove tautological condition

st_size is an off_t, it cannot hold values larger than OFF_MAX.

CID: 1008931

pkg(7): replace usage of sbuf(9) with open_memstream(3)

open_memstream(3) is a standard way to obtain the same feature we do get
by using sbuf(9) (aka dynamic size buffer), switching to using it makes
pkg(7) more portable, and reduces its number of dependencies.

Reviewed by: manu
Differential Revision: https://reviews.freebsd.org/D30005

pkg(7): indentation fixes

Non functional changes

Make pkg(7) use environment variables specified in pkg.conf

Modify /usr/sbin/pkg to use environment variables specified in pkg.conf.
This allows control over underlying libraries like fetch(3), which can
be configured by setting HTTP_PROXY.

MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D29820

pkg(7): add an -r reponame option for bootstrap and add

This is limited to bootstrap/add because some real pkg(8) commands
have -r flags with an incompatible meaning/usage, e.g., pkg-audit.
pkg(7) will still commence the search as it has, but it will ignore any
repo objects without the given name so that overrides and whatnot still
work as expected.

The use of it for add is noted in the manpage; notably, that the
signature config for that repository will be used over global config if
it's specified. i.e., pkg(7) should assume that the given pkg did come
from that repository and treat it appropriately.

Reviewed by: bapt, manu
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D28524

usr.sbin/pkg: Don't re-define roundup2

The file already includes sys/param.h and should use that definition.
I found this while testing D28332.

Reviewed By: bapt
Differential Revision: https://reviews.freebsd.org/D28331

Make use of the getlocalbase() function for run-time adjustment of the
local software base directory, as committed in SVN rev. 367813.

The pkg and mailwrapper programs used the LOCALBASE environment variable
for this purpose and this functionality is preserved by getlocalbase().

After this change, the value of the user.localbase sysctl variable is used
if present (and not overridden in the environment).

The nvmecontrol program gains support of a dynamic path to its plugin
directory with this update.

Differential Revision: https://reviews.freebsd.org/D27237

Replace literal uses of /usr/local in C sources with _PATH_LOCALBASE

Literal references to /usr/local exist in a large number of files in
the FreeBSD base system. Many are in contributed software, in configuration
files, or in the documentation, but 19 uses have been identified in C
source files or headers outside the contrib and sys/contrib directories.

This commit makes it possible to set _PATH_LOCALBASE in paths.h to use
a different prefix for locally installed software.

In order to avoid changes to openssh source files, LOCALBASE is passed to
the build via Makefiles under src/secure. While _PATH_LOCALBASE could have
been used here, there is precedent in the construction of the path used to
a xauth program which depends on the LOCALBASE value passed on the compiler
command line to select a non-default directory.

This could be changed in a later commit to make the openssh build
consistently use _PATH_LOCALBASE. It is considered out-of-scope for this
commit.

Reviewed by: imp
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D26942

Use asprintf instead of sbuf

various: general adoption of SPDX licensing ID tags.

Mainly focus on files that use BSD 2-Clause license, however the tool I
was using misidentified many licenses so this was mostly a manual - error
prone - task.

The Software Package Data Exchange (SPDX) group provides a specification
to make it easier for automated tools to detect and summarize well known
opensource licenses. We are gradually adopting the specification, noting
that the tags are considered only advisory and do not, in any way,
superceed or replace the license texts.

No functional change intended.

Remove some unneeded headers

Implement pubkey support for the bootstrap

Note that to not interfer with finger print it expects a signature on pkg itself
which is named pkg.txz.pubkeysign

To genrate it:
echo -n "$(sha256 -q pkg.txz)" | openssl dgst -sha256 -sign /thekey \
-binary -out ./pkg.txz.pubkeysig

Note the "echo -n" which prevent signing the '\n' one would get otherwise

PR: 202622
MFC after: 1 week

Use pkg-1.4-style platform identifiers based on MACHINE_ARCH (e.g.
FreeBSD:11:amd64 instead of freebsd:11:x86:64) when bootstrapping pkg.
Thanks to portmgr for providing symlinks so both styles work.

Reviewed by: bapt
MFC after: 3 weeks

Fix ASSUME_ALWAYS_YES not being parsed properly from config after UCL conversion.

Sponsored by: EMC / Isilon Storage Division
MFC after: 1 week

Fix ABI from /usr/local/etc/pkg.conf not being respected.

Regression from r259266.

Sponsored by: EMC / Isilon Storage Division
MFC after: 1 week

Update my copyright

Switch pkg(7) from libyaml to libucl

Implicit include of sys/queue.h instead of relying on gelf.h/libelf.h to bring it
Newer version of gelf.h and libelf.h does not include sys/queue.h anymore

Submitted by: kaiw
MFC after: 3 days

Fix build: include sys/queue.h header explicitly since the gelf.h
header comes with the new libelf will no longer include sys/queue.h.

Fix multi-repository support by properly respecting 'enabled' flag.

This will read the REPOS_DIR env/config setting (default is /etc/pkg
and /usr/local/etc/pkg/repos) and use the last enabled repository.

This can be changed in the environment using a comma-separated list,
or in /usr/local/etc/pkg.conf with JSON array syntax of:
REPOS_DIR: ["/etc/pkg", "/usr/local/etc/pkg/repos"]

Approved by: bapt
MFC after: 1 week

Support checking signature for pkg bootstrap.

If the pkg.conf is configured with SIGNATURE_TYPE: FINGERPRINTS,
and FINGERPRINTS: /etc/keys/pkg then a pkg.sig file is fetched along
with pkg.txz. The signature contains the signature provided by the
signing server, and the public key. The .sig is the exact output
from the signing server in the following format:

SIGNATURE
<openssl signed>
CERT
<rsa public key>
END

The signature is verified with the following logic:

- If the .sig file is missing, it fails.
- If the .sig doesn't validate, it fails.
- If the public key in the .sig is not in the known trusted fingerprints,
it fails.
- If the public key is in the revoked key list, it fails.

Approved by: bapt
MFC after: 2 days
Discussed by: bapt with des, jonathan, gavin

Add support for reading configuration files from /etc/pkg.
For now only /etc/pkg/FreeBSD.conf is supported. Its style is:

Repo: {
URL: "...",
MIRROR_TYPE: "...",
...
}

The configuration will be read from /usr/local/etc/pkg.conf if exists,
otherwise /etc/pkg/FreeBSD.conf

Approved by: bapt
MFC after: 2 days

Wrap long lines

Approved by: bapt
MFC after: 2 days

Add support for using "pkg+http://" for the PACKAGESITE.

pkg 1.2 is adding this support as well. This should help
lessen the confusion on why the default SRV PACKAGESITE
does not load in a browser.

Adapated from: matthew's upstream pkg change
Approved by: bapt
MFC after: 2 days

Rename libbsdyml to libyaml, make private, and bump
SHLIB_MAJOR to 1.0

Suggested by: des
Approved by: bapt
MFC after: 1 week

Add support to detect arm vs armv6

There are two different versions of the ARM ABI depending on the
TARGET_ARCH. As these are sligntly different a package built for
one may not work on another. We need to detect which one we are on
by parsing the .ARM.attributes section.

This will only work on the ARM EABI as this section is part of the
ABI definition. As armv6 only supports the ARM EABI this is not a
problem for the oabi.

Older versions of libelf in FreeBSD fail to read the
.ARM.attributes section needed. As armv6 is unsupported on these
versions we can assume we are running on arm.

Submitted by: andrew
Approved by: re (delphij)
Obtained from: pkgng git

Fix detection of arm ABIs

Submitted by: andrew
Obtained from: pkg git

choose in priority the allocated ABI if any to do the ${ABI} substitution in PACKAGESITE

Reviewed by: bdrewery

Add the ability to correctly read pkg.conf is exists.

Only look for boostrap useful options:
- PACKAGESITE
- ABI
- MIRROR_TYPE
- ASSUME_ALWAYS_YES

While here makes PACKAGESITE expand the ${ABI} variable.
Allow to deactivate any SRV record look up (MIRROR_TYPE=none)
Use the same mechanism as for pkgng itself: first get configuration out of
environment variable and fallback on pkg.conf if exists.

Reviewed by: bdrewery