package: fix dependency generation

A bug in release/packages/generate-ucl.sh causes package dependencies
(other than shlib depends) to not be generated correctly, meaning
packages are missing their dependencies.

generate-ucl.sh creates the UCL file by:

1. copying ${uclsource} (template.ucl) to ${uclfile}
2. appending dependencies to ${uclfile}
3. calling generate-ucl.lua on ${uclsource} to create ${uclfile}

This breaks because the dependencies added in step 2 are overwritten in
step 3.

Fix this by calling generate-ucl.lua with ${uclfile} as both the input
and output file, so anything we added to ${uclfile} is preserved.

Reviewed by: des, imp
Pull Request: https://github.com/freebsd/freebsd-src/pull/1203

package: FreeBSD-periodic should depend on FreeBSD-cron

Reported by: des
Reviewed by: imp, des
Pull Request: https://github.com/freebsd/freebsd-src/pull/1204

pkgbase: rework certctl package to only run rehash on the main package

Rework how ucl manifest are generated leveraging ucl features and flua

now the ucl generation is done via a lua script which uses libucl to
ingest the template and use variables as defined in its command line.

the template will include only if it exist a ucl file named after the
package name which will complement the template or overwrite what was
defined in the template if defined in this specific ucl file

this allows to overwrite license, but add script only to the packages
who actually needs them.

As a results the post install scripts are now only added to the right
package and not also added to the subpackages like -man or -dev

Reviewed by: markj
Differential Revision: https://reviews.freebsd.org/D44374

pkgbase: kill circular dependency

clang was set to depend on clang-dev, but clang-dev already depends
on clang by design

MFC After: 3 days

pkgbase: Fix ucl for libcompiler_rt

It's only a -dev package now so add it to the list of -dev package
that don't have non-dev package.

PR: 273859

pkgbase: Move headers and libs out of runtime and utilities

Headers from src/include were in the runtime-dev package but
subdirectories of src/include ended up in utilities-dev by default.
Neither package is a good choice - the headers in src/include are not
useful without the libraries contained in clibs-dev.

This moves the standard C headers to clibs-dev (C++ headers are already
in this package). While working on this, I found that various clang
libraries and headers were also bundled into utilities-dev by default
so these are also moved to clang-dev.

I also added a FreeBSD-build-essential meta package to make it simple to
install all the toolchain parts.

PR: 254173
Reviewed byb: manu
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D41815

Remove $FreeBSD$: one-line sh pattern

Remove /^\s*#[#!]?\s*\$FreeBSD\$.*$\n/

pkgbase: reorganise caroot and openssl packages

This splits out the certctl utility into a new certctl package and the
openssl libs into an openssl-lib package.

PR: 272816
Reviewed by: manu
Differential Revision: https://reviews.freebsd.org/D41321

pkgbase: share rules for packages without pkgdeps

Have liby and libcompat share *-dev and *-lib32_dev rules, and share
*-man rule for libcompat and libelftc. Also correct name substitution
and description for man rules.

Reviewed by: manu
Fixes: 5391bcf0f7e4 ("pkgbase: Do not record dependency on...")
Fixes: 65fa2fd23b9d ("pkgbase: Do record dependency on non-...")
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D37964

pkgbase: Do record dependency on non-existent libelftc package

libelftc is an internal lib so it's never installed.
When building with WITH_MANSPLITPKG=yes the libelftc-man package
will have a dependecy on a non-existent libelftc package.

Sponsored by: Beckhoff Automation GmbH & Co. KG

pkgbase: Do not record dependency on non-existent libcompat package

libcompat only provides a .a which is put in the -dev package.
Add a special record for it so we do not require a dependency on the
non-existent package FreeBSD-libcompat.

Sponsored by: Beckhoff Automation GmbH & Co. KG

pkgbase: Stop adding FreeBSD-runtime as a dep for every package

shlib_required/provided is enough for the dependencies and this also
causes problems for packages like rescue which shouldn't depend on runtime
at all.

PR: 268063
Sponsored by: Beckhoff Automation GmbH & Co. KG

pkgbase: do not record dependency on non-existent liby package

liby-dev provides (only) liby.a. liby has no headers or man pages, and
there is no liby package. Add a special case to record no dependency on
the package that does not exist.

PR: 266923
Reviewed by: bapt
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D37429

pkgbase: Remove runtime dependency on bootloader

It isn't needed and will cause problem when upgrading to a newer release.

Reviewed by: emaste
MFC after: 2 weeks
Sponsored by: Beckhoff Automation GmbH & Co. KG
Differential Revision: https://reviews.freebsd.org/D33878

pkgbase: Put certctl in the FreeBSD-caroot package

There is no reason that this shouldn't be there.
Change the dependency of caroot from utilities to openssl as it's
the only command that it uses not in runtime.

MFC after: 2 weeks
Sponsored by: Beckhoff Automation GmbH & Co. KG
Differential Revision: https://reviews.freebsd.org/D33458

pkgbase: Remove vital flags for FreeBSD-utilities

Per the pkg-descr: "Non-vital programs and librairies"

Reviewed by: emaste
MFC after: 2 weeks
Sponsored by: Beckhoff Automation GmbH & Co. KG
Differential Revision: https://reviews.freebsd.org/D33436

pkgbase: Remove package name mangling in generate-ucl.sh

The mangling was present in the initial revision of the script, but its
purpose is not clear. It may have been to avoid defining make(1)
variables with a dash in the name, but this is permitted. Furthermore,
it results in invalid dependency information if a dependency's name
contains an underscore, causing e.g., libcompiler_rt-dev to depend on
libcompiler-rt, and resulting in warnings when installing base system
packages. Remove the mangling.

Reviewed by: manu
MFC after: 2 months
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D29883

pkgbase: make only vital packages vital, not their sub-packages

make "vital" a replaceable, which defaults to "false" and only set it for
the main clib, utilities and runtime packages, not their sub-packages

PR: 254174
Differential Revision: https://reviews.freebsd.org/D29224

pkgbase: Add an src.conf option for splitting man pages

Man pages can be big in total, add an options to split man pages
in -man packages so we produce smaller packages.
This is useful for small jails or mfsroot produced of pkgbase.
The option is off by default.

Reviewed by: bapt, Mina Galić <me@igalic.co>
Differential Revision: https://reviews.freebsd.org/D29169
MFC after: 2 weeks

pkgbase: Remove case for runtime and jail package ucl generation

They aren't needed and produce wrong package comments :
We use to have "runtime-dev package" instead of
"FreeBSD Base System (Development Files)" for example

Reviewed by: bapt
Differential Revision: https://reviews.freebsd.org/D29168
MFC after: 2 weeks

pkgbase: Replace / with | for PKG_WWW

PKG_WWW contain / char, replace the delimiter by a '|'.

Reported by: 0mp

pkgbase: Add PKG_NAME_PREFIX, PKG_MAINTAINER and PKG_WWW

This is useful for downstream users to customize the packages.
Reviewed by: emaste
Differential Revision: https://reviews.freebsd.org/D26019

pkgbase: Fix dependancies

The package name is converted with _ instead of - as we have some variables
that cannot contain -
Convert back the dependancies with - instead of _

pkgbase: use -dev,-dbg instead of -development,-debug

-development is long and awkward, and is also inconsistent with prior art
from the Linux world, which uses -dev (Debian) or -devel (Red Hat). Follow
the Debian convention, and similarly for debug info packages.

Also remove redundant pkgbase development tag from includes. We already tag
include files with package=runtime,dev; there is no need to separately tag
them as dev.

Discussed with: bapt
Reviewed by: manu
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D24139

pkgbase: remove logic for _profile packages

We don't produce these anymore as of r356797, remove the remnant in
generate-ucl.sh that accounted for them. This isn't strictly necessary, but
future work is needed for the various packages that can be generated on a
lib build.

Namely, we may produce -development packages for private/internal libs that
should be installed but won't have the base FreeBSD-libfoo pkg to depend on
because it's internal (e.g. liby, libpmcstat, libifconfig) but we want the
headers installed. It may be a better move to just shove these into
-runtime-development instead, but if not then we've just simplified the
cases that need to take private/internal libs into account.

pkgbase: generate dependencies in generate-ucl.sh

This cuts out a large chunk of duplicated *.ucl files that just needed
dependencies tacked on, and gives generate-ucl.sh some leeway in case a
future pkg may need more than one dependency.

Run-through to determine which ones could simply be removed done with for i
in *.ucl; do diff -U3 template.ucl ${i}; done | less and inspecting for any
differences beyond just adding deps at the end. The remaining ucl files are
basically all differently-licensed, require scripts, or are marked vital.

I've opted to remove %PKGDEPS% entirely without regard for third-party ucl,
as pkgbase is not yet considered production. However, I do not hold a strong
position on this and there is approximately 0 chance it will return.

clibs should have been added previously in generate-ucl.sh as one that
doesn't have any dependencies, but do so now that we would otherwise be
tacking on the runtime dependency.

Reviewed by: manu, bapt (earlier version)
Differential Revision: https://reviews.freebsd.org/D23415

pkgbase: fix caroot packaging and add post-install script

The original intention for caroot was to be packaged separately, perhaps so
that users can have a more/less conservative upgrade policy for this
separated from the rest of base.

secure/caroot/Makefile doesn't have anything interesting to package, but its
subdirectories might. Move the PACKAGE= to Makefile.inc so both blacklisted
and trusted get packaged consistently into the correct one rather than the
default -utilities. Also tag the directories for package=caroot, as they
could also be empty; blacklisted is empty by default, but trusted is not.

Add a post-install script to do certctl rehash, along with a note should we
eventually come up with a way to detect that files have been added or
removed that requires a rehash.

-caroot gets a dependency on -utilities, as that's where we provide certctl
at the moment. We can perhaps reconsider this and put certctl into this
package in the future, but there are some bits within -utilities that
unconditionally invoke certctl so let's hold off for now.

Reviewed by: manu (earlier version, before -utilities dep added)
Differential Revision: https://reviews.freebsd.org/D23352

Simplify using bsd.endian.mk and have it provide CAP_MKDB_ENDIAN, since it is
the most common usage.

Approved by: bapt (mentor)

Chase the pwd_mkdb endian changes.

Approved by: bapt (mentor)

Set the 'vital' flag on the runtime and jail packages.

The default pkg(8) from pkg.freebsd.org requires libjail.so,
so mark the jail package as vital along with the runtime
package to avoid errors when libjail.so is removed. This is
a no-op for systems with WITHOUT_JAIL in src.conf(5) and pkg(8)
built from the Ports Collection.

In order to make this work without marking packages such as
the jail-lib32, for example, the jail.ucl file needed to be
split out into separate files similarly to the runtime-*.ucl
files.

Glanced at by: brd
MFC after: 5 days
Sponsored by: The FreeBSD Foundation

Prefix lib32, profile, debug, and development packages
with 'runtime-' for consistency with other packages.

Sponsored by: The FreeBSD Foundation

Rework the way <packagename>.ucl files are generated.
One of the major pain points with how this was implemented
is the requirement of in-tree, hard-coded <name>.ucl, as
well as <name>-<suffix>.ucl where <suffix> can be lib32,
profile, development, debug, or any combination of the four.

This created significant overhead when adding new packages
and any of the files in any of the combinations were missing.

Instead of test(1)-ing if the <packagename>.ucl file exists,
hand off to a script to figure out what the final ucl file
name should be before invoking pkg(8).

The default behavior is 'template.ucl' is used as a fallback.

This affects only the userland packages, as the kernel code
is already smart enough to handle these variations.

Sponsored by: The FreeBSD Foundation