crunchgen: add --list option

"bsdbox --list" will print all tools in the binary, one per line. The
main use case for this is to make it easier to create links:

for t in $(bsdbox --list); do
ln -s bsdbox $t
done

The name --list was taken from busybox.

This just adds a new "program" with the name "--list". I don't think we
need to do real argument parsing here, and this is also how busybox does
it.

An additional minor change is that just "bsdbox" will no longer print
the binary name itself ("bsdbox" in this case). Before it would do:

% bsdbox
usage: boxlike <prog> <args> ..., where <prog> is one of:
cp ls mv bsdbox

And now just:

% bsdbox
usage: boxlike <prog> <args> ..., where <prog> is one of:
cp ls mv

And just "bsdbox" will also exit with code 0 (and print to stdout)
rather than exit with 0 and print to stderr

Example output:

% ./bsdbox
usage: bsdbox program [args ...]
bsdbox --list
program [args ...]

bsdbox combines several programs in one executable. Create a link to this
executable with the program name to run that program, or give the program
name as the first argument.

Currently defined programs:
true false tail head uname

% ./bsdbox --list
true
false
tail
head
uname

% ./bsdbox uname -a
FreeBSD freebsd 13.2-RELEASE-p4 FreeBSD 13.2-RELEASE-p4 GENERIC amd64

% ln -s bsdbox uname
% ./uname -a
FreeBSD freebsd 13.2-RELEASE-p4 FreeBSD 13.2-RELEASE-p4 GENERIC amd64

Pull Request: https://github.com/freebsd/freebsd-src/pull/894
Signed-off-by: Martin Tournoij <martin@arp242.net>
Reviewed by: imp
Pull Request: https://github.com/freebsd/freebsd-src/pull/894

usr.sbin: Remove repeated words

Signed-off-by: Elyes Haouas <ehaouas@noos.fr>
Reviewed by: imp
Pull Request: https://github.com/freebsd/freebsd-src/pull/887

crunchgen: slap a dependency on the generated makefile for .lo

crunchgen generates a foo.lo for each binary it will end up crunching
into the final product. While they have a dependency on the libs that
are used to link them, nothing will force relinking if the set of libs
needed to link them is changed. Because of this, incremental builds may
not be possible if one builds a version of, e.g., rescue/ with a broken
set of libs specified for a project -- a subsequent fix won't be rolled
in cleanly, it will require purging the rescue/ objdir.

This is a bit crude, but the foo.mk we generate doesn't actually get
regenerated all that often in practice, so a spurious relink for the
vast majority of crunched objects won't actually happen all that often.

Reviewed by: bapt, emaste, imp
Differential Revision: https://reviews.freebsd.org/D43869

usr.sbin: Automated cleanup of cdefs and other formatting

Apply the following automated changes to try to eliminate
no-longer-needed sys/cdefs.h includes as well as now-empty
blank lines in a row.

Remove /^#if.*\n#endif.*\n#include\s+<sys/cdefs.h>.*\n/
Remove /\n+#include\s+<sys/cdefs.h>.*\n+#if.*\n#endif.*\n+/
Remove /\n+#if.*\n#endif.*\n+/
Remove /^#if.*\n#endif.*\n/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/types.h>/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/param.h>/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/capsicum.h>/

Sponsored by: Netflix

crunchgen: fix "keep" for an ELF world, break it out

"keep" currently adds a leading underscore, which hasn't been useful or
accurate since a.out days. Preserve the symbol name as it's given
rather than mangle it to match ELF-style symbol names.

This was partially fixed back in
6cd35234a092d ("Assume ELF-style symbol names now.") for crunchgen, but
the keeplist wasn't changed to match it.

While we're here, break it out to bsd.crunchgen.mk for later use in
bsdbox.

Reviewed by: adrian, imp
Differential Revision: https://reviews.freebsd.org/D42499

Remove $FreeBSD$: one-line .c pattern

Remove /^[\s*]*__FBSDID\("\$FreeBSD\$"\);?\s*\n/

crunchgen: remove -dc from linker invocation

In GNU ld and ld.lld, -dc is used with -r to allocate space to COMMON
symbols. It is presumably to work around legacy code which cannot
handle COMMON symbols in relocatable output. ld.lld may remove -dc or
make it a no-op for the 15.0.0 release.

As of 7420b323a014 crunch/crunchide does not require -dc, as the symbol
hiding technique no longer relied on making symbols local.

In addition -fno-common is now the default in Clang and GCC, so -dc
serves no purpose as the compiler does not generate COMMON symbols
anyway.

See https://maskray.me/blog/2022-02-06-all-about-common-symbols for more
detail on common symbols.

Reviewed by: emaste
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D34215

crunchgen: use realpath(3) instead of ``pwd -P''

r366466 (9c7bd4f198e1) fixed a subtle bug by stripping the trailing
'\n' appended to the output of popen("cd %s && pwd -P", p->srcdir).

Replace this cumbersome implementation with a single realpath(3) call
which avoids spawning a shell, reading from the stream with fgets(3),
and final strdup(3).

Reviewed by: arichardson, kevans
Approved by: imp
Differential Revision: https://reviews.freebsd.org/D26734

crunchgen: fix remaining issues under WARNS=6

Entirely variables that should be static, save for one 'no previous
declaration' in mkskel.sh.

Reviewed by: arichardson, imp
Differential Revision: https://reviews.freebsd.org/D31610

crunchgen: sprinkle some const-poisoning around

Reviewed by: arichardson, imp
Differential Revision: https://reviews.freebsd.org/D31607

crunchgen: fix MK_AUTO_OBJ logic after r364166

r364166 converted echo -n `/bin/pwd` to a raw pwd invocation, leaving a
trailing newline at the end of path. This caused a later stat() of it to
erroneously fail and the fallback to MK_AUTO_OBJ=no logic proceeded as
unexpected.

Harry Schmalzbauer bissected the resulting build failure he experienced
(stable/12 host, -HEAD build) down to r365887. This change is mostly
unrelated, except it switches the build to bootstrapped crunchgen - clue!

I then bissected recent crunchgen changes going back a bit since we wouldn't
observe the failure immediately with -CURRENT in most configurations, which
landed me on r364166. After many intense head-scratching minutes and printf
debugging, I realized that the newline was the difference. This is where our
tale ends.

Reported by: Harry Schmalzbauer, O. Hartmann, Mike Tancsa, kevans
MFC after: 3 days

crunchgen(8): fix crunched application build with WARNS=6

This was revealed by the rescue build with a patch I'm working on to default
WARNS=6 everywhere. The issues resolved were:

- Missing prototype for _crunched_${ident}_stub in the *_stub.c generated
bits
- Missing prototype for crunched_main
- Incomplete prototype for _crunched_${ident}_stub in the generated parts of
crunched_main
- Literal strings in the stub table must drop const qualifier, unless we
const'ify name
- f field in struct stub didn't have a proper prototype

Most of these issues are minor formalities and easily addressed.

I note that if my patch to eventually raise WARNS for the rescue build
lands, we'll need to bump the __FreeBSD_version requirement for
bootstrapping crunchgen and wipe out the rescue .OBJDIR if it's stale, which
we should be able to detect pretty easily from a couple of the issues that
have been fixed here.

Reviewed by: arichardson
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D26363

crunchgen: use pwd -P without env

The -P flag is required by POSIX so we don't have to care whether pwd is
a shell builtin or not. This also allows removing pwd from the list of
bootstrap tools since all shells we care about for building have a
builtin pwd command. This effectively reverts r364190.

Suggested By: rgrimes, jrtc27

Use env pwd instead of pwd in crunchgen.c

In r364166 I changed /bin/pwd to pwd, but pwd can be shell builtin that
may not correctly return a real path. To ensure that all symlinks are
resolved use `env pwd -P` instead (the -P flag is part of POSIX so
should be supported everywhere).

Reported By: rgrimes
Suggested By: jrtc27

Fix crunchgen usage of mkstemp()

On Glibc systems mkstemp can only be used once with the same template
string since it will be modified in-place and no longer contain any 'X' chars.
It is fine to reuse the same file here but we need to be explicit and use
open() instead of mkstemp() on the second use.

While touching this file also avoid a hardcoded /bin/pwd since that may not
work when building on non-FreeBSD systems.

Reviewed By: brooks
Differential Revision: https://reviews.freebsd.org/D25990

crunchgen: quiet -Wmissing-prototypes

Reviewed by: emaste (previous version)
Sponsored by: Dell EMC Isilon
Differential Revision: https://reviews.freebsd.org/D12107

crunchgen: do not strip crunched binary (it will be done by install)

Reviewed by: adrian, imp (earlier)
Differential Revision: https://reviews.freebsd.org/D2003

META_MODE/DIRDEPS_BUILD: Fix various issues with crunch builds.

- DIRDEPS_BUILD: Fix crunchgen builds losing their library dependencies
on a nop-rebuild.
- META_MODE: Fix not rebuilding various crunch.mk targets if their .meta
files warrant a rebuild. They were lacking .meta files previously.
This adds .NOMETA to the crunch objects being used since they are
already built. Bmake was forcing a rebuild on them since their
.meta files were not in the expected place; there is no reason to
rebuild them.

MFC after: 2 weeks
Sponsored by: EMC / Isilon Storage Division

WITH_AUTO_OBJ: Fix crunchgen builds.

Passing MAKEOBJDIRPREFIX to the main prog build (rescue) would confuse
WITH_AUTO_OBJ and cause it to create a recursed object directory that
then broke the actual prog build. This is normally not a problem since
we do not call 'make -f prog.mk obj' before building anything in it.

Crunchgen(1) also assumes that if -o is not passed then if an object
directory does not already exist then it should build in the source
directories. The normal buildworld process will have already ran 'make
obj' in each of the component directories so this is not a problem.
With WITH_AUTO_OBJ though this is not the case. So we must tell
crunchgen(1) that MK_AUTO_OBJ=yes will create the directory and to not
require it be present before generating its Makefile.

Sponsored by: EMC / Isilon Storage Division

Fix rescue build after r284356 with STRIP= by using proper STRIPBIN per build(7).

This was causing the following error:

rescue
sh: rescue: not found
*** [rescue] Error code 127

make[1]: stopped in /usr/obj/usr/src/rescue/rescue

Sponsored by: EMC / Isilon Storage Division

Change -Wl,-r back to -r so gcc doesn't pass --relax to the linker on some
architectures.

PR: 200881

Tell the compiler that -dc and -r are linker flags.

Reviewed by: adrian

Fix typo - this upsets llvm/gcc-4.9, but not gcc-4.2 for some reason.

Fix up crunchgen binary generation to work with external cross-build
tools.

* Allow STRIP to be overridden by the environment
* Use CC to tie things together, not LD

Tested:

* i386, mips32

Submitted by: kan

crunchgen: Permit use of alternative linkers.

Submitted by: Pete Chou
MFC after: 1 week

crunch: Sync some NetBSD changes

Revert a comment that was not supposed to go away.

Obtained from: NetBSD

crunch: Sync some NetBSD changes.

crunchide:
Apr 11, 2009: fix some -Wsign-compare issues.
Sep 20, 1999: Free the right thing.

crunchgen:
Apr 14, 2009: Fix some WARNS=4 issues (-Wshadow -Wcast-qual)
Oct 30, 2004: Add (unsigned char) cast to ctype functions
Feb 5, 2001: fix nested extern.

examples:
Aug 30, 2007: NetBSD 36867 - trsp references are deprecated

Obtained from: NetBSD
MFC after: 1 week

Some amount of style(9)
-- function definitions, header ordering, and $FreeBSD$.

Ensure crunchen uses the same make binary as the rest of the build.

Submitted by: Simon Gerraty <sjg@juniper.net>

A new jail(8) with a configuration file, to replace the work currently done
by /etc/rc.d/jail.

Create the altix project branch. The altix project will add support
for the SGI Altix 350 to FreeBSD/ia64. The hardware used for porting
is a two-module system, consisting of a base compute module and a
CPU expansion module. SGI's NUMAFlex architecture can be an excellent
platform to test CPU affinity and NUMA-aware features in FreeBSD.

Run with -B and just .POSIX.

Set .POSIX mode, along with disable parallel mode. Crunchgen parses the
output from make(1) and its picky what it should look like.
Also use make's conditional assignment operator rather than test and set.

- Import the HEAD csup code which is the basis for the cvsmode work.

Back out half of my previous change to support parallel makes.
The generated Makefile clashes with the src/bin/sh/Makefile, causing
it to try to use a rule to build something it doesn't need to.

Fix another parallel make problem with the generated make file.

Define the xxx_OBJPATHS earlier and then use it in the xxx_make
target because each obj is actually made through that.

This allows the crunch to work with -j32 on sun4v.

The makefile generated is still poor, though. It really shouldn't use
the general 'make all' to do the submakes in the app directories being
crunched because each of those objects is listed as a dependency in
the generated crunch makefile. Doing that really requires a unique rule
to generate them.

Add the SUBMAKE_TARGETS (like ppp_make) to the dependency list
for linking the crunched app so that when a parallel make is run
(like -j32 on sun4v), the link waits for the sub-make processes
to complete.

Commands like gmirror, graid3, ... and others which use dlopen() to load
classes from say, /lib/geom, cannot be statically linked completely.
Moreover, those shared objects may require other shared objects (i.e.
for geom, libraries like -lmd, -lcrypto).

The libs_so extension to crunchgen fixes this by allowing some libraries
to be linked in dynamically. This requires that a copy of rtld and the
shared libraries be made available to the crunched binary, and so is not
suitable for all environments. Crunchgen configurations which do not
use the 'libs_so' keyword are unaffected and produce identical binaries
with and without this commit.

Approved by: murray (mentor, in spirit), jhb
In collaboration with: Adrian Steinmann <ast at marabu dot ch>
MFC After: 6 weeks

In crunchgen(1), when calling make(1), don't redirect stderr to stdout,
just rely on the exit status to detect an error. This makes crunchgen(1)
safe to use with certain make(1) debugging flags.

MFC after: 1 week

Fix a 4.X -> 5.X upgrade problem: crunchgen used to emit a line 'MAKE=make'
or 'env MAKEOBJDIRPREFIX=... make' depending on the setting of
MAKEOBJDIRPREFIX in the environment. In any case this line kills the
original value of ${MAKE}. When during buildworld a new make is built (as
is the case during the upgrade) this causes a wrong make to be picked up
(the first one in the path). Use the same technique as Makefile.inc1:
create a MAKEENV variable and a CRUNCHMAKE that calls ${MAKE} with that
MAKEENV prefixed. Use CRUNCHMAKE instead of MAKE throughout the generated
makefile. This leaves the original ${MAKE} undisturbed.

The second chdir(1) that I trimmed is needed too
if the program has an object directory.

Explained by: Tim Kientzle <kientzle@acm.org>

chdir(1) into a source directory before attempting to determine the
OBJS list. This is needed to crunch any program that relies on the
correct .CURDIR setting, e.g. src/bin/csh.

Submitted by: Tim Kientzle <kientzle@acm.org>

Add a "special progname lib xxx ..." command to crunchgen
so the .lo files can be partially linked against libraries
which redefine symbols in the standard libs, or which reference
symbols in the objects.

Submitted by: Sam Leffler
MFC After: 3 days

Make source crunches work as well as object crunches; broken in the
last commit that fixed object crunches.

Repeat after me, "no excuses for not testing".

Make crunchgen crunch pre-built object files again, i.e.:

progs prog1
special prog1 objdir ../../prog1/obj
special prog1 objs prog1.o

This fixes a bug that I introduced around the time of 4.2-release.

Reported by: Larry Baird <lab@gta.com>

Use 'LIBS+= ...' instead of 'LIBS= ...' in the generated makefile
so that the user can override libraries at build time. This is
makes it easier to cross-compile crunch builds.

Suggested by: luigi
MFC after: 3 days

- Call `subclean' after `clean' for each program.
- Eliminate compiler warning.

`buildopts' may affect the selection of object files.
Make sure we pass $(BUILDOPTS) to the `clean' target
so that `make clean' works on the same set of object
files. Otherwise, we may end up with an incorrectly
built and up-to-date object file.

Liberal application of style(9).

Flag an error and remove the program from the list if there is
no source directory or no objects can be determined.

Initialise a structure variable.

Clean up the source directory finding code, and in the process fix a
problem that occurs when a program's source isn't found.

Use one of the TMPDIR env variable or _PATH_TMP instead of hardcoding
/tmp.

Submitted by: jedgar

Avoid potential problems with the existance of a shadow obj directory
confusing the 'make -f tmpfile' in the tmpfile, by creating it in /tmp.

Clarify an ambiguous 'else' introduced by a recent commit.

o check strdup() return values
o strcpy() -> strlcpy()
o sprintf() -> snprintf()
o mktemp() -> mkstemp()
o use err() instead of errx() in out_of_memory() function since
errno will probably be set

Submitted by: jedgar

Hopefully the last patch in a series to try and teach crunchgen to
handle MAKEOBJDIRPREFIX better, in conjunction with the -o and -p
flags.

Make crunchgen aware of obj directories. Formerly it had
/usr/obj hardwired in the code, now you can override it
with a command line option or MAKEOBJDIRPREFIX env. variable.
The above is useful to build picobsd-specific objects in some other
place than /usr/obj

While at it, fix documentation and change a few sprintf -> snprintf.

Add and document a new configuration command: buildopts.
This allows global make options to be specified within the crunch
configuration file.

Define a new special type: buildopts.

This allows build flags to be specified for a particular program from
within the crunch.conf file, eg:

prog ppp
special ppp buildopts -DNOKLDLOAD -DNOINET6 -DNONAT -DNOATM

This adds '-DNOKLDLOAD -DNOINET6 -DNONAT -DNOATM' to make targets
related to ppp when determining which object files to build and
when calculating dependencies and building the targets.

Cause prog_make targets to include a ``make obj'' rule if the -o flag
is given to crunchgen.
[ This fixes the previous commit which silently added ``make obj'' ]

A missing feature of crunchgen was the ability to supply make options on
a per program basis.

This has now been added in the following way:

* Harness the make header file that's specified with the -h argument:
- Allow the user to define $(OPTS) to specify make arguments that should
be added to every program target.
- Allow the user to define $(prog_OPTS) to specify make arguments that
should just be added to the build of 'prog'.

* Make sure that $(OPTS) and $(prog_OPTS) are defined when looking through
each program's make file to determine which object files to crunch.

* When building the crunchgen makefile add $(OPTS) and $(prog_OPTS)
to the depend and build rules for $(prog_OBJS).

Add a few features to crunchgen to simplify the use of existing
makefiles (for use with picobsd among other things).
See the manpage for details, but:

* -h makefile-include-name
can be used to specify a file to include in the makefiles
generated by crunchgen . This is a good place to specify make
variables such as RELEASE_CRUNCH, NOTHIS, NOTHAT and the like.

* special progname objvar variable_name
in the crunch config file declares a different variable than
OBJS to be used to get the list of objects.

* crunchgen now looks first for Makefile.<progname> in the current
directory to override the makefile in <progname> source dir.
This in many cases avoids the need to patch the original makefile
if the above two features are still not enough.

Approved-By: jordan

Assume ELF-style symbol names now.
Submitted by: jhay

Use /bin/pwd instead of a potentially shell builtin that does
not return the "real" path.
This makes it possible to have the .o files put in the /usr/obj
hierachy instead of in the /usr/src one.

Remove the necessity of -ldes and -lalias etc from release/Makefile.
When building a release, RELEASE_CRUNCH is defined for a `make' of
the objects required by the crunch of each program. The object list
is still obtained in the same way, so you must make sure that all
objects are built (empty if necessary) by this make. ppp/Makefile
provides an example.

Reviewed by: jkh

Add a new special called `keep' to the list of special options. It
provides for a means to specify an argument for crunchide's -k option.
(This is required by ntpdate.)

Submitted by: peter@rhiannon.clari.net.au (Peter Hawkins)

Use err(3).

Initialize memory obtained by malloc().

Detected by: phkmalloc -AJ

Enhance crunchgens understanding of our new /usr/obj/`pwd` strategy.

Make crunchgen match the new obj/ convention.

Remove trailing whitespace.

Add a '-l' option, which lists the argv[0]'s we will respond to. This will
simplify the src/release Makefile quite a bit.

This is the new crunch utility for making distribution floppies from
James da Silva. We need to integrate this before 1.1.5 so that we can
actually make the boot floppies.