Remove $FreeBSD$: one-line sh pattern

Remove /^\s*#[#!]?\s*\$FreeBSD\$.*$\n/

libmd / md5: Add SHA-512/224.

While there, remove .Tn from man pages.

Also remove an obsolete comment about the 80386.

MFC after: 1 week
Sponsored by: Klara, Inc.
Reviewed by: kevans, allanjude
Differential Revision: https://reviews.freebsd.org/D38373

src.conf: Introduce WITHOUT_MACHDEP knob.

Summary:
This knob can be used to make buildsystem prefer generic C implentations of
various functions, instead of machine-specific assembler ones.

Test Plan: `make buildworld` on amd64

Reviewed by: imp, emaste

Differential Revision: https://reviews.freebsd.org/D36076

MFC after: 3 days

Add accelerated arm64 sha512 to libmd

As with sha256 add support for accelerated sha512 support to libmd on
arm64. This depends on clang 13+ to build as this is the first release
with the needed intrinsics. Gcc should also support them, however from
a currently unknown release.

Reviewed by: cem
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D33373

Revert "Disable the accelerated arm64 sha25 in static libraries"

This is now unneeded as arm64 can now resolve ifuncs in static binaries

This reverts commit c81ea895b563c4d1e39468a8525284b7474fe850.

Sponsored by: The FreeBSD Foundation

Disable the accelerated arm64 sha25 in static libraries

We don't have ifunc support in static arm64 binaries. Until we do
disable the accelerated sha256 code in a static libmd as it uses an
ifunc.

Reported by: brd
Sponsored by: The FreeBSD Foundation

Use arm64 sha256 intrinsics in libmd

Summary:
When running on a CPU that supports the arm64 sha256 intrinsics use them
to improve perfromance of sha256 calculations.

With this changethe following improvement has been seen on an Apple M1
with FreeBS running under Parallels, with similar results on a
Neoverse-N1 r3p1.

x sha256.orig
+ sha256.arm64
+--------------------------------------------------------------------+
|++ x x|
|+++ xxx|
||A |A||
+--------------------------------------------------------------------+
N Min Max Median Avg Stddev
x 5 3.41 3.5 3.46 3.458 0.042661458
+ 5 0.47 0.54 0.5 0.504 0.027018512
Difference at 95.0% confidence
-2.954 +/- 0.0520768
-85.4251% +/- 0.826831%
(Student's t, pooled s = 0.0357071)

Reviewed by: cem
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D31284

libmd: add dependency workaround for r366344

r366344 fixed and reenabled the assembly optimized skein implementation,
but skein_block objects were not being rebuilt in no-clean builds. This
resulted in failing no-clean builds. SKEIN_USE_ASM controls which
routines come from C vs assembly, and with no explicit dependency
r366344's change to SKEIN_USE_ASM did not cause skein_block.{o,pico}
to be rebuilt.

Add a dependency on this Makefile for the skein_block objects. This
dependency is broader in scope than absolutely required (that is, the
skein_block objects will now be rebuilt on any change to this Makefile).
There are ways this could be addressed, but it is probably not worth the
additional effort or testing time to pursue them.

PR: 248221
Reported by: kevans, Jeremy Faulkner
Discussed with: kevans
Sponsored by: The FreeBSD Foundation

libmd: fix assembly optimized skein implementation

The assembly implementation incorrectly used logical AND instead of
bitwise AND. Fix, and re-enable in libmd.

Submitted by: Yang Zhong <yzhong@freebsdfoundation.org>
Reviewed by: cem (earlier)
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D26614

libmd: temporarily disable optimized assembly skein1024 implementation

It is apparently broken when assembled by contemporary GNU as as well as
Clang IAS (which is used in the default configuration).

PR: 248221
Reported by: pizzamig
Sponsored by: The FreeBSD Foundation

Remove warning that is no longer accurate after r361853

We now build the skein assembly with clangs integrated assembler.

Reviewed By: emaste
Differential Revision: https://reviews.freebsd.org/D25664

Rename skein_block_asm.s to .S and assemble using Clang IAS

Comparing the object files produced by GNU as 2.17.50 and Clang IAS
shows many immaterial changes in strtab etc., and one material change
in .text:

1bac: 4c 8b 4f 18 mov 0x18(%rdi),%r9
1bb0: eb 0e jmp 1bc0 <Skein1024_block_loop>
- 1bb2: 66 66 2e 0f 1f 84 00 data16 nopw %cs:0x0(%rax,%rax,1)
- 1bb9: 00 00 00 00
- 1bbd: 0f 1f 00 nopl (%rax)
+ 1bb2: 66 2e 0f 1f 84 00 00 nopw %cs:0x0(%rax,%rax,1)
+ 1bb9: 00 00 00
+ 1bbc: 0f 1f 40 00 nopl 0x0(%rax)

0000000000001bc0 <Skein1024_block_loop>:
Skein1024_block_loop():
1bc0: 4c 8b 47 10 mov 0x10(%rdi),%r8
1bc4: 4c 03 85 c0 00 00 00 add 0xc0(%rbp),%r8

That is, GNU as and Clang's integrated assembler use different multi-
byte NOPs for alignment (GNU as emits an 11 byte NOP + a 3 byte NOP,
while Clang IAS emits a 10 byte NOP + a 4 byte NOP).

Dependency cleanup hacks are not required, because we do not create
.depend files from GNU as.

Reviewed by: allanjude, arichardson, cem, tsoome
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D8434

Also pass SKEIN_USE_ASM to the assembler, via AFLAGS

Apply C SKEIN_LOOP setting only to skein_block.c

Otherwise if assembling skein_block_asm.s with Clang's integrated assembler
we can pass conflicting SKEIN_LOOP settings (via CFLAGS and ACFLAGS).

pkgbase: Put a lot of binaries and lib in FreeBSD-runtime

All of them are needed to be able to boot to single user and be able
to repair a existing FreeBSD installation so put them directly into
FreeBSD-runtime.

Reviewed by: bapt, gjb
Differential Revision: https://reviews.freebsd.org/D21503

r338270 had the side effect of no longer installing libmd.so into /lib.
For users who have a seperate zfs mount of /usr or /usr/lib, this will
cause dynamic loading failures when attempting to execute zfs mount on
bootup. E.g. the system won't boot.

Including <src.opts.mk> sets SHLIBDIR, so SHLIBDIR?= has no
effect. The other lib/ Makefiles solve this problem by moving the
SHLIBDIR assignment to before .include <src.opts.mk>.

Submitted by: jilles
Reviewed by: allanjude
Approved by: re (rgrimes)
Differential Revision: https://reviews.freebsd.org/D16910

Don't build skein_block_asm.s if we don't have an as binary

This fixes building libmd on MacOS/Linux. The real fix is probably to
build it as a .S file with $CC instead. It might also be better to just
compile the C file in userspace since the compiler can the use SSE/AVX.

Reviewed By: emaste, brooks
Approved By: jhb (mentor)
Differential Revision: https://reviews.freebsd.org/D16844

Allow bootstrapping libmd on MacOS

The assembly files use directives that only work for ELF targets so skip
them when bootstrapping on MacOS.

Reviewed By: imp
Approved By: jhb (mentor)
Differential Revision: https://reviews.freebsd.org/D14247

Integrate SHA2-224 with userspace components

The double compilation of the kernel sources in libmd and libcrypt is
baffling, but add yet another define hack to prevent duplicate symbols.

Add documentation and SHA2-224 test cases to libmd.

Integrate with the md5(1) command, document, and add more test cases;
self-tests pass.

Revert r334090.

It causes the 32bit compat build of libmd to fail with:

libmd/rmd160c.c:86:9: error: 'ripemd160_block' macro redefined
#define ripemd160_block ripemd160_block_x86
^
libmd/ripemd.h:122:9: note: previous definition is here
#define ripemd160_block _libmd_ripemd160_block

libmd: build with WARNS=1

- build with WARNS=1. This works without any changes
- remove two unused variables noticed at WARNS=2

Increase loop unrolling for skein hashes

This patch was inspired by an opposite change made to shrink the code
for the boot loader.

On my i7-4770, it increases the skein1024 speed from 470 to 550 MB/s

Reviewed by: sbruno
MFC after: 1 month
Sponsored by: ScaleEngine Inc.
Differential Revision: https://reviews.freebsd.org/D7824

Fix bootstrapping libmd on older systems after r314709.

This follows another fix to bootstrap libmd after r313404. The
MD5FileChunk prototype is needed to build libmd, but it is
only reliably in the src tree's sys/md5.h header. Rather than
polluting the legacy build with this header for the entire build,
just symlink it in here for now as is done in the elftoolchain
build. Libmd is already referencing other src tree headers by
its used of CFLAGS+= ${SRCTOP}/sys/crypto/sha2. This, and
other uses of CFLAGS+= ${SRCTOP}/sys..., may later change to
be in the legacy mechanism.

Reported by: bde, ian, sjg
Tested by: ian

Use SRCTOP-relative paths to other directories instead of .CURDIR-relative ones

This simplifies pathing in make/displayed output

MFC after: 3 weeks
Sponsored by: Dell EMC Isilon

libmd: strip local absolute symbols (to link with lld)

Old versions of gas produce an invalid section index. That is ignored by
old versions of ld, but prevents a link with lld.

Submitted by: Rafael Ávila de Espíndola (earlier version)
Reviewed by: allanjude
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D6789

Implement SHA-512 truncated (224 and 256 bits)

This implements SHA-512/256, which generates a 256 bit hash by
calculating the SHA-512 then truncating the result. A different initial
value is used, making the result different from the first 256 bits of
the SHA-512 of the same input. SHA-512 is ~50% faster than SHA-256 on
64bit platforms, so the result is a faster 256 bit hash.

The main goal of this implementation is to enable support for this
faster hashing algorithm in ZFS. The feature was introduced into ZFS
in r289422, but is disconnected because SHA-512/256 support was missing.
A further commit will enable it in ZFS.

This is the follow on to r292782

Reviewed by: cem
Sponsored by: ScaleEngine Inc.
Differential Revision: https://reviews.freebsd.org/D6061

First pass through library packaging.

Sponsored by: The FreeBSD Foundation

Replace sys/crypto/sha2/sha2.c with lib/libmd/sha512c.c

cperciva's libmd implementation is 5-30% faster

The same was done for SHA256 previously in r263218

cperciva's implementation was lacking SHA-384 which I implemented, validated against OpenSSL and the NIST documentation

Extend sbin/md5 to create sha384(1)

Chase dependancies on sys/crypto/sha2/sha2.{c,h} and replace them with sha512{c.c,.h}

Reviewed by: cperciva, des, delphij
Approved by: secteam, bapt (mentor)
MFC after: 2 weeks
Sponsored by: ScaleEngine Inc.
Differential Revision: https://reviews.freebsd.org/D3929

Add new include path for sha256.h

This fixes the bootstrap build on FreeBSD 10.

Submitted by: andrew

Revert r284417 it is not necessary anymore

Enforce overwritting SHLIBDIR

Since METAMODE has been added, sys.mk loads bsd.mkopt.mk which ends load loading
bsd.own.mk which then defines SHLIBDIR before all the Makefile.inc everywhere.

This makes /lib being populated again.

Reported by: many

Unbreak MIPS build following rev. 282726

Introduce further adjustments to the renaming of libmd
symbols: make sure that we do not generate dangling weak
aliases, as this causes build failures on MIPS.

Tested by: sbruno

replace the kernel's version w/ cperciva's implementation... In all
my tests, it is faster ~20%, even on an old IXP425 533MHz it is ~45%
faster... This is partly due to loop unrolling, so the code size does
significantly increase... I do plan on committing a version that
rolls up the loops again for smaller code size for embedded systems
where size is more important than absolute performance (it'll save ~6k
code)...

The kernel implementation is now shared w/ userland's libcrypt and
libmd...

We drop support for sha256 from sha2.c, so now sha2.c only contains
sha384 and sha512...

Reviewed by: secteam@

Remove the RFC 1319 MD2 Message-Digest Algorithm routines from libmd.

1. The licensing terms for the MD2 routines from RFC is not under a BSD-like
license. Instead it is only granted for non-commercial Internet
Privacy-Enhanced Mail.
2. MD2 is quite deprecated as it is no longer considered a cryptographically
strong algorithm.

Discussed with: so (cperciva), core

Add SHA512 (Actually, this is Colin Percival's code for SHA256, with
relevant constants changed).

While I'm here clean up the tests and Makefile.

PR: misc/124164
Submitted by: KIMURA Yasuhiro < yasu utahime org >
MFC after: 1 month

Fix some leftover binaries and shared libraries in the system that still
have an executable stack, due to linking in hand-assembled .S or .s
files, that have no .GNU-stack sections:

RWX --- --- /lib/libcrypto.so.6
RWX --- --- /lib/libmd.so.5
RWX --- --- /lib/libz.so.6
RWX --- --- /lib/libzpool.so.2
RWX --- --- /usr/lib/liblzma.so.5

These were found using scanelf, from the sysutils/pax-utils port.

Reviewed by: kib

A new jail(8) with a configuration file, to replace the work currently done
by /etc/rc.d/jail.

Create the altix project branch. The altix project will add support
for the SGI Altix 350 to FreeBSD/ia64. The hardware used for porting
is a two-module system, consisting of a base compute module and a
CPU expansion module. SGI's NUMAFlex architecture can be an excellent
platform to test CPU affinity and NUMA-aware features in FreeBSD.

Build lib/ with WARNS=6 by default.

Similar to libexec/, do the same with lib/. Make WARNS=6 the norm and
lower it when needed.

I'm setting WARNS?=0 for secure/. It seems secure/ includes the
Makefile.inc provided by lib/. I'm not going to touch that directory.
Most of the code there is contributed anyway.

Make the "test" target test the compiled version, instead of the
installed version of the md library.

- Import the HEAD csup code which is the basis for the cvsmode work.

In light of the recent 2^69 operation collision-finding attack on SHA1,
add support for SHA256.

Tested on: i386, sparc64
Tested using: NIST test vectors, built-in tests
X-MFC-after: 5.4-RELEASE

Backed out rev.1.6 and subsequent copying of it (bogus addition of
-static to CFLAGS). It just turned rev.1.5 into an obfuscated no-op.
As explained in the log for rev.1.5, testing should be done in the
host environment but there is a problem in cross-compilation environments.
As not explained in the log for rev.1.6, there was apparently a practical
problem with cross-compiling (makeworld should have set -static in
LDFLAGS but apparently didn't). Cross-compilation was especially
complicated because the relevant programs are test programs that were
run at beforeinstall time -- dynamic libraries might or might not exist
depending on the build options. The complications became moot in
rev.1.8 when beforeinstall was renamed "test".

style.Makefile(5)

Stage 3 of dynamic root support. Make all the libraries needed to run
binaries in /bin and /sbin installed in /lib. Only the versioned files
reside in /lib, the .so symlink continues to live /usr/lib so the
toolchain doesn't need to be modified.

Initiate deorbit burn for the i386-only a.out related support. Moves are
under way to move the remnants of the a.out toolchain to ports. As the
comment in src/Makefile said, this stuff is deprecated and one should not
expect this to remain beyond 4.0-REL. It has already lasted WAY beyond
that.

Notable exceptions:
gcc - I have not touched the a.out generation stuff there.
ldd/ldconfig - still have some code to interface with a.out rtld.
old as/ld/etc - I have not removed these yet, pending their move to ports.
some includes - necessary for ldd/ldconfig for now.

Tested on: i386 (extensively), alpha

Don't build manuals at inappropriate time.
Collapse generation of md[245].3 manpages.

MAN[1-9] -> MAN.

Add a new entrypoint to the hashes in libmd:
char *
FooFileChunk(const char *filename, char *buf, off_t offset, off_t length)
Which only hashes part of a file.
Implement FooFile() in terms of this function.

Submitted by: roam

Replace beforeinstall target with new variables used by .mk system.

Reviewed by: marcel, and make world

$Id$ -> $FreeBSD$

add more MLINKS

Don't even attempt to build the assembly-language versions of RIPEMD160
and SHA-1 when OBJFORMAT is not ELF. Add a warning to the man page
about how SHA-1 uses bswapl, which will trap on 80386es (and the kernel
should, but doesn't currently, emulate).

Add Eric Young's RIPEMD160 implementation as well, in case SHA-1
should prove weak. Also fix a few problems with the SHA-1 build.

Fix bug in MDx test suite.
Add Eric Young's SHA-[01] implementations.

Build libmd shared for a.out too. Required for some PAM modules.

$@ -> ${.TARGET}

BINFORMAT -> OBJFORMAT ready for E-day.

Don't use the beforedepend target. It was a no-op here except for
helping bsd.dep.mk break `make -jN depend'.

If not building a.out, still build a pic version of libmd. This allows
things like libskey.so to be dynamically self contained.

Things like md5(1) where speed is critical should still link with libmd.a,
but for things like login, where it's a once-off call if skey is used, it's
not worth the hassle.

Revert $FreeBSD$ to $Id$

Make the long-awaited change from $Id$ to $FreeBSD$

This will make a number of things easier in the future, as well as (finally!)
avoiding the Id-smashing problem which has plagued developers for so long.

Boy, I'm glad we're not using sup anymore. This update would have been
insane otherwise.

cmp -s || install -c ==> install -C, use a .for loop

Don't install mdX.3, it's the template used to build the other pages.

Bill was a little to fast here...

Add some missing MLINKS, correct some cross references, correct some
file locations and some minor formatting/style problems.

Added some missing MLINKS for section 3 man pages.
Also corrected a few minor formatting errors, file location and cross
references in some of the section 3 man pages.

This shuts up a lot of the output from "manck" for section 3.

Back out that last change; we don't build this shared.

Bumped shared lib rev to 2.1.

Change `install' to `${INSTALL}' so that default install flags can be
specified in the top level Makefiles.

Previously I missed dozens of Makefiles that skip the install after
using `cmp -s' to decide that the install isn't necessary.

Change this to do what it should have done from the start.
Add argument for buffer for output.
Fix manuals.

Remove trailing whitespace.

Eliminate a bogus tab.

Clean up beforeinstall

Added beforeinstall rule to install .h files. We might need some .mk
support for this kind of thing. Look at src/Makefile target "includes".

libmd no longer built as shared-lib, only static.
Renamed the beforeinstall to test.

Build *.3 and *.ref at build time, not at install time.

Make mddriver static.
Submitted by: jkh

Use ${LDFLAGS} instead of static for compiling binaries. Neither is
sufficient for cross compiling but it's best to test with the flags
normally used.

Use ${ECHO} instead of `echo' so that `make -s' is fairly quiet.

Added ${.CURDIRb to vuilding of man pages so it works with obj

Submitted by: Paul Richards

Boy, was *this* ever bollixed!

1. Copyright files looked for in the wrong place

2. cmp was looking in wrong place for test data.

3. Driver for test not linked static, thus dynamic resolution of library
not working.

4. Man page installation not consistent with source.

Reviewed by:
Submitted by: jkh

Make obj directory work right. md?hl.c should probably be
deleted, but I'll leave that up to PHK.

Reviewed by: phk
Imported libmd. This library contains MD2, MD4 and MD5.
These three boggers pop up all over the place all of the time, so I
decided we needed a library with them. In general they are used for
security checks, so if you use them you want to link them static.