MFC r360613:

Fix intermittent cleanup failures in the ggated test

MFC 348206,348231,348454: GELI crypto deprecation warnings.

348206:
Add deprecation warnings for weaker algorithms to geli(4).

- Triple DES has been formally deprecated in Kerberos (RFC 8429)
and is soon to be deprecated in IPsec (RFC 8221).
- Blowfish is deprecated. FreeBSD doesn't support its successor
(Twofish).
- MD5 is generally considered a weak digest that has known attacks.

geli refuses to create new volumes using these algorithms via 'geli
init'. It also warns when attaching to existing volumes or creating
temporary volumes via 'geli onetime' . The plan is to fully remove
support for these algorithms in FreeBSD 13.

Note that none of these algorithms have ever been the default
algorithm used by geli(8). Users would have had to explicitly select
these algorithms when creating volumes in the past.

348231:
Correct the argument passed to g_eli_algo2str()

348454:
Remove tests for the deprecated algorithms in r348206

The tests are failing because the return value and output have changed, but
before test code structure adjusted, removing these test cases help people
be able to focus on more important cases.

Approved by: re (gjb)
Relnotes: yes

MFC r341390, r341392, r341667

r341390:
Remove some dead code from the geli tests

This is detritus in the Makefile, leftover from 327662.

r341392:
Unbreak geli/gmirror testcases if their geom classes cannot be loaded

The problem with the logic prior to this commit was twofold:

1. The wrong set of idioms (TAP-compatible) were being applied to the ATF
testcases when run, resulting in confusing ATF failure results on setup.
2. The cleanup subroutines were broken when the geom classes could not be
loaded as they exited with 0 unexpectedly.

This commit changes the test code to source the class-specific configuration
(conf.sh) once globally, instead of sourcing it per testcase and per cleanup
subroutine, and to call the ATF-specific setup subroutine(s) inline in
the testcases.

The refactoring done is effectively a no-op for the TAP testcases, modulo
any refactoring done to create common code between the ATF and TAP
testcases.

This unbreaks the geli testcases converted to ATF in r327662 and r327683,
and the gmirror testcases added in r327780, respectively, when the geom
class could not be loaded.

tests/sys/geom/class/mirror/...
While here, ignore errors when turning debug failpoint sysctl off, which
could occur if the gmirror class was not loaded.

Submitted by: ngie
Pull Request: https://github.com/freebsd/freebsd/pull/241

r341667:
geom tests: Fix cleanup of ATF tests since r341392

r341392 changed common test cleanup routines in a way that allowed them to
be used by TAP tests as well as ATF tests. However, a late change made
during code review resulted in cleanup being broken for ATF tests, which
source geom_subr.sh separately during the body and cleanup phases of the
test. The result was that md(4) devices wouldn't get cleaned up.

X-MFC-With: 341392

MFC r329273, r329275, r329277, r329284, r329344

r329273:
geli: append "/eli" to the underlying provider's physical path

If the underlying provider's physical path is null, then the geli device's
physical path will be, too. Otherwise, it will append "/eli". This will make
geli work better with zfsd(8).

PR: 224962
Differential Revision: https://reviews.freebsd.org/D13979

r329275:
gpart: append partition name to the underlying provider's physical path

If the underlying provider's physical path is null, then the gpart device's
physical path will be, too. Otherwise, it will append the partition name,
such as "/p1" or "/s1/a". This will make gpart work better with zfsd(8).

PR: 224965
Differential Revision: https://reviews.freebsd.org/D14010

r329277:
Add mtree entry for 329275

X-MFC-With: 329275
Sponsored by: Spectra Logic Corp

r329284:
zfsd: Allow zfsd to work on any type of GEOM provider

cddl/usr.sbin/zfsd/zfsd_event.cc
Remove the check for da and ada devices. This way zfsd can work on md,
geli, glabel, gstripe, etc devices. geli in particular is useful
combined with ZFS. gnop is also useful for simulating drive pulls in
the ZFSD test suite.

Also, eliminate the DevfsEvent class entirely. Move its
responsibilities into GeomEvent. We can get everything we need to know
just from listening to GEOM events.

lib/libdevdctl/event.cc
Fix GeomEvent::DevName for CREATE events. Oddly, the relevant field is
named "cdev" for CREATE events but "devname" for disk events.

Relnotes: Yes (probably worth mentioning the geli part)
Sponsored by: Spectra Logic Corp

r329344:
Optimize zfsd for the happy case

If there are no damaged pools, then ignore all GEOM events. We only use
them to fix damaged pools. However, still pay attention to ZFS events.

X-MFC-With: 329284
Sponsored by: Spectra Logic Corp

MFC r328108:

gnop(8): add the ability to set a nop provider's physical path

While I'm here, expand the existing tests a bit.

Differential Revision: https://reviews.freebsd.org/D13579

MFC geli test suite changes

MFC r306188, r306395, r327346-r327347, r327352-r327353, r327662, r327666, r327682-r327683, r327685

r306188 by br:
Use bsdlabel as we don't have hardlink disklabel -> bsdlabel
on some platforms.

Reviewed by: ngie
Sponsored by: DARPA, AFRL
Sponsored by: HEIF5
Differential Revision: https://reviews.freebsd.org/D7968

r306395 by br:
Increase timeouts for geli tests. It takes 2-3x more time to proceed the
tests on MIPS64EB in QEMU.

Sponsored by: DARPA, AFRL
Sponsored by: HEIF5

r327346:
Fix potential TOCTTOU bug in the geli tests

This change mostly reverts r293436, which introduced the bug due to a belief
that geli(8) would allocate md(4) devices by itself. However, that belief is
incorrect. Instead of using linear probing to find available md(4) numbers,
it's best to use the existing attach_md function.

Reviewed by: ngie
Sponsored by: Spectra Logic Corp
Differential Revision: https://reviews.freebsd.org/D13666

r327347:
geli: factor out some common code in the geli tests

No functional change.

Sponsored by: Spectra Logic Corp

r327352:
Fix a harmless typo from r310786

I copy/pasted a reference to an undefined shell variable.

r327353:
geli: fix the resize test on arm64

The resize test used bsdlabel(8), which is not available on all
architectures. Change it to use gpart(8) instead, which should be available
everywhere.

PR: 221763
Reported by: andrew

r327662:
geli: convert most tests from TAP to ATF

I'm leaving readonly_test and nokey_test alone for now. In a future commit
they should be broken up into several smaller test cases and distributed
between multiple files.

Reviewed by: ngie
Differential Revision: https://reviews.freebsd.org/D13717

r327666:
geli: fix parallel execution of tests

The trick is not to destroy an md(4) device during a test. That can create
a "double-free" situation, because we also destroy md devices during test
cleanup.

r327682:
Fix typo from r327666

X-MFC-With: 327666

r327683:
geli: convert remaining TAP tests to ATF

r327685:
geli: optimize tests

Reduce the geli tests' runtime by about a third:

* In integrity_test:copy, use a file-backed md(4) device instead of a
malloc'd one. That way we can corrupt the underlying storage without
needing to detach and reattach the geli device.

* In integrity_test:{copy, hmac, data} and onetime_test:{onetime,
onetime_a}, move reads of /dev/random out of the loop.

MFC r327779, r327780:
Fix handling of read errors during synchronization.

MFC r315292

r315292 (by glebius)

Fix buildworld broken in r315230 when /sys doesn't point into a right
source tree. Usage of SYSDIR is apparently wrong here.

Discussed with: allanjude

MFC r313962, r313972-r313973, r315230

r313962 by allanjude:
improve PBKDF2 performance

The PBKDF2 in sys/geom/eli/pkcs5v2.c is around half the speed it could be

GELI's PBKDF2 uses a simple benchmark to determine a number of iterations
that will takes approximately 2 seconds. The security provided is actually
half what is expected, because an attacker could use the optimized
algorithm to brute force the key in half the expected time.

With this change, all newly generated GELI keys will be approximately 2x
as strong. Previously generated keys will talk half as long to calculate,
resulting in faster mounting of encrypted volumes. Users may choose to
rekey, to generate a new key with the larger default number of iterations
using the geli(8) setkey command.

Security of existing data is not compromised, as ~1 second per brute force
attempt is still a very high threshold.

PR: 202365
Original Research: https://jbp.io/2015/08/11/pbkdf2-performance-matters/
Submitted by: Joe Pixton <jpixton@gmail.com> (Original Version), jmg (Later Version)
Reviewed by: ed, pjd, delphij
Approved by: secteam, pjd (maintainer)
Differential Revision: https://reviews.freebsd.org/D8236

r313972 by ngie:
Unbreak the build when "make obj" is executed beforehand

Using relative paths imply working directory (in this case .OBJDIR), whereas the
sources live in the .CURDIR-relative path.

X-MFC with: r313962
Pointyhat to: allanjude
Sponsored by: Dell EMC Isilon

r313973 by ngie:
A forced commit to note other portion of the Makefile change accidentally
committed in r313972

The code committed in r313962 implicitly relies on python 2.x to generate
testvect.h . There are a handful of issues with this approach:

- python is not an explicit build dependency for FreeBSD
- python 2.x is deprecated and will be removed sometime in the future
(potentially before 11.x's EOL), and the script does not function with
python 3.5 (it uses deprecated idioms and incompatible function calls).
- python(1) (by default) lives in /usr/local/bin (${LOCALBASE}/bin) and
gentestvect.py is a dependency of testvect.h (prior to r313972) which
means that if the mtime of the generator script was newer than the
mtime of the test vector, it could cause a spurious build failure in
build time or at install time.

A better solution using C/C++ should be devised.

Discussed with: allanjude
X-MFC with: r313962, r313972
Sponsored by: Dell EMC Isilon

r315230 by ngie:
Move .../sys/geom/eli/pbkdf2... to .../sys/geom/class/eli/...

This change moves the tests added in r313962 to an existing directory
structure used by the geli TAP tests. It also, renames the test from
pbkdf2 to pbkdf2_test .

The changes to ObsoleteFiles.inc are being committed separately as they
aren't needed for the MFC to ^/stable/11, etc, if the MFC for the tests
is done all in one commit.

X-MFC with: r313962, r313972-r313973
Reviewed by: allanjude
Sponsored by: Dell EMC Isilon
Differential Revision: D9985

MFC r326861-r326863:
Add some gmirror tests and fix indentation in existing tests.

MFC r326877:
Skip gnop tests if the corresponding kernel module isn't available.

MFC r324478:

Check the exit code from fsck_ffs instead of relying on MODIFIED being in the output

^/head@r323923 changed when MODIFIED is printed at exit. It's better to follow the
documented way of determining whether or not a filesystem is clean per fsck_ffs, i.e.,
ensure that the exit code is either 0 or 7.

The pass/fail determination is brittle prior to this commit, and ^/head@r323923 made
the issue apparent -- thus this needs to be fixed independent of ^/head@r323923.

PR: 222780
MFC with: r323923

MFC note: tests/sys/kern/coredump_phnum_test.sh changes omitted

MFC r322214:

Make test scripts under tests/... non-executable

Executable bits should be set at install time instead of in the repo.
Setting executable bits on files triggers false positives with Phabricator.

MFC r317713:
r317713 (by markj):

Add regression tests for r317712 and r306743.

MFC r321702,r321703:

r321702:

Load geom_gate(4) if necessary before running tests; skip if it can't be loaded

The test code prior to r311893 loaded geom_gate at test start if necessary and
skipped the tests if it couldn't be loaded.

The ATF-ifcation of this test done in r311893 unfortunately dropped this
functionality.

This change restores the geom_gate module load and skips the test(s) if unavailable
in an ATF-like way.

PR: 220164

r321703:

Remove superfluous `exit 0` added in r321702

atf_skip triggers equivalent functionality, which means the `exit 0`
is unreachable code.

PR: 220164
MFC with: r321702

MFC r311893, r313008, r313081

r311893:
ATFify the geom gate tests.

This ensures their cleanup routines will be run even if they should timeout.

tests/sys/geom/class/gate/ggate_test.sh
tests/sys/geom/class/gate/Makefile
Add an ATF test with three testcases, one for each TAP test. Use
ATF-style cleanup functions, and convert sleeps to polling loops.

ObsoleteFiles.inc
tests/sys/geom/class/gate/conf.sh
tests/sys/geom/class/gate/1_test.sh
tests/sys/geom/class/gate/2_test.sh
tests/sys/geom/class/gate/3_test.sh
Delete TAP test files

Reviewed by: ngie
MFC after: 4 weeks
Sponsored by: Spectra Logic Corp
Differential Revision: https://reviews.freebsd.org/D8891

r313008:
Wait for /dev/ggate* to appear after calling `ggatel create` in :ggatel_{file,md}

The test assumed that `ggatel create` created a device on completion, but that's
incorrect. This squashes the race by waiting for the device to appear, as
`ggatel create` daemonizes before issuing an ioctl to geom_gate(4) if not called
with `-v`.

Discussed with: asomers
MFC after: 1 week
PR: 204616
Sponsored by: Dell EMC Isilon

r313081:
Replace for/retry loops with "wait_for_ggate_device" calls and check
results of commands

As noted in r313008, the underlying issue was that geom_gate device
creation wasn't created at ggatel command completion, but some short
time after. ggatec(8) employs similar logic when creating geom_gate(4)
devices.

Switch from retry loops (after the ggatec/dd write calls) to
wait_for_ggate_device function calls after calling ggatec(8) instead
to detect the presence of the /dev/ggate* device, as this function is
sufficient for determining whether or not the character device is ready
for testing

While here, use atf_check consistently with all dd calls to ensure that
data output is as expected.

MFC after: 1 week
Reviewed by: asomers
Differential Revision: D9409
Sponsored by: Dell EMC Isilon

MFC r310786, r310803, r310985, r311894

r310786:
Reduce the runtime of the GELI tests

There is no reduction in test coverage. On my system runtime is reduced
from 38m32s to 6m24s.

tests/sys/geom/class/eli/conf.sh
tests/sys/geom/class/eli/init_a_test.sh
tests/sys/geom/class/eli/init_test.sh
tests/sys/geom/class/eli/integrity_copy_test.sh
tests/sys/geom/class/eli/integrity_data_test.sh
tests/sys/geom/class/eli/integrity_hmac_test.sh
tests/sys/geom/class/eli/onetime_a_test.sh
tests/sys/geom/class/eli/onetime_test.sh
Move the looping code into common functions in conf.sh, and remove
alias ciphers from the list.

tests/sys/geom/class/eli/init_a_test.sh
tests/sys/geom/class/eli/init_test.sh
tests/sys/geom/class/eli/integrity_copy_test.sh
tests/sys/geom/class/eli/integrity_data_test.sh
tests/sys/geom/class/eli/integrity_hmac_test.sh
tests/sys/geom/class/eli/onetime_a_test.sh
Move a few commands that don't need to be in the inner loop out.

tests/sys/geom/class/eli/init_test.sh
tests/sys/geom/class/eli/onetime_a_test.sh
Reduce the sector count

tests/sys/geom/class/eli/Makefile
tests/sys/geom/class/eli/init_alias_test.sh
Add a test for initializing a GELI device using one of the cipher
aliases, and check that the alias is correctly interpreted.

MFC after: 4 weeks
Sponsored by: Spectra Logic Corp
Differential Revision: https://reviews.freebsd.org/D8814

r310803:
ATFify the gnop tests

Also, add test cases for the -p, -P, and -s options to gnop create

Reviewed by: ngie
MFC after: 4 weeks
Differential Revision: https://reviews.freebsd.org/D8892

r310985:
Update ObsoleteFiles.inc for r310803

MFC after: 26 days
X-MFC-with: 310803

r311894:
Fix typo from change 310985 in ObsoleteFiles.inc

MFC after: 16 days
X-MFC-With: 310803
Sponsored by: Spectra Logic Corp

Copy head@r302406 to stable/11 as part of the 11.0-RELEASE cycle.
Prune svn:mergeinfo from the new branch, as nothing has been merged
here.

Additional commits post-branch will follow.

Approved by: re (implicit)
Sponsored by: The FreeBSD Foundation

Merge ^/user/ngie/release-pkg-fix-tests to unbreak how test files are installed
after r298107

Summary of changes:

- Replace all instances of FILES/TESTS with ${PACKAGE}FILES. This ensures that
namespacing is kept with FILES appropriately, and that this shouldn't need
to be repeated if the namespace changes -- only the definition of PACKAGE
needs to be changed
- Allow PACKAGE to be overridden by callers instead of forcing it to always be
`tests`. In the event we get to the point where things can be split up
enough in the base system, it would make more sense to group the tests
with the blocks they're a part of, e.g. byacc with byacc-tests, etc
- Remove PACKAGE definitions where possible, i.e. where FILES wasn't used
previously.
- Remove unnecessary TESTSPACKAGE definitions; this has been elided into
bsd.tests.mk
- Remove unnecessary BINDIRs used previously with ${PACKAGE}FILES;
${PACKAGE}FILESDIR is now automatically defined in bsd.test.mk.
- Fix installation of files under data/ subdirectories in lib/libc/tests/hash
and lib/libc/tests/net/getaddrinfo
- Remove unnecessary .include <bsd.own.mk>s (some opportunistic cleanup)

Document the proposed changes in share/examples/tests/tests/... via examples
so it's clear that ${PACKAGES}FILES is the suggested way forward in terms of
replacing FILES. share/mk/bsd.README didn't seem like the appropriate method
of communicating that info.

MFC after: never probably
X-MFC with: r298107
PR: 209114
Relnotes: yes
Tested with: buildworld, installworld, checkworld; buildworld, packageworld
Sponsored by: EMC / Isilon Storage Division

Merge the projects/release-pkg branch to head.

This allows packaging the base system with pkg(8), including
but not limited to providing the ability to provide upstream
binary update possibilities for non-tier-1 architectures.

This merge is a requirement of the 11.0-RELEASE, and as such,
thank you to everyone that has tested the project branch.

Documentation in build(7) etc. is still somewhat sparse, but
updates to those parts will follow.

Sponsored by: The FreeBSD Foundation

Use a SKIP testplan instead of bailing out if/when the tester isn't
root, or the geom class can't be loaded cleanly [*]

This makes sure that scenarios that are easy to hit aren't counted
as false positives with kyua test

MFC after: 1 week
PR: 208101
Sponsored by: EMC / Isilon Storage Division

DIRDEPS_BUILD: Connect MK_TESTS.

Sponsored by: EMC / Isilon Storage Division

Similar to r295116, add an additional 1 second sleep after calling ggatel
before calling dd to defeat a race when writing out to the geom_gate(4)
device

MFC after: 1 month
Reported by: Jenkins
Sponsored by: EMC / Isilon Storage Division

Use the pidfile support added to ggated(8) in r294973 to ensure that the
ggated(8) daemon used by the tests is the instance specifically invoked by
the tests instead of one or more daemon instances running on the system

MFC after: 1 month
Sponsored by: EMC / Isilon Storage Division

Add an additional 1 second sleep to after calling ggatec before calling
dd to defeat a race when writing out to the geom_gate(4) device

This will quell the Jenkins failure emails until I come up with a better
solution

MFC after: 1 month
Reported by: Jenkins
Sponsored by: EMC / Isilon Storage Division

PID file support hasn't been committed for ggated(8) yet. Unbreak running
the testcase more than once by restoring the "killall ggated"

MFC after: 15 days
Sponsored by: EMC / Isilon Storage Division

Remove unnecessary kldload logic added to geom_subr.sh in r293028

MFC after: 2 weeks
Sponsored by: EMC / Isilon Storage Division

Integrate
tools/regression/geom_{concat,eli,gate,mirror,nop,raid3,shsec,stripe,uzip}
in to the FreeBSD test suite as
tests/sys/geom/class/{concat,eli,gate,mirror,nop,raid3,shsec,stripe,uzip}

The tools/regression/geom and tools/regression/geom_part testcases are being
left alone because both test sets are both currently broken.

The majority of this work was done on ^/user/ngie/more-tests2 . The differences
are as follows:
- tests/sys/geom/class/Makefile.inc is not present; it was
inlined into the class's Makefiles for explicitness.
- The testcases officially require root via kyua
- The geom_gate(4) tests don't use the pidfile changes proposed in
https://reviews.freebsd.org/D4836 .

MFC after: 1 month
Sponsored by: EMC / Isilon Storage Division