#
d0b2dbfa |
|
16-Aug-2023 |
Warner Losh <imp@FreeBSD.org> |
Remove $FreeBSD$: one-line sh pattern Remove /^\s*#[#!]?\s*\$FreeBSD\$.*$\n/
|
#
194d5628 |
|
06-Oct-2020 |
Li-Wen Hsu <lwhsu@FreeBSD.org> |
Make capsicum test cases fine-grained Add a wrapping script to use ATF to run tests written with Googletest one by one. This helps locating and tracking the failing case in CI easier. This is a temporarily solution while Googletest support in Kyua is developing. We will revert this once Kyua+Googletest integration is ready. Reviewed by: emaste Sponsored by: The FreeBSD Foundation Differential Revision: https://reviews.freebsd.org/D25896
|
#
bd71398d |
|
19-Feb-2019 |
Enji Cooper <ngie@FreeBSD.org> |
Add googletest.test.mk and integrate into bsd.test.mk googletest.test.mk is a rudimentary wrapper around the plain test interface (for now), which only supports C++ programs, specified by the `GTESTS` variable. In the future, kyua will support gtests in a more native manner.
|
#
f324fafc |
|
27-Jul-2018 |
Eitan Adler <eadler@FreeBSD.org> |
Feex a cuple of small typos
|
#
2474da32 |
|
17-Jul-2017 |
Enji Cooper <ngie@FreeBSD.org> |
Sort the tests alphabetically before adding them to the Kyuafiles This is being done to aid in debugging test runs, in the event the output shifts due to refactored Makefiles, added tests, etc. MFC after: 1 month
|
#
430f7286 |
|
04-May-2016 |
Enji Cooper <ngie@FreeBSD.org> |
Merge ^/user/ngie/release-pkg-fix-tests to unbreak how test files are installed after r298107 Summary of changes: - Replace all instances of FILES/TESTS with ${PACKAGE}FILES. This ensures that namespacing is kept with FILES appropriately, and that this shouldn't need to be repeated if the namespace changes -- only the definition of PACKAGE needs to be changed - Allow PACKAGE to be overridden by callers instead of forcing it to always be `tests`. In the event we get to the point where things can be split up enough in the base system, it would make more sense to group the tests with the blocks they're a part of, e.g. byacc with byacc-tests, etc - Remove PACKAGE definitions where possible, i.e. where FILES wasn't used previously. - Remove unnecessary TESTSPACKAGE definitions; this has been elided into bsd.tests.mk - Remove unnecessary BINDIRs used previously with ${PACKAGE}FILES; ${PACKAGE}FILESDIR is now automatically defined in bsd.test.mk. - Fix installation of files under data/ subdirectories in lib/libc/tests/hash and lib/libc/tests/net/getaddrinfo - Remove unnecessary .include <bsd.own.mk>s (some opportunistic cleanup) Document the proposed changes in share/examples/tests/tests/... via examples so it's clear that ${PACKAGES}FILES is the suggested way forward in terms of replacing FILES. share/mk/bsd.README didn't seem like the appropriate method of communicating that info. MFC after: never probably X-MFC with: r298107 PR: 209114 Relnotes: yes Tested with: buildworld, installworld, checkworld; buildworld, packageworld Sponsored by: EMC / Isilon Storage Division
|
#
f68f6b3d |
|
08-Mar-2016 |
Bryan Drewery <bdrewery@FreeBSD.org> |
Remove things set already by bsd.progs.mk. MFC after: 2 weeks Sponsored by: EMC / Isilon Storage Division
|
#
342863e7 |
|
26-Feb-2016 |
Bryan Drewery <bdrewery@FreeBSD.org> |
Move PROGS logic to proper place and remove redundant and unneeded logic. - bsd.progs.mk is safe to include regardless of PROGS/PROGS_CXX/SCRIPTS being set. - bsd.progs.mk includes bsd.prog.mk always and will bring in bsd.files.mk and bsd.obj.mk. - DIRDEPS_BUILD: There's no need for _SKIP_BUILD or _SKIP_STAGING as the PROGS were implicitly being built by the staging dependency anyway. This was also preventing other objects from building if they were not part of the staging sets. - DIRDEPS_BUILD: This fixes PROGS without bsd.test.mk not staging. MFC after: 2 weeks Sponsored by: EMC / Isilon Storage Division
|
#
d3157f09 |
|
15-Feb-2016 |
Bryan Drewery <bdrewery@FreeBSD.org> |
Test directories can build in parallel fine. Sponsored by: EMC / Isilon Storage Division
|
#
71b7fa12 |
|
07-Feb-2016 |
Enji Cooper <ngie@FreeBSD.org> |
Simplify running the FreeBSD test suite Replace `make regress` (legacy test make target) and `make test` (incomplete test make target added with the FreeBSD test suite) with make check as it's consistent with other open source projects. `make check` defaults to running tests from `.OBJDIR`, but can be overridden with the `CHECKDIR` variable. Add `make checkworld` target to simplify running the FreeBSD test suite from `TESTSBASE` (i.e. the top-level tests directory), similar to buildworld. Document `make check` and `make checkworld` in build(7). Other minor changes: - Rename intermediate file (`Kyuafile.auto`) to `Kyuafile` to simplify `make check`. - Remove terse warnings attached to `beforetest`/`aftertest`. - Add kyua binary check to check target in suite.test.mk; error out if it's not found The MFC is [partly] contingent on other build related changes being MFCed. Differential Revision: https://reviews.freebsd.org/D4406 MFC after: 2 months X-MFC to: stable/10 Relnotes: yes Reviewed by: bdrewery, Evan Cramer <eccramer@gmail.com> Sponsored by: EMC / Isilon Storage Division
|
#
26dfa135 |
|
19-Dec-2015 |
Enji Cooper <ngie@FreeBSD.org> |
- Use LOCALBASE instead of KYUA_PREFIX for the --prefix to kyua(1) - Use LOCALBASE instead of hardcoding /usr/local for perl MFC after: 1 week Differential Revision: https://reviews.freebsd.org/D4406 (part of a larger diff) Reviewed by: emaste, Evan Cramer <eccramer@gmail.com> Sponsored by: EMC / Isilon Storage Division
|
#
da3110ed |
|
11-Oct-2015 |
Enji Cooper <ngie@FreeBSD.org> |
Default TESTSDIR to /usr/tests/${RELDIR:H} When run from bin/ls/tests, for example, the value of TESTSDIR would be ${TESTSBASE}/${RELDIR:H} -> /usr/tests/bin/ls/tests/.. -> /usr/tests/bin/ls Document the new behavior in bsd.README. While here, also document TESTSBASE Relnotes: yes Differential Revision: D1022 MFC after: 2 weeks Sponsored by: EMC / Isilon Storage Division
|
#
abb02fa2 |
|
23-Sep-2015 |
Bryan Drewery <bdrewery@FreeBSD.org> |
Fix most cases of bsd.progs.mk running duplicate or missing commands. This mostly fixes an interaction with bsd.test.mk with PROGS and SCRIPTS. This was most notable with 'make clean' and 'make install', which r281055 and r272055 attempted to address but were inadequate. It also addresses similar issues in bsd.progs.mk when not using bsd.test.mk. This also fixes cases of NOT running commands in the parent when using bsd.progs.mk: - 'make clean' was not run for the main process for Makefiles which had both FILES and SUBDIR but no PROGS or SCRIPTS. This usually was just a leftover Kyuafile.auto. One such example is usr.bin/bmake/tests/sysmk/t1/2. - 'make obj' was not running in the current directory with bsd.test.mk due to early inclusion of bsd.subdir.mk. This was not really a problem due to the SUBDIRS using 'mkdir -p' for their objdirs. There were subtle bugs causing this wrong behavior: 1. bsd.progs.mk needs to set SCRIPTS to empty when recursing to avoid the sub-makes from installing, cleaning or building the SCRIPTS; only the parent make should be doing this. r281055 effectively did the same but wasn't enough. 2. CLEANFILES may contain (especially from *.test.mk) files which only the parent should clean, such as from FILES and SCRIPTS. To resolve sub-makes also cleaning these, reset CLEANFILES and CLEANDIRS in the children before including bsd.prog.mk. A tempting alternative would be to only handle CLEANFILES in the parent but then the child bsd.prog.mk CLEANFILES of per-PROGS wouldn't be setup. 3. bsd.subdir.mk was included too soon in bsd.test.mk. It needs to be included after bsd.prog.mk as the SCRIPTS logic is short-circuitted if 'install:' is already defined (which bsd.subdir.mk does). There is actually no need to include bsd.subdir.mk from bsd.test.mk as bsd.prog.mk and bsd.obj.mk will do so in the proper order. The description in r257095 covers this for FILES and was fixed differently, though changing the handling of target(install) in bsd.prog.mk may make sense after more research. 4. bsd.progs.mk had extra logic to handle recursing SCRIPTS if PROGS was empty, which isn't its business to be doing. SCRIPTS is handled fine by bsd.prog.mk. This mostly reverts and reworks the fix in r259209 and partially reverts r272055. 5. bsd.progs.mk has no need to depend 'all:' on SCRIPTS and FILES. These are handled by bsd.prog.mk/bsd.files.mk fine. This also partially reverts r272055. 6. bsd.progs.mk was not drop-in safe for bsd.prog.mk. Move the PROGS check from r273186 to allow it to be used safely. Specific tested cases: SCRIPTS:no PROGS:no FILES:yes SUBDIR:yes usr.bin/bmake/tests/sysmk/t1/2 SCRIPTS:yes PROGS:no FILES:yes SUBDIR:no usr.bin/bmake/tests/sysmk/t1/2/1 SCRIPTS:yes PROGS:yes FILES:yes SUBDIR:yes lib/libthr/tests SCRIPTS:yes PROGS:no FILES:yes SUBDIR:no usr.bin/yacc/tests libexec/atf/atf-sh/tests A full buildworld/installworld/clean comparison with mtree was also done. The only relevant difference was the new fixed behavior of removing Kyuafile.auto from the objdir in 'clean'. Converting SCRIPTS to be a special case FILES group will make this less fragile and is being explored. One known remaining issue is 'cleandepend' removing the tags files for every recursive call. Note that the 'make clean' command runs for the CURDIR last, which can make it appear to run multiple times when cleaning in tests/, but each command is for a SUBDIR returning up the chain. This is purely bsd.subdir.mk behavior. PR: 191055 PR: 191955 MFC after: 2 weeks Sponsored by: EMC / Isilon Storage Division
|
#
08796a78 |
|
16-Jun-2015 |
Baptiste Daroussin <bapt@FreeBSD.org> |
Move tests into a single FreeBSD-tests package
|
#
c4af37bb |
|
15-Jun-2015 |
Enji Cooper <ngie@FreeBSD.org> |
Ensure TESTSDIR is defined before bsd.test.mk is .include'd MFC after: 1 week
|
#
05f0ff5a |
|
28-Oct-2014 |
Enji Cooper <ngie@FreeBSD.org> |
Fix the logic inversion in the previous commit by ensuring that the matched expression (:M) is empty, not the not matched (:N) is empty. The former case means we have not found the TEST_SUBDIR value in SUBDIR Reported by: rodrigc X-MFC with: r273803 Pointyhat to: me (did not use a clean install root) Sponsored by: EMC / Isilon Storage Division
|
#
9aa97389 |
|
28-Oct-2014 |
Enji Cooper <ngie@FreeBSD.org> |
Filter out TESTS_SUBDIRS already added to SUBDIR instead of blindly appending the TESTS_SUBDIRS variable to SUBDIR Duplicate directory entries can cause unexpected side effects, like installing the same files multiple times. This can be easily reproduced via the following testcase prior to this commit: SUBDIR= dir TESTS_SUBDIRS+= dir .include <bsd.test.mk> Sponsored by: EMC / Isilon Storage Division
|
#
e8a34402 |
|
10-Jun-2014 |
Julio Merino <jmmv@FreeBSD.org> |
Put the test suite in its own tests.txz distribution file. Force all the contents of /usr/tests to go into a separate distribution file so that users of binary releases can easily choose to not install it. To make this possible, we need two fixes: - bsd.subdir.mk needs to properly honor NO_SUBDIR in all cases so that we do not recurse into 'tests' subdirectories when we needn't. Otherwise, we end up with some Kyuafiles in base.txz. - etc/Makefile needs to skip installing tests in its 'distribute' target so that a Kyuafile doesn't leak into base.txz. Approved by: gjb
|
#
ada17d7b |
|
09-Jun-2014 |
Julio Merino <jmmv@FreeBSD.org> |
Ensure files are created during the build when using bsd.subdir.mk. When FILES is defined in a Makefile that _also_ includes bsd.subdir.mk, the build of the files (if any) was not properly triggered during the build stage. This was because bsd.files.mk did not define the buildfiles target if it was already defined... and bsd.subdir.mk defined this target on its own, thus causing a conflict. Fix this by unconditionally defining buildfiles from bsd.files.mk; this is safe because nothing else in the tree needs to redefine this and because the target itself contains no commands: all it does is define dependencies. Also ensure that bsd.files.mk is always pulled in by bsd.test.mk regardless of what bsd.prog.mk does. These fixes allow "make installworld" to run cleanly on a system with read-only src and obj trees. This is "make tinderbox" clean. Reviewed by: imp Obtained from: jilles
|
#
c115b818 |
|
10-May-2014 |
Warner Losh <imp@FreeBSD.org> |
Remove last two NO_MAN= in the tree. In both of these cases, MAN= is what is needed.
|
#
40b86d77 |
|
05-Apr-2014 |
Warner Losh <imp@FreeBSD.org> |
The proper way to request no man pages currently is NO_MAN=xxx. Use it in preference to the user WITHOUT_MAN knob, which should never be set in normal src Makefiles.
|
#
a784098c |
|
13-Mar-2014 |
Julio Merino <jmmv@FreeBSD.org> |
Move FreeBSD Test Suite-specific code to a suite.test.mk file. The new suite.test.mk file contains all the logic needed to install test programs under /usr/tests/ and to support Kyua as the run-time engine. This file is included by default by bsd.test.mk so Makefiles do not need to care about its existence. Specific Makefiles can define NOT_FOR_TEST_SUITE to indicate that whatever test programs they are building are not supposed to be installed under /usr/tests/ nor run by Kyua. (The effect of passing this setting is that suite.test.mk is simply not included.) NOT_FOR_TEST_SUITE should never be used by Makefiles in the base system. This functionality is provided so that third-parties can hook in their own test code, with different semantics, if they wish. This was asked for by sjg@.
|
#
f5fd950e |
|
14-Mar-2014 |
Julio Merino <jmmv@FreeBSD.org> |
Make bsd.test.mk the only public mk fragment for the building of tests. Change {atf,plain,tap}.test.mk to be internal implementation details of bsd.test.mk. Makefiles that build tests should now only include bsd.test.mk and declaratively specify what they want to build, without worrying about the internal implementation of the mk files. The reason for this change is to permit building test programs of different interfaces from a single directory, which is something I had a need for while porting tests over from src/tools/regression/. Additionally, this change makes it possible to perform some other requested changes to bsd.test.mk in an easier manner. Coming soon.
|
#
8c465f58 |
|
14-Jan-2014 |
Julio Merino <jmmv@FreeBSD.org> |
Support defining test program metadata from the Makefiles. Introduce a new, per-test-program TEST_METADATA.<program> variable that contains a list of key/value paris describing metadata properties for that test program. These properties are later written into the auto-generated Kyuafile when using the KYUAFILE=auto functionality. This is to avoid having to supply hand-crafted Kyuafiles when the needs for metadata overrides are trivial. While doing this, and because I am documenting TEST_METADATA, take the chance to document the TEST_INTERFACE setting as well. MFC after: 5 days
|
#
d7efee23 |
|
10-Jan-2014 |
Julio Merino <jmmv@FreeBSD.org> |
Allow tests to provide a Kyuafile when they relied on auto-generation. When generating a Kyuafile in the KYUAFILE=auto case, use a filename that is unlikely to clash with the filename used by explicitly-provided Kyuafiles. This allows a Makefile to set KYUAFILE=yes and provide a Kyuafile in the same directory when such Makefile was previously relying on KYUAFILE=auto. Fixes issues with new Kyuafiles not being picked up in NO_CLEAN builds (although manual intervention is required once, unfortunately, as described in UPDATING). Reviewed by: sjg MFC after: 1 week
|
#
5e6f3a8e |
|
24-Oct-2013 |
Rui Paulo <rpaulo@FreeBSD.org> |
Move the TESTSBASE definition to bsd.own.mk. We need to be able to reference the value of TESTSBASE without requiring the inclusion of bsd.test.mk (e.g. in etc/Makefile), so move its definition to the more generic bsd.own.mk. Submitted by: Julio Merino jmmv google.com Reviewed by: sjg MFC after: 2 weeks
|
#
7bb2d4b3 |
|
19-Oct-2013 |
Rui Paulo <rpaulo@FreeBSD.org> |
Plug kyua into the 'test' target. If kyua exists in KYUA_PREFIX, the test target is automatically defined to use it for the execution of test programs. Submitted by: Julio Merino jmmv google.com MFC after: 2 weeks
|
#
30cc088d |
|
19-Oct-2013 |
Rui Paulo <rpaulo@FreeBSD.org> |
Plug atf-run into the 'test' target. If atf-run exists in ATF_PREFIX and if ALLOW_DEPRECATED_ATF_TOOLS has been set to yes, the test target is automatically defined to use it for the execution of test programs. Submitted by: Julio Merino jmmv google.com MFC after: 2 weeks
|
#
25b6a535 |
|
19-Oct-2013 |
Rui Paulo <rpaulo@FreeBSD.org> |
Add the automatic generation of Kyuafile files. These files are generated from bsd.test.mk because kyua is able to run test programs implemented using different libraries/frameworks. In order to make this possible, this change also extends the various *.test.mk file to explicitly indicate the interface of every test program. Submitted by: Julio Merino jmmv google.com MFC after: 2 weeks
|
#
26ea29af |
|
19-Oct-2013 |
Rui Paulo <rpaulo@FreeBSD.org> |
Add the automatic generation of Atffile files. These are only used by the deprecated atf-run and atf-report tools. Generating them is easy and provides a mechanism for people to experiment with these tools if they wish. But, because these tools and files are deprecated, doing this only happens if the user has explicitly set ALLOW_DEPRECATED_ATF_TOOLS to yes. Submitted by: Julio Merino jmmv google.com MFC after: 2 weeks
|
#
d4a14c85 |
|
19-Oct-2013 |
Rui Paulo <rpaulo@FreeBSD.org> |
Clearly split the logic to build ATF and plain tests apart. This change introduces a new plain.test.mk file that provides the build infrastructure to build test programs that don't use any framework. Most of the code previously in bsd.test.mk moves to plain.test.mk and atf.test.mk is extended with the missing pieces. In doing so, this change pushes all test program building logic to the various *.test.mk files instead of trying to reuse some tiny bits. In fact, this attempt to reuse some definitions makes the code harder to read and harder to extend. The clear benefit of this is that the interface of bsd.test.mk is now clearly delimited. Submitted by: Julio Merino jmmv google.com MFC after: 2 weeks
|
#
6f5639e3 |
|
12-Oct-2013 |
Simon J. Gerraty <sjg@FreeBSD.org> |
Sync-up with what's being use in Juniper for building ATF in meta mode.
|
#
50c5d27b |
|
07-Nov-2012 |
Simon J. Gerraty <sjg@FreeBSD.org> |
Enable ATF testing. Submitted by: Garrett Cooper Approved by: marcel (mentor)
|