1164190SjkoshyGuidelines for test developers 2164190Sjkoshy============================== 3164190Sjkoshy 4164190SjkoshyHow to add recipes 5164190Sjkoshy------------------ 6164190Sjkoshy 7164190SjkoshyFor any test that you want to perform, you write a script located in 8164190Sjkoshy`test/recipes/`, named `{nn}-test_{name}.t`, 9164190Sjkoshywhere `{nn}` is a two digit number and 10164190Sjkoshy`{name}` is a unique name of your choice. 11164190Sjkoshy 12164190SjkoshyPlease note that if a test involves a new testing executable, you will need to 13164190Sjkoshydo some additions in test/build.info. Please refer to the section 14164190Sjkoshy["Changes to test/build.info"](README.md#changes-to-testbuildinfo) below. 15164190Sjkoshy 16164190SjkoshyNaming conventions 17164190Sjkoshy------------------ 18164190Sjkoshy 19164190SjkoshyA test executable is named `test/{name}test.c` 20164190Sjkoshy 21164190SjkoshyA test recipe is named `test/recipes/{nn}-test_{name}.t`, where `{nn}` is a two 22164190Sjkoshydigit number and `{name}` is a unique name of your choice. 23164190Sjkoshy 24164190SjkoshyThe number `{nn}` is (somewhat loosely) grouped as follows: 25164190Sjkoshy 26164190Sjkoshy 00-04 sanity, internal and essential API tests 27164190Sjkoshy 05-09 individual symmetric cipher algorithms 28164190Sjkoshy 10-14 math (bignum) 29164190Sjkoshy 15-19 individual asymmetric cipher algorithms 30164190Sjkoshy 20-24 openssl commands (some otherwise not tested) 31164190Sjkoshy 25-29 certificate forms, generation and verification 32164190Sjkoshy 30-35 engine and evp 33164190Sjkoshy 60-79 APIs: 34164190Sjkoshy 60 X509 subsystem 35164190Sjkoshy 61 BIO subsystem 36164190Sjkoshy 65 CMP subsystem 37164190Sjkoshy 70 PACKET layer 38164190Sjkoshy 80-89 "larger" protocols (CA, CMS, OCSP, SSL, TSA) 39241720Sed 90-98 misc 40164190Sjkoshy 99 most time consuming tests [such as test_fuzz] 41164190Sjkoshy 42164190SjkoshyA recipe that just runs a test executable 43164190Sjkoshy----------------------------------------- 44164190Sjkoshy 45164190SjkoshyA script that just runs a program looks like this: 46164190Sjkoshy 47164190Sjkoshy #! /usr/bin/env perl 48164190Sjkoshy 49164190Sjkoshy use OpenSSL::Test::Simple; 50164190Sjkoshy 51164190Sjkoshy simple_test("test_{name}", "{name}test", "{name}"); 52164190Sjkoshy 53164190Sjkoshy`{name}` is the unique name you have chosen for your test. 54164190Sjkoshy 55164190SjkoshyThe second argument to `simple_test` is the test executable, and `simple_test` 56164190Sjkoshyexpects it to be located in `test/` 57164190Sjkoshy 58164190SjkoshyFor documentation on `OpenSSL::Test::Simple`, 59164190Sjkoshydo `perldoc util/perl/OpenSSL/Test/Simple.pm`. 60164190Sjkoshy 61164190SjkoshyA recipe that runs a more complex test 62164190Sjkoshy-------------------------------------- 63164190Sjkoshy 64164190SjkoshyFor more complex tests, you will need to read up on Test::More and 65164190SjkoshyOpenSSL::Test. Test::More is normally preinstalled, do `man Test::More` for 66164190Sjkoshydocumentation. For OpenSSL::Test, do `perldoc util/perl/OpenSSL/Test.pm`. 67164190Sjkoshy 68164190SjkoshyA script to start from could be this: 69164190Sjkoshy 70164190Sjkoshy #! /usr/bin/env perl 71164190Sjkoshy 72164190Sjkoshy use strict; 73164190Sjkoshy use warnings; 74164190Sjkoshy use OpenSSL::Test; 75164190Sjkoshy 76164190Sjkoshy setup("test_{name}"); 77164190Sjkoshy 78165032Sjkoshy plan tests => 2; # The number of tests being performed 79164190Sjkoshy 80164190Sjkoshy ok(test1, "test1"); 81164190Sjkoshy ok(test2, "test1"); 82164190Sjkoshy 83164190Sjkoshy sub test1 84164190Sjkoshy { 85164190Sjkoshy # test feature 1 86164190Sjkoshy } 87164190Sjkoshy 88164190Sjkoshy sub test2 89164190Sjkoshy { 90164190Sjkoshy # test feature 2 91164190Sjkoshy } 92164190Sjkoshy 93164190SjkoshyChanges to test/build.info 94164190Sjkoshy-------------------------- 95164190Sjkoshy 96164190SjkoshyWhenever a new test involves a new test executable you need to do the 97164190Sjkoshyfollowing (at all times, replace {NAME} and {name} with the name of your 98164190Sjkoshytest): 99164190Sjkoshy 100164190Sjkoshy * add `{name}` to the list of programs under `PROGRAMS_NO_INST` 101 102 * create a three line description of how to build the test, you will have 103 to modify the include paths and source files if you don't want to use the 104 basic test framework: 105 106 SOURCE[{name}]={name}.c 107 INCLUDE[{name}]=.. ../include ../apps/include 108 DEPEND[{name}]=../libcrypto libtestutil.a 109 110Generic form of C test executables 111---------------------------------- 112 113 #include "testutil.h" 114 115 static int my_test(void) 116 { 117 int testresult = 0; /* Assume the test will fail */ 118 int observed; 119 120 observed = function(); /* Call the code under test */ 121 if (!TEST_int_eq(observed, 2)) /* Check the result is correct */ 122 goto end; /* Exit on failure - optional */ 123 124 testresult = 1; /* Mark the test case a success */ 125 end: 126 cleanup(); /* Any cleanup you require */ 127 return testresult; 128 } 129 130 int setup_tests(void) 131 { 132 ADD_TEST(my_test); /* Add each test separately */ 133 return 1; /* Indicates success. Return 0 */ 134 /* to produce an error with a */ 135 /* usage message and -1 for */ 136 /* failure to set up with no */ 137 /* usage message. */ 138 } 139 140You should use the `TEST_xxx` macros provided by `testutil.h` to test all failure 141conditions. These macros produce an error message in a standard format if the 142condition is not met (and nothing if the condition is met). Additional 143information can be presented with the `TEST_info` macro that takes a `printf` 144format string and arguments. `TEST_error` is useful for complicated conditions, 145it also takes a `printf` format string and argument. In all cases the `TEST_xxx` 146macros are guaranteed to evaluate their arguments exactly once. This means 147that expressions with side effects are allowed as parameters. Thus, 148 149 if (!TEST_ptr(ptr = OPENSSL_malloc(..))) 150 151works fine and can be used in place of: 152 153 ptr = OPENSSL_malloc(..); 154 if (!TEST_ptr(ptr)) 155 156The former produces a more meaningful message on failure than the latter. 157 158Note that the test infrastructure automatically sets up all required environment 159variables (such as `OPENSSL_MODULES`, `OPENSSL_CONF`, etc.) for the tests. 160Individual tests may choose to override the default settings as required. 161