1regression test suite for security components. 2by Michael Brouwer 3 4 5GOALS 6===== 7 8The goals of this test setup are to have something that required 90 configuration and setup and allows developers to quickly write 10new standalone test cases. 11 12 13USAGE 14===== 15 16The tests are runnable from the top level Makefile by typing: 17 make test 18or individually from the command line or with gdb. Tests will be 19built into a directory called build by default or into LOCAL_BUILD_DIR 20if you set that in your environment. 21 22 23DIRECTORY LAYOUT 24================ 25 26Currently there are subdirectories for a number of different parts 27of the security stack. Each directory contains some of the unit 28tests I've managed to find from radar and other places. 29 30The test programs output their results in a format called TAP. This 31is described here: 32 http://search.cpan.org/~petdance/Test-Harness-2.46/lib/Test/Harness/TAP.pod 33Because of this we can use Perl's Test::Harness to run the tests 34and produce some nice looking output without the need to write an 35entire test harness. 36 37Tests can be written in C, C++ or Objective-C or perl (using 38Test::More in perl). 39 40 41WRITING TESTS 42============= 43 44To add a new test simply copy one of the existing ones and hack away. 45Any file with a main() function in it will be built into a test 46automatically by the top level Makefile (no configuration required). 47 48To use the testmore C "library" all you need to do is #include 49"testmore.h" in your test program. 50 51Then in your main function you must call: 52 53plan_tests(NUMTESTS) where NUMTESTS is the number of test cases you 54test program will run. After that you can start writing tests. 55There are a couple of macros to help you get started: 56 57The following are ways to run an actual test case (as in they count 58towards the NUMTESTS number above): 59 60ok(EXPR, NAME) 61 Evaluate EXPR if it's true the test passes if false it fails. 62 The second argument is a descriptive name of the test for debugging 63 purposes. 64 65is(EXPR, VALUE, NAME) 66 Evaluate EXPR if it's equal to VALUE the test passes otherwise 67 it fails. This is equivalent to ok(EXPR == VALUE, NAME) except 68 this produces nicer output in a failure case. 69 CAVEAT: Currently EXPR and VALUE must both be type int. 70 71isnt(EXPR, VALUE, NAME) 72 Opposite of is() above. 73 CAVEAT: Currently EXPR and VALUE must both be type int. 74 75cmp_ok(EXPR, OP, VALUE, NAME) 76 Succeeds if EXPR OP VALUE is true. Produces a diagnostic if not. 77 CAVEAT: Currently EXPR and VALUE must both be type int. 78 79ok_status(EXPR, NAME) 80 Evaluate EXPR, if it's 0 the test passes otherwise print a 81 diagnostic with the name and number of the error returned. 82 83is_status(EXPR, VALUE, NAME) 84 Evaluate EXPR, if the error returned equals VALUE the test 85 passes, otherwise print a diagnostic with the expected and 86 actual error returned. 87 88ok_unix(EXPR, NAME) 89 Evaluate EXPR, if it's >= 0 the test passes otherwise print a 90 diagnostic with the name and number of the errno. 91 92is_unix(EXPR, VALUE, NAME) 93 Evaluate EXPR, if the errno set by it equals VALUE the test 94 passes, otherwise print a diagnostic with the expected and 95 actual errno. 96 97Finally if you somehow can't express the success or failure of a 98test using the macros above you can use pass(NAME) or fail(NAME) 99explicitly. These are equivalent to ok(1, NAME) and ok(0, NAME) 100respectively. 101 102 103LEAKS 104===== 105 106If you want to check for leaks in your test you can #include 107"testleaks.h" in your program and call: 108 109ok_leaks(NAME) 110 Passes if there are no leaks in your program. 111 112is_leaks(VALUE, NAME) 113 Passes if there are exactly VALUE leaks in your program. Useful 114 if you are calling code that is known to leak and you can't fix 115 it. But you still want to make sure there are no new leaks in 116 your code. 117 118 119C++ 120=== 121 122For C++ programs you can #include "testcpp.h" which defines these 123additional macros: 124no_throw(EXPR, NAME) 125 Success if EXPR doesn't throw. 126 127does_throw(EXPR, NAME) 128 Success if EXPR does throw. 129 130is_throw(EXPR, CLASS, FUNC, VALUE, NAME) 131 Success if EXPR throws an exception of type CLASS and CLASS.FUNC == VALUE. 132 Example usage: 133 is_throw(CssmError::throwMe(42), CssmError, osStatus(), 42, "throwMe(42)"); 134 135 136TODO and SKIP 137============= 138 139Sometimes you write a test case that is known to fail (because you 140found a bug). Rather than commenting out that test case you should 141put it inside a TODO block. This will cause the test to run but 142the failure will not be reported as an error. When the test starts 143passing (presumably because someone fixed the bug) you can comment 144out the TODO block and leave the test in place. 145 146The syntax for doing this looks like so: 147 148 TODO: { 149 todo("<rdar://problem/4000000> ER: AAPL target: $4,000,000/share"); 150 151 cmp_ok(apple_stock_price(), >=, 4000000, "stock over 4M"); 152 } 153 154Sometimes you don't want to run a particular test case or set of 155test cases because something in the environment is missing or you 156are running on a different version of the OS than the test was 157designed for. To achieve this you can use a SKIP block. 158 159The syntax for a SKIP block looks like so: 160 161 SKIP: { 162 skip("only runs on Tiger and later", 4, os_version() >= os_tiger); 163 164 ok(tiger_test1(), "test1"); 165 ok(tiger_test2(), "test2"); 166 ok(tiger_test3(), "test3"); 167 ok(tiger_test4(), "test4"); 168 } 169 170How it works is like so: If the third argument to skip evaluates 171to false it breaks out of the SKIP block and reports N tests as 172being skipped (where N is the second argument to skip) The reason 173for the test being skipped is given as the first argument to skip. 174 175 176Utility Functions 177================= 178 179Anyone writing tests can add new utility functions. Currently there 180is a pair called tests_begin and tests_end. To get them 181#include "testenv.h". Calling them doesn't count as running a test 182case, unless you wrap them in an ok() macro. tests_begin creates 183a unique dir in /tmp and sets HOME in the environment to that dir. 184tests_end cleans up the mess that tests_begin made. 185 186When writing your own utility functions you will probably want to use 187the setup("task") macro so that any uses of ok() and other macros 188don't count as actual test cases run, but do report errors when they 189fail. Here is an example of how tests_end() does this: 190 191int 192tests_end(int result) 193{ 194 setup("tests_end"); 195 /* Restore previous cwd and remove scratch dir. */ 196 return (ok_unix(fchdir(current_dir), "fchdir") && 197 ok_unix(close(current_dir), "close") && 198 ok_unix(rmdir_recursive(scratch_dir), "rmdir_recursive")); 199} 200 201Setup cases all tests unil the end of the current funtion to not count 202against your test cases test count and they output nothing if they 203succeed. 204 205There is also a simple utility header called "testcssm.h" which 206currently defines cssm_attach and cssm_detach functions for loading 207and initializing cssm and loading a module. 208