Kenji Arai / mbed-os_TYBLE16

Important changes to repositories hosted on mbed.com

Mbed hosted mercurial repositories are deprecated and are due to be permanently deleted in July 2026.

To keep a copy of this software download the repository Zip archive or clone locally using Mercurial.

It is also possible to export all your personal repositories from the account settings page.

Dependents:   TYBLE16_simple_data_logger TYBLE16_MP3_Air

features/frameworks/utest/README.md@0:5b88d5760320, 2019-12-17 (annotated)

Committer:
kenjiArai
Date:
Tue Dec 17 23:23:45 2019 +0000
Revision:
0:5b88d5760320
mbed-os5 only for TYBLE16

Who changed what in which revision?

UserRevisionLine numberNew contents of line
kenjiArai 0:5b88d5760320 1 # utest: Asynchronous C++ Test Harness
kenjiArai 0:5b88d5760320 2
kenjiArai 0:5b88d5760320 3 This test harness allows you to execute a specified series of (asynchronous) C++ test cases with sensible default reporting and useful customization options.
kenjiArai 0:5b88d5760320 4
kenjiArai 0:5b88d5760320 5 Please note that this is a purposefully lean test harness, only dealing with test execution and providing default reporting handlers. It specifically does not support auto-discovery of test cases and does not provide you with test macros or other convenience functions.
kenjiArai 0:5b88d5760320 6 Instead, the macros in the [unity module](https://github.com/ARMmbed/mbed-os/tree/master/features/frameworks/unity) can be used for this purpose. However, you are not required to use these, and can use your own macros if you wish.
kenjiArai 0:5b88d5760320 7
kenjiArai 0:5b88d5760320 8 Furthermore, test failure recovery through the use of exceptions or `longjmp` is not supported; the test will either continue and ignore failures or die by busy-waiting.
kenjiArai 0:5b88d5760320 9
kenjiArai 0:5b88d5760320 10 ## Theory of Operation
kenjiArai 0:5b88d5760320 11
kenjiArai 0:5b88d5760320 12 A test specification contains a setup handler, several test cases and a teardown handler.
kenjiArai 0:5b88d5760320 13 Each test case contains a textual description, setup, teardown and failure handler as well as the actual test handler.
kenjiArai 0:5b88d5760320 14
kenjiArai 0:5b88d5760320 15 The order of handler execution is:
kenjiArai 0:5b88d5760320 16
kenjiArai 0:5b88d5760320 17 1. Test setup handler.
kenjiArai 0:5b88d5760320 18 1. For each test case:
kenjiArai 0:5b88d5760320 19 1. Test case setup handler.
kenjiArai 0:5b88d5760320 20 1. Test case execution handler.
kenjiArai 0:5b88d5760320 21 1. (wait for timeout or callback validation in case of an asynchronous test case.)
kenjiArai 0:5b88d5760320 22 1. (repeat test case execution handler if specified.)
kenjiArai 0:5b88d5760320 23 1. Test case teardown handler.
kenjiArai 0:5b88d5760320 24 1. Test teardown handler.
kenjiArai 0:5b88d5760320 25
kenjiArai 0:5b88d5760320 26 ## Example
kenjiArai 0:5b88d5760320 27
kenjiArai 0:5b88d5760320 28 The following example showcases a lot of functionality and proper integration with the [Greentea test tool](https://github.com/ARMmbed/mbed-os-tools/tree/master/packages/mbed-greentea), while making use of the [unity test macros](https://github.com/ARMmbed/mbed-os/tree/master/features/frameworks/unity):
kenjiArai 0:5b88d5760320 29
kenjiArai 0:5b88d5760320 30 ```cpp
kenjiArai 0:5b88d5760320 31 #include "mbed.h"
kenjiArai 0:5b88d5760320 32 #include "greentea-client/test_env.h"
kenjiArai 0:5b88d5760320 33 #include "utest/utest.h"
kenjiArai 0:5b88d5760320 34 #include "unity/unity.h"
kenjiArai 0:5b88d5760320 35
kenjiArai 0:5b88d5760320 36 using namespace utest::v1;
kenjiArai 0:5b88d5760320 37
kenjiArai 0:5b88d5760320 38 void test_simple() {
kenjiArai 0:5b88d5760320 39 TEST_ASSERT_EQUAL(0, 0);
kenjiArai 0:5b88d5760320 40 printf("Simple test called\n");
kenjiArai 0:5b88d5760320 41 }
kenjiArai 0:5b88d5760320 42
kenjiArai 0:5b88d5760320 43 status_t test_repeats_setup(const Case *const source, const size_t index_of_case) {
kenjiArai 0:5b88d5760320 44 // Call the default handler for proper reporting
kenjiArai 0:5b88d5760320 45 status_t status = greentea_case_setup_handler(source, index_of_case);
kenjiArai 0:5b88d5760320 46 printf("Setting up for '%s'\n", source->get_description());
kenjiArai 0:5b88d5760320 47 return status;
kenjiArai 0:5b88d5760320 48 }
kenjiArai 0:5b88d5760320 49
kenjiArai 0:5b88d5760320 50 control_t test_repeats(const size_t call_count) {
kenjiArai 0:5b88d5760320 51 printf("Called for the %u. time\n", call_count);
kenjiArai 0:5b88d5760320 52 TEST_ASSERT_NOT_EQUAL(3, call_count);
kenjiArai 0:5b88d5760320 53 // Specify how often this test is repeated ie. n total calls
kenjiArai 0:5b88d5760320 54 return (call_count < 2) ? CaseRepeatAll : CaseNext;
kenjiArai 0:5b88d5760320 55 }
kenjiArai 0:5b88d5760320 56
kenjiArai 0:5b88d5760320 57 void test_callback_validate() {
kenjiArai 0:5b88d5760320 58 // You may also use assertions here!
kenjiArai 0:5b88d5760320 59 TEST_ASSERT_EQUAL_PTR(0, 0);
kenjiArai 0:5b88d5760320 60 // Validate the callback
kenjiArai 0:5b88d5760320 61 Harness::validate_callback();
kenjiArai 0:5b88d5760320 62 }
kenjiArai 0:5b88d5760320 63
kenjiArai 0:5b88d5760320 64 control_t test_asynchronous() {
kenjiArai 0:5b88d5760320 65 TEST_ASSERT_TRUE_MESSAGE(true, "(true == false) o_O");
kenjiArai 0:5b88d5760320 66 // Set up a callback in the future. This may also be an interrupt!
kenjiArai 0:5b88d5760320 67 EventQueue *queue = mbed_event_queue();
kenjiArai 0:5b88d5760320 68 queue->call_in(100, test_callback_validate);
kenjiArai 0:5b88d5760320 69 // Set a 200ms timeout starting from now
kenjiArai 0:5b88d5760320 70 return CaseTimeout(200);
kenjiArai 0:5b88d5760320 71 }
kenjiArai 0:5b88d5760320 72
kenjiArai 0:5b88d5760320 73 control_t test_asynchronous_timeout(const size_t call_count) {
kenjiArai 0:5b88d5760320 74 TEST_ASSERT_TRUE_MESSAGE(true, "(true == false) o_O");
kenjiArai 0:5b88d5760320 75 // Set a 200ms timeout starting from now,
kenjiArai 0:5b88d5760320 76 // but automatically repeat only this handler on timeout.
kenjiArai 0:5b88d5760320 77 if (call_count >= 5) {
kenjiArai 0:5b88d5760320 78 // but after the 5th call, the callback finally gets validated
kenjiArai 0:5b88d5760320 79 EventQueue *queue = mbed_event_queue();
kenjiArai 0:5b88d5760320 80 queue->call_in(100, test_callback_validate);
kenjiArai 0:5b88d5760320 81 }
kenjiArai 0:5b88d5760320 82 return CaseRepeatHandlerOnTimeout(200);
kenjiArai 0:5b88d5760320 83 }
kenjiArai 0:5b88d5760320 84
kenjiArai 0:5b88d5760320 85 // Custom setup handler required for proper Greentea support
kenjiArai 0:5b88d5760320 86 status_t greentea_setup(const size_t number_of_cases) {
kenjiArai 0:5b88d5760320 87 GREENTEA_SETUP(20, "default_auto");
kenjiArai 0:5b88d5760320 88 // Call the default reporting function
kenjiArai 0:5b88d5760320 89 return greentea_test_setup_handler(number_of_cases);
kenjiArai 0:5b88d5760320 90 }
kenjiArai 0:5b88d5760320 91
kenjiArai 0:5b88d5760320 92 // Specify all your test cases here
kenjiArai 0:5b88d5760320 93 Case cases[] = {
kenjiArai 0:5b88d5760320 94 Case("Simple Test", test_simple),
kenjiArai 0:5b88d5760320 95 Case("Repeating Test", test_repeats_setup, test_repeats),
kenjiArai 0:5b88d5760320 96 Case("Asynchronous Test (200ms timeout)", test_asynchronous),
kenjiArai 0:5b88d5760320 97 Case("Asynchronous Timeout Repeat", test_asynchronous_timeout)
kenjiArai 0:5b88d5760320 98 };
kenjiArai 0:5b88d5760320 99
kenjiArai 0:5b88d5760320 100 // Declare your test specification with a custom setup handler
kenjiArai 0:5b88d5760320 101 Specification specification(greentea_setup, cases);
kenjiArai 0:5b88d5760320 102
kenjiArai 0:5b88d5760320 103 int main()
kenjiArai 0:5b88d5760320 104 { // Run the test specification
kenjiArai 0:5b88d5760320 105 Harness::run(specification);
kenjiArai 0:5b88d5760320 106 }
kenjiArai 0:5b88d5760320 107 ```
kenjiArai 0:5b88d5760320 108
kenjiArai 0:5b88d5760320 109 Running this test will output the following:
kenjiArai 0:5b88d5760320 110
kenjiArai 0:5b88d5760320 111 ```
kenjiArai 0:5b88d5760320 112 {{timeout;20}}
kenjiArai 0:5b88d5760320 113 {{host_test_name;default_auto}}
kenjiArai 0:5b88d5760320 114 {{description;utest greentea example}}
kenjiArai 0:5b88d5760320 115 {{test_id;MBED_OS}}
kenjiArai 0:5b88d5760320 116 {{start}}
kenjiArai 0:5b88d5760320 117 >>> Running 4 test cases...
kenjiArai 0:5b88d5760320 118
kenjiArai 0:5b88d5760320 119 >>> Running case #1: 'Simple Test'...
kenjiArai 0:5b88d5760320 120 Simple test called
kenjiArai 0:5b88d5760320 121 >>> 'Simple Test': 1 passed, 0 failed
kenjiArai 0:5b88d5760320 122
kenjiArai 0:5b88d5760320 123 >>> Running case #2: 'Repeating Test'...
kenjiArai 0:5b88d5760320 124 Setting up for 'Repeating Test'
kenjiArai 0:5b88d5760320 125 Called for the 1. time
kenjiArai 0:5b88d5760320 126 >>> 'Repeating Test': 1 passed, 0 failed
kenjiArai 0:5b88d5760320 127
kenjiArai 0:5b88d5760320 128 >>> Running case #2: 'Repeating Test'...
kenjiArai 0:5b88d5760320 129 Setting up for 'Repeating Test'
kenjiArai 0:5b88d5760320 130 Called for the 2. time
kenjiArai 0:5b88d5760320 131 >>> 'Repeating Test': 2 passed, 0 failed
kenjiArai 0:5b88d5760320 132
kenjiArai 0:5b88d5760320 133 >>> Running case #3: 'Asynchronous Test (200ms timeout)'...
kenjiArai 0:5b88d5760320 134 >>> 'Asynchronous Test (200ms timeout)': 1 passed, 0 failed
kenjiArai 0:5b88d5760320 135
kenjiArai 0:5b88d5760320 136 >>> Running case #4: 'Asynchronous Timeout Repeat'...
kenjiArai 0:5b88d5760320 137 >>> failure with reason 'Ignored: Timed Out'
kenjiArai 0:5b88d5760320 138 >>> failure with reason 'Ignored: Timed Out'
kenjiArai 0:5b88d5760320 139 >>> failure with reason 'Ignored: Timed Out'
kenjiArai 0:5b88d5760320 140 >>> failure with reason 'Ignored: Timed Out'
kenjiArai 0:5b88d5760320 141 >>> failure with reason 'Ignored: Timed Out'
kenjiArai 0:5b88d5760320 142 >>> 'Asynchronous Timeout Repeat': 1 passed, 0 failed
kenjiArai 0:5b88d5760320 143
kenjiArai 0:5b88d5760320 144 >>> Test cases: 4 passed, 0 failed
kenjiArai 0:5b88d5760320 145 {{success}}
kenjiArai 0:5b88d5760320 146 {{end}}
kenjiArai 0:5b88d5760320 147 ```
kenjiArai 0:5b88d5760320 148
kenjiArai 0:5b88d5760320 149 ## Detailed Description
kenjiArai 0:5b88d5760320 150
kenjiArai 0:5b88d5760320 151 ### Handlers
kenjiArai 0:5b88d5760320 152
kenjiArai 0:5b88d5760320 153 There are five handler types you can, but do not have to, override to customize operation.
kenjiArai 0:5b88d5760320 154 Please see the `utest/types.h` file for a detailed description.
kenjiArai 0:5b88d5760320 155
kenjiArai 0:5b88d5760320 156 1. `status_t test_setup_handler_t(const size_t number_of_cases)`: called before execution of any test case.
kenjiArai 0:5b88d5760320 157 1. `void test_teardown_handler_t(const size_t passed, const size_t failed, const failure_t failure)`: called after execution of all test cases, and if testing is aborted.
kenjiArai 0:5b88d5760320 158 1. `void test_failure_handler_t(const failure_t failure)`: called whenever a failure occurs anywhere in the specification.
kenjiArai 0:5b88d5760320 159 1. `status_t case_setup_handler_t(const Case *const source, const size_t index_of_case)`: called before execution of each test case.
kenjiArai 0:5b88d5760320 160 1. `status_t case_teardown_handler_t(const Case *const source, const size_t passed, const size_t failed, const failure_t reason)`: called after execution of each test case, and if testing is aborted.
kenjiArai 0:5b88d5760320 161 1. `status_t case_failure_handler_t(const Case *const source, const failure_t reason)`: called whenever a failure occurs during the execution of a test case.
kenjiArai 0:5b88d5760320 162
kenjiArai 0:5b88d5760320 163 All handlers are defaulted for integration with the [Greentea testing tool](https://github.com/ARMmbed/mbed-os-tools/tree/master/packages/mbed-greentea).
kenjiArai 0:5b88d5760320 164
kenjiArai 0:5b88d5760320 165 ### Test Case Handlers
kenjiArai 0:5b88d5760320 166
kenjiArai 0:5b88d5760320 167 There are three test case handlers:
kenjiArai 0:5b88d5760320 168
kenjiArai 0:5b88d5760320 169 1. `void case_handler_t(void)`: executes once, if the case setup succeeded.
kenjiArai 0:5b88d5760320 170 1. `control_t case_control_handler_t(void)`: executes (asynchronously) as many times as you specify, if the case setup succeeded.
kenjiArai 0:5b88d5760320 171 1. `control_t case_call_count_handler_t(const size_t call_count)`: executes (asynchronously) as many times as you specify, if the case setup succeeded.
kenjiArai 0:5b88d5760320 172
kenjiArai 0:5b88d5760320 173 To specify a test case you must wrap it into a `Case` class: `Case("mandatory description", case_handler)`. You may override the setup, teardown and failure handlers in this wrapper class as well.
kenjiArai 0:5b88d5760320 174 The `Case` constructor is overloaded to allow you a comfortable declaration of all your callbacks and the order of arguments is:
kenjiArai 0:5b88d5760320 175
kenjiArai 0:5b88d5760320 176 1. Description (required).
kenjiArai 0:5b88d5760320 177 1. Setup handler (optional).
kenjiArai 0:5b88d5760320 178 1. Test case handler (required).
kenjiArai 0:5b88d5760320 179 1. Teardown handler (optional).
kenjiArai 0:5b88d5760320 180 1. Failure handler (optional).
kenjiArai 0:5b88d5760320 181
kenjiArai 0:5b88d5760320 182 #### Test Case Attributes
kenjiArai 0:5b88d5760320 183
kenjiArai 0:5b88d5760320 184 You can modify test case behavior by returning `control_t` modifiers:
kenjiArai 0:5b88d5760320 185
kenjiArai 0:5b88d5760320 186 - `CaseNext`: never repeats and immediately moves to next test case
kenjiArai 0:5b88d5760320 187 - `CaseNoRepeat`: never repeats.
kenjiArai 0:5b88d5760320 188 - `CaseRepeatAll`: repeats test case **with** setup and teardown handlers.
kenjiArai 0:5b88d5760320 189 - `CaseRepeatHandler`: repeats test case **without** set and teardown handlers.
kenjiArai 0:5b88d5760320 190 - `CaseNoTimeout`: immediately moves to next test case.
kenjiArai 0:5b88d5760320 191 - `CaseAwait`: waits indefinitely for callback validation (*use with caution*).
kenjiArai 0:5b88d5760320 192 - `CaseTimeout(uint32_t ms)`: waits for callback validation for `ms` milliseconds, times out after that (fails with `REASON_TIMEOUT`).
kenjiArai 0:5b88d5760320 193 - `CaseRepeatAllOnTimeout(uint32_t ms)`: waits for callback validation for `ms` milliseconds, repeats test case **with** setup and teardown handlers on time out.
kenjiArai 0:5b88d5760320 194 - `CaseRepeatHandlerOnTimeout(uint32_t ms)`: waits for callback validation for `ms` milliseconds, repeats test case **without** setup and teardown handlers on time out.
kenjiArai 0:5b88d5760320 195
kenjiArai 0:5b88d5760320 196 Returning `CaseRepeatAll` from your test case handler tells the test harness to repeat the test handler. You can use the `call_count` (starts counting at 1) to decide when to stop.
kenjiArai 0:5b88d5760320 197 By default the setup and teardown handlers are called on every repeated test cases, however, you may only repeat the case handler by returning `CaseRepeatHandler`. To stop the harness from repeating the test case, return `CaseNext`.
kenjiArai 0:5b88d5760320 198
kenjiArai 0:5b88d5760320 199 For asynchronous test cases, you must return a `CaseTimeout(uint32_t ms)`.
kenjiArai 0:5b88d5760320 200 If you want to automatically repeat the test case on a timeout, use `CaseRepeatAllOnTimeout(uint32_t ms)` and `CaseRepeatHandlerOnTimeout(uint32_t ms)`.
kenjiArai 0:5b88d5760320 201
kenjiArai 0:5b88d5760320 202 To validate your callback, you must call `Harness::validate_callback()` in your asynchronous callback before the timeout fires.
kenjiArai 0:5b88d5760320 203 This will schedule the execution of the next test case.
kenjiArai 0:5b88d5760320 204
kenjiArai 0:5b88d5760320 205 For repeating asynchronous cases, you can "add" both attributes together: `CaseTimeout(200) + CaseRepeatAll` will wait for 200ms for the callback validation and then repeat the test case. See the section on arbitration logic for more details.
kenjiArai 0:5b88d5760320 206
kenjiArai 0:5b88d5760320 207 Note that you can also add attributes during callback validation, however, only repeat attributes are considered. This allows you to return `CaseTimeout(500)` to wait up to 500ms for the callback validation and delegate the decision to repeat to the time the callback occurs: `Harness::validate_callback(CaseRepeatHandler)`.
kenjiArai 0:5b88d5760320 208
kenjiArai 0:5b88d5760320 209 Keep in mind that you can only validate a callback once. If you need to wait for several callbacks, you need to write your own helper function that validates the expected callback only when all your custom callbacks arrive.
kenjiArai 0:5b88d5760320 210 This custom functionality is purposefully not part of this test harness, you can achieve it externally with additional code.
kenjiArai 0:5b88d5760320 211
kenjiArai 0:5b88d5760320 212 ### Failure Handlers
kenjiArai 0:5b88d5760320 213
kenjiArai 0:5b88d5760320 214 A failure may occur during any phase of the test. The appropriate failure handler is then called with `failure_t`, which contains the failure reason and location.
kenjiArai 0:5b88d5760320 215
kenjiArai 0:5b88d5760320 216 The failure reasons are:
kenjiArai 0:5b88d5760320 217
kenjiArai 0:5b88d5760320 218 - `REASON_NONE`: No failure occurred
kenjiArai 0:5b88d5760320 219 - `REASON_UNKNOWN`: An unknown failure occurred
kenjiArai 0:5b88d5760320 220 - `REASON_CASES`: A failure occurred in at least one test case
kenjiArai 0:5b88d5760320 221 - `REASON_EMPTY_CASE`: The test case contains only empty handlers
kenjiArai 0:5b88d5760320 222 - `REASON_TIMEOUT`: An expected asynchronous call timed out
kenjiArai 0:5b88d5760320 223 - `REASON_ASSERTION`: An assertion failed
kenjiArai 0:5b88d5760320 224 - `REASON_TEST_SETUP`: Test setup failed
kenjiArai 0:5b88d5760320 225 - `REASON_TEST_TEARDOWN`: Test teardown failed
kenjiArai 0:5b88d5760320 226 - `REASON_CASE_SETUP`: Case setup failed
kenjiArai 0:5b88d5760320 227 - `REASON_CASE_HANDLER`: Case handler failed
kenjiArai 0:5b88d5760320 228 - `REASON_CASE_TEARDOWN`: Case teardown failed
kenjiArai 0:5b88d5760320 229 - `REASON_CASE_INDEX`: Case index returned from test setup or case teardown handler is invalid
kenjiArai 0:5b88d5760320 230 - `REASON_SCHEDULER`: Underlying scheduler is not asynchronous
kenjiArai 0:5b88d5760320 231
kenjiArai 0:5b88d5760320 232 The failure locations are:
kenjiArai 0:5b88d5760320 233
kenjiArai 0:5b88d5760320 234 - `LOCATION_NONE`: No location information
kenjiArai 0:5b88d5760320 235 - `LOCATION_UNKNOWN`: A failure occurred in an unknown location
kenjiArai 0:5b88d5760320 236 - `LOCATION_TEST_SETUP`: A failure occurred in the test setup
kenjiArai 0:5b88d5760320 237 - `LOCATION_TEST_TEARDOWN`: A failure occurred in the test teardown
kenjiArai 0:5b88d5760320 238 - `LOCATION_CASE_SETUP`: A failure occurred in the case setup
kenjiArai 0:5b88d5760320 239 - `LOCATION_CASE_HANDLER`: A failure occurred in the case handler
kenjiArai 0:5b88d5760320 240 - `LOCATION_CASE_TEARDOWN`: A failure occurred in the case teardown
kenjiArai 0:5b88d5760320 241
kenjiArai 0:5b88d5760320 242 If the setup or teardown handlers fail, they may return a `STATUS_ABORT` code, which will call the failure handler with the appropriate failure reason (`REASON_CASE_{SETUP|TEARDOWN}`) and failure location (`LOCATION_CASE_{SETUP|TEARDOWN}`).
kenjiArai 0:5b88d5760320 243 If the setup handler fails, the test case is never executed. Instead, the teardown handler is called in an attempt to salvage the situation.
kenjiArai 0:5b88d5760320 244 Please note that if a teardown handler fails, the system can be considered too unstable to continue testing.
kenjiArai 0:5b88d5760320 245
kenjiArai 0:5b88d5760320 246 You may also raise a failure manually by calling `Harness::raise_failure(failure_reason_t reason)`. In fact, this is how you can integrate assertion failures from custom test macros, as done with the unity macros, which raise a failure with the `REASON_ASSERTION` reason.
kenjiArai 0:5b88d5760320 247
kenjiArai 0:5b88d5760320 248 When waiting for an asynchronous callback, if the timeout fires, `REASON_TIMEOUT` is raised.
kenjiArai 0:5b88d5760320 249
kenjiArai 0:5b88d5760320 250 The failure handler decides whether to continue or abort testing by returning `STATUS_CONTINUE` or `STATUS_ABORT` respectively.
kenjiArai 0:5b88d5760320 251 You can also ignore any raised failure by returning `STATUS_IGNORE` and the harness will then not count this failure.
kenjiArai 0:5b88d5760320 252 In case of an abort, the test harness dies by busy waiting in a forever loop.
kenjiArai 0:5b88d5760320 253 This is needed because we cannot unwind the stack without exception support, and the asynchronous nature of the test harness breaks with using `longjmp`s.
kenjiArai 0:5b88d5760320 254
kenjiArai 0:5b88d5760320 255 Note that when `REASON_IGNORE` is `OR`ed into the failure reason, the failure handler is expected to return `STATUS_IGNORE`.
kenjiArai 0:5b88d5760320 256 This is done automatically for test cases repeating after a timeout, and the default failure handlers also report this failure, but tell the harness to ignore it.
kenjiArai 0:5b88d5760320 257 Furthermore, the unity macros may decide to ignore assertion failures as well, in which case the assertion is ignored intentionally.
kenjiArai 0:5b88d5760320 258
kenjiArai 0:5b88d5760320 259 ### Default Handlers
kenjiArai 0:5b88d5760320 260
kenjiArai 0:5b88d5760320 261 Three sets of default handlers with different behaviors are provided for your convenience:
kenjiArai 0:5b88d5760320 262
kenjiArai 0:5b88d5760320 263 1. `greentea_abort_handlers` (default): Greentea-style reporting, aborts on the first failure, but requires custom test setup handler.
kenjiArai 0:5b88d5760320 264 1. `greentea_continue_handlers`: Greentea-style reporting, always continues testing, but requires custom test setup handler.
kenjiArai 0:5b88d5760320 265 1. `verbose_continue_handlers`: always continues testing and reporting, except when a teardown failed.
kenjiArai 0:5b88d5760320 266 1. `selftest_handlers`: Greentea-style reporting, but aborts on the first assertion failure raised. This allows the use of unity macros for self testing without recursive failure handler calls.
kenjiArai 0:5b88d5760320 267
kenjiArai 0:5b88d5760320 268 These default handlers are called when you have not overridden a custom handler, and they only contain reporting functionality and do not modify global state.
kenjiArai 0:5b88d5760320 269
kenjiArai 0:5b88d5760320 270 You can specify which default handlers you want to use when wrapping your test cases in the `Specification` class:
kenjiArai 0:5b88d5760320 271
kenjiArai 0:5b88d5760320 272 ```cpp
kenjiArai 0:5b88d5760320 273 // Declare your test specification with a custom setup handler
kenjiArai 0:5b88d5760320 274 // and set the default handlers to the predefined “greentea continue” behavior
kenjiArai 0:5b88d5760320 275 Specification specification(greentea_setup, cases, greentea_continue_handlers);
kenjiArai 0:5b88d5760320 276 ```
kenjiArai 0:5b88d5760320 277
kenjiArai 0:5b88d5760320 278 ### Custom Handlers
kenjiArai 0:5b88d5760320 279
kenjiArai 0:5b88d5760320 280 You may override any of the default handlers with your own custom handler.
kenjiArai 0:5b88d5760320 281
kenjiArai 0:5b88d5760320 282 To ignore a handler completely and not call a custom or default handler, you may use the `ignore_handler` hint.
kenjiArai 0:5b88d5760320 283 To explicitly invoke the default handler, use the `default_handler` hint.
kenjiArai 0:5b88d5760320 284
kenjiArai 0:5b88d5760320 285 To use your own custom handler, provide a function with the correct signature for the handler that you want to customize and provide it in your test case wrapper or specification wrapper.
kenjiArai 0:5b88d5760320 286 To turn a `failure_t` into a meaningful string use the `stringify(failure_t)` method.
kenjiArai 0:5b88d5760320 287
kenjiArai 0:5b88d5760320 288 **We strongly recommend that you call the predefined `greentea_*` handlers inside your custom callback, as they report the current condition in a properly formatted fashion.**
kenjiArai 0:5b88d5760320 289 By calling these handlers inside your custom callback your unit test does not need to be modified if the test logging needs to be changed in the future.
kenjiArai 0:5b88d5760320 290
kenjiArai 0:5b88d5760320 291 For the `Specification` class the order of arguments is:
kenjiArai 0:5b88d5760320 292
kenjiArai 0:5b88d5760320 293 1. Test setup handler (optional).
kenjiArai 0:5b88d5760320 294 1. Array of test cases (required).
kenjiArai 0:5b88d5760320 295 1. Test teardown handler (optional).
kenjiArai 0:5b88d5760320 296 1. Test failure handler (optional).
kenjiArai 0:5b88d5760320 297 1. Default handlers (optional).
kenjiArai 0:5b88d5760320 298
kenjiArai 0:5b88d5760320 299 ### Test Case Attribute Arbitration
kenjiArai 0:5b88d5760320 300
kenjiArai 0:5b88d5760320 301 When adding conflicting modifiers together
kenjiArai 0:5b88d5760320 302
kenjiArai 0:5b88d5760320 303 - the more restrictive (=shorter) timeout is chosen, but `CaseNoTimeout` always wins arbitration: `CaseNoTimeout` > `CaseTimeout(100)` > `CaseTimeout(200)` > `CaseAwait`.
kenjiArai 0:5b88d5760320 304 - the more invasive repeat method is chosen, but `CaseNoRepeat` always wins arbitration: `CaseNoRepeat` > `CaseRepeatAll`/`CaseRepeatAllOnTimeout(ms)` > `CaseRepeatHandler`/`CaseRepeatHandlerOnTimeout(ms)`.
kenjiArai 0:5b88d5760320 305 - `CaseNext` always wins arbitration.
kenjiArai 0:5b88d5760320 306
kenjiArai 0:5b88d5760320 307 The following table shows this arbitration logic in detail:
kenjiArai 0:5b88d5760320 308
kenjiArai 0:5b88d5760320 309 | + | `CaseNext` | `CaseNoRepeat` | `CaseRepeatAll` | `CaseRepeatHandler` | `CaseNoTimeout` | `CaseAwait` | `CaseTimeout(aa)` | `CaseRepeatAllOnTimeout(aa)` | `CaseRepeatHandlerOnTimeout(aa)`
kenjiArai 0:5b88d5760320 310 |:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|
kenjiArai 0:5b88d5760320 311 | `CaseNext` | no repeat &<br> no timeout
kenjiArai 0:5b88d5760320 312 | `CaseNoRepeat` | no repeat &<br> no timeout | no repeat
kenjiArai 0:5b88d5760320 313 | `CaseRepeatAll` | no repeat &<br> no timeout | no repeat | repeat all
kenjiArai 0:5b88d5760320 314 | `CaseRepeatHandler` | no repeat &<br> no timeout | no repeat | repeat all | repeat handler
kenjiArai 0:5b88d5760320 315 | `CaseNoTimeout` | no repeat &<br> no timeout | no repeat &<br> no timeout | repeat all &<br> no timeout | repeat handler &<br> no timeout | no timeout
kenjiArai 0:5b88d5760320 316 | `CaseAwait` | no repeat &<br> no timeout | no repeat &<br> infinite timeout | repeat all &<br> infinite timeout | repeat handler &<br> infinite timeout | no timeout | infinite timeout
kenjiArai 0:5b88d5760320 317 | `CaseTimeout(bb)` | no repeat &<br> no timeout | no repeat &<br> `bb`ms timeout | repeat all &<br> `bb`ms timeout | repeat handler &<br> `bb`ms timeout | no timeout | `bb`ms timeout | `min(aa,bb)`ms timeout
kenjiArai 0:5b88d5760320 318 | `CaseRepeatAllOnTimeout(bb)` | no repeat &<br> no timeout | no repeat &<br> `bb`ms timeout | repeat all on validate & repeat all on `bb`ms timeout | repeat all on validate & repeat all on `bb`ms timeout | repeat all & no timeout | repeat all on `bb`ms timeout | repeat all on `min(aa,bb)`ms timeout | repeat all on `min(aa,bb)`ms timeout |
kenjiArai 0:5b88d5760320 319 | `CaseRepeatHandlerOnTimeout(bb)` | no repeat &<br> no timeout | no repeat &<br> `bb`ms timeout | repeat all on validate & repeat all on `bb`ms timeout | repeat handler on validate & repeat handler on `bb`ms timeout | repeat handler & no timeout | repeat handler on `bb`ms timeout | repeat handler on `min(aa,bb)`ms timeout | repeat all on `min(aa,bb)`ms timeout | repeat handler on `min(aa,bb)`ms timeout
kenjiArai 0:5b88d5760320 320
kenjiArai 0:5b88d5760320 321 ### Atomicity
kenjiArai 0:5b88d5760320 322
kenjiArai 0:5b88d5760320 323 All handlers execute with interrupts enabled, **except the case failure handler!**.
kenjiArai 0:5b88d5760320 324 This means you can write test cases that poll for interrupts to be completed inside any handler, except the failure handler.
kenjiArai 0:5b88d5760320 325
kenjiArai 0:5b88d5760320 326 If you setup an interrupt that validates its callback using `Harness::validate_callback()` inside a test case and it fires before the test case completed, the validation will be buffered.
kenjiArai 0:5b88d5760320 327 If the test case then returns a timeout value, but the callback is already validated, the test harness just continues normally.
kenjiArai 0:5b88d5760320 328
kenjiArai 0:5b88d5760320 329 ### Custom Scheduler
kenjiArai 0:5b88d5760320 330
kenjiArai 0:5b88d5760320 331 By default, a Timeout object is used for scheduling the harness operations.
kenjiArai 0:5b88d5760320 332 In case this is not available you can provide your own custom scheduler implementation and make the harness use it with the `Harness::set_scheduler(your_custom_implementation)` function.
kenjiArai 0:5b88d5760320 333
kenjiArai 0:5b88d5760320 334 The scheduler requirements are very simple: Execute a `void(void)` function in you main loop (with a delay of *N* ms). Only one function is scheduled by the harness *at any given time*.
kenjiArai 0:5b88d5760320 335 Note that you do not need to implement the delay functionality, if your tests do not require timeouts. You will still be able to use repeating test cases, but an error is thrown if your tests attempt to use a timeout, when your underlying scheduler does not support it.
kenjiArai 0:5b88d5760320 336
kenjiArai 0:5b88d5760320 337 There are two functions you need to implement:
kenjiArai 0:5b88d5760320 338
kenjiArai 0:5b88d5760320 339 - `void* post_callback(const utest_v1_harness_callback_t callback, const uint32_t delay_ms)`: schedules a `void(void)` callback function in *N* ms.
kenjiArai 0:5b88d5760320 340 - `int32_t cancel_callback_t(void *handle)`: cancels an asynchronous callback.
kenjiArai 0:5b88d5760320 341
kenjiArai 0:5b88d5760320 342 Please see [their doxygen documentation for implementation details](utest/scheduler.h).
kenjiArai 0:5b88d5760320 343
kenjiArai 0:5b88d5760320 344 ### Example Synchronous Scheduler
kenjiArai 0:5b88d5760320 345
kenjiArai 0:5b88d5760320 346 Here is the most [basic scheduler implementation without any asynchronous support](test/minimal_scheduler/main.cpp). Note that this does not require any hardware support at all, but you cannot use timeouts in your test cases!
kenjiArai 0:5b88d5760320 347 ```cpp
kenjiArai 0:5b88d5760320 348 volatile utest_v1_harness_callback_t minimal_callback;
kenjiArai 0:5b88d5760320 349
kenjiArai 0:5b88d5760320 350 static void* utest_minimal_post(const utest_v1_harness_callback_t callback, const uint32_t delay_ms) {
kenjiArai 0:5b88d5760320 351 minimal_callback = callback;
kenjiArai 0:5b88d5760320 352 // this scheduler does not support asynchronous callbacks
kenjiArai 0:5b88d5760320 353 return (delay_ms ? NULL : (void*)1);
kenjiArai 0:5b88d5760320 354 }
kenjiArai 0:5b88d5760320 355 static int32_t utest_minimal_cancel(void*) {
kenjiArai 0:5b88d5760320 356 return -1; // canceling not supported either
kenjiArai 0:5b88d5760320 357 }
kenjiArai 0:5b88d5760320 358 static const utest_v1_scheduler_t utest_minimal_scheduler = {utest_minimal_post, utest_minimal_cancel};
kenjiArai 0:5b88d5760320 359
kenjiArai 0:5b88d5760320 360 // [...] Add your test cases and specification here.
kenjiArai 0:5b88d5760320 361
kenjiArai 0:5b88d5760320 362 void main() // or whatever your custom entry point is
kenjiArai 0:5b88d5760320 363 {
kenjiArai 0:5b88d5760320 364 // You MUST set the custom scheduler before running the specification.
kenjiArai 0:5b88d5760320 365 Harness::set_scheduler(utest_minimal_scheduler);
kenjiArai 0:5b88d5760320 366 Harness::run(specification);
kenjiArai 0:5b88d5760320 367
kenjiArai 0:5b88d5760320 368 while(1) {
kenjiArai 0:5b88d5760320 369 if (minimal_callback) {
kenjiArai 0:5b88d5760320 370 // copy the callback and reset the shared memory
kenjiArai 0:5b88d5760320 371 utest_v1_harness_callback_t callback = minimal_callback;
kenjiArai 0:5b88d5760320 372 minimal_callback = NULL;
kenjiArai 0:5b88d5760320 373 callback(); // execute the copied callback
kenjiArai 0:5b88d5760320 374 }
kenjiArai 0:5b88d5760320 375 }
kenjiArai 0:5b88d5760320 376 }
kenjiArai 0:5b88d5760320 377 ```
kenjiArai 0:5b88d5760320 378
kenjiArai 0:5b88d5760320 379 ### Example Asynchronous Scheduler
kenjiArai 0:5b88d5760320 380
kenjiArai 0:5b88d5760320 381 Here is the a [complete scheduler implementation with any asynchronous support](test/minimal_scheduler_async/main.cpp). Note that this does require at least a hardware timer, in this case we have used `mbed-hal/us_ticker`! Note that you must not execute the callback in the timer interrupt context, but in the main loop context!
kenjiArai 0:5b88d5760320 382 ```cpp
kenjiArai 0:5b88d5760320 383 volatile utest_v1_harness_callback_t minimal_callback;
kenjiArai 0:5b88d5760320 384 volatile utest_v1_harness_callback_t ticker_callback;
kenjiArai 0:5b88d5760320 385 const ticker_data_t *ticker_data;
kenjiArai 0:5b88d5760320 386 ticker_event_t ticker_event;
kenjiArai 0:5b88d5760320 387
kenjiArai 0:5b88d5760320 388 static void ticker_handler(uint32_t) {
kenjiArai 0:5b88d5760320 389 minimal_callback = ticker_callback; // interrupt context!
kenjiArai 0:5b88d5760320 390 }
kenjiArai 0:5b88d5760320 391 static void* utest_minimal_post(const utest_v1_harness_callback_t callback, const uint32_t delay_ms) {
kenjiArai 0:5b88d5760320 392 if (delay_ms) {
kenjiArai 0:5b88d5760320 393 ticker_callback = callback;
kenjiArai 0:5b88d5760320 394 ticker_insert_event(ticker_data, &ticker_event, ticker_read(ticker_data) + delay_ms * 1000, 0);
kenjiArai 0:5b88d5760320 395 }
kenjiArai 0:5b88d5760320 396 else minimal_callback = callback;
kenjiArai 0:5b88d5760320 397 return (void*)1;
kenjiArai 0:5b88d5760320 398 }
kenjiArai 0:5b88d5760320 399 static int32_t utest_minimal_cancel(void*) {
kenjiArai 0:5b88d5760320 400 ticker_remove_event(ticker_data, &ticker_event);
kenjiArai 0:5b88d5760320 401 return 0; // canceling is supported
kenjiArai 0:5b88d5760320 402 }
kenjiArai 0:5b88d5760320 403 static const utest_v1_scheduler_t utest_minimal_scheduler = {utest_minimal_post, utest_minimal_cancel};
kenjiArai 0:5b88d5760320 404
kenjiArai 0:5b88d5760320 405 // [...] Add your test cases and specification here.
kenjiArai 0:5b88d5760320 406
kenjiArai 0:5b88d5760320 407 void main() // or whatever your custom entry point is
kenjiArai 0:5b88d5760320 408 {
kenjiArai 0:5b88d5760320 409 ticker_data = get_us_ticker_data(); // initialize the ticker data.
kenjiArai 0:5b88d5760320 410 ticker_set_handler(ticker_data, ticker_handler);
kenjiArai 0:5b88d5760320 411 // You MUST set the custom scheduler before running the specification.
kenjiArai 0:5b88d5760320 412 Harness::set_scheduler(utest_minimal_scheduler);
kenjiArai 0:5b88d5760320 413 Harness::run(specification);
kenjiArai 0:5b88d5760320 414
kenjiArai 0:5b88d5760320 415 while(1) {
kenjiArai 0:5b88d5760320 416 if (minimal_callback) {
kenjiArai 0:5b88d5760320 417 // copy the callback and reset the shared memory
kenjiArai 0:5b88d5760320 418 utest_v1_harness_callback_t callback = minimal_callback;
kenjiArai 0:5b88d5760320 419 minimal_callback = NULL;
kenjiArai 0:5b88d5760320 420 callback(); // execute the copied callback
kenjiArai 0:5b88d5760320 421 }
kenjiArai 0:5b88d5760320 422 }
kenjiArai 0:5b88d5760320 423 }
kenjiArai 0:5b88d5760320 424 ```