erkin yucel / mbed-os

mbed os with nrf51 internal bandgap enabled to read battery level

Dependents:   BLE_file_test BLE_Blink ExternalEncoder

docs/TESTING.md@0:f269e3021894, 2016-10-23 (annotated)

Committer:
elessair
Date:
Sun Oct 23 15:10:02 2016 +0000
Revision:
0:f269e3021894
Initial commit

Who changed what in which revision?

UserRevisionLine numberNew contents of line
elessair 0:f269e3021894 1 # Mbed SDK automated test suite
elessair 0:f269e3021894 2 ## Introduction
elessair 0:f269e3021894 3
elessair 0:f269e3021894 4 Test suit allows users to run locally on their machines Mbed SDK’s tests included in Mbed SDK repository. It also allows users to create their own tests and for example add new tests to test set as they progress with their project. If test is generic enough it could be included into official Mbed SDK test pool just do it via normal pull request!
elessair 0:f269e3021894 5
elessair 0:f269e3021894 6 Each test is supervised by python script called “host test” which will at least Test suite is using build script API to compile and build test source together with required by test libraries like CMSIS, Mbed, Ethernet, USB etc.
elessair 0:f269e3021894 7
elessair 0:f269e3021894 8 ## What is host test?
elessair 0:f269e3021894 9 Test suite supports test supervisor concept. This concept is realized by separate Python script called ```host test```. Host tests can be found in ```mbed/tools/host_tests/``` directory. Note: In newer mbed versions (mbed OS) host tests will be separate library.
elessair 0:f269e3021894 10
elessair 0:f269e3021894 11 Host test script is executed in parallel with test runner to monitor test execution. Basic host test just monitors device's default serial port for test results returned by test runner. Simple tests will print test result on serial port. In other cases host tests can for example judge by test results returned by test runner if test passed or failed. It all depends on test itself.
elessair 0:f269e3021894 12
elessair 0:f269e3021894 13 In some cases host test can be TCP server echoing packets from test runner and judging packet loss. In other cases it can just check if values returned from accelerometer are actually valid (sane).
elessair 0:f269e3021894 14
elessair 0:f269e3021894 15 ## Test suite core: singletest.py script
elessair 0:f269e3021894 16
elessair 0:f269e3021894 17 ```singletest.py``` script located in ```mbed/tools/``` is a test suite script which allows users to compile, build tests and test runners (also supports CppUTest unit test library). Script also is responsible for test execution on devices selected by configuration files.
elessair 0:f269e3021894 18
elessair 0:f269e3021894 19 ### Parameters of singletest.py
elessair 0:f269e3021894 20
elessair 0:f269e3021894 21 Test execution script ```singletest.py``` is a fairly powerful tool to run tests for mbed SDK platform. It is flexible and allows users to configure test execution process and define which mbed platforms will be tested.
elessair 0:f269e3021894 22
elessair 0:f269e3021894 23 By specifying external configuration files (in JSON format) you can gain flexibility and prepare many different test scenarios. Just pass configuration file names to your script and run it.
elessair 0:f269e3021894 24
elessair 0:f269e3021894 25 #### MUTs Specification
elessair 0:f269e3021894 26 You can easily configure your MUTs (Mbed Under Test) by creating configuration file with MUTs description.
elessair 0:f269e3021894 27 Note: This configuration file must be in [JSON format](http://www.w3schools.com/json/).
elessair 0:f269e3021894 28 Note: Unfortunately JSON format is not allowing you to have comments inside JSON code.
elessair 0:f269e3021894 29
elessair 0:f269e3021894 30 Let’s see some example and let's try to configure small "test farm" with three devices connected to your host computer. In this example no peripherals (like SD card or EEPROM) are connected to our Mbed boards. We will use three platforms in this example:
elessair 0:f269e3021894 31 * [NXP LPC1768](https://mbed.org/platforms/mbed-LPC1768) board.
elessair 0:f269e3021894 32 * \[Freescale KL25Z](https://mbed.org/platforms/KL25Z) board and
elessair 0:f269e3021894 33 * [STMicro Nucleo F103RB](https://mbed.org/platforms/ST-Nucleo-F103RB) board.
elessair 0:f269e3021894 34 After connecting boards to our host machine (PC) we can check which serial ports and disks they occupy. For our example let's assume that:
elessair 0:f269e3021894 35 * ```LPC1768``` serial port is on ```COM4``` and disk drive is ```J:```.
elessair 0:f269e3021894 36 * ```KL25Z``` serial port is on ```COM39``` and disk drive is ```E:```.
elessair 0:f269e3021894 37 * ```NUCLEO_F103RB``` serial port is on ```COM11``` and disk drive is ```I:```.
elessair 0:f269e3021894 38 If you are working under Linux your port and disk could look like /dev/ttyACM5 and /media/usb5.
elessair 0:f269e3021894 39
elessair 0:f269e3021894 40 This information is needed to create ```muts_all.json``` configuration file. You can create it in ```mbed/tools/``` directory:
elessair 0:f269e3021894 41 ```
elessair 0:f269e3021894 42 $ touch muts_all.json
elessair 0:f269e3021894 43 ```
elessair 0:f269e3021894 44
elessair 0:f269e3021894 45 Its name will be passed to ```singletest.py``` script after ```-M``` (MUTs specification) switch. Let’s see how this file's content would look like in our example below:
elessair 0:f269e3021894 46 ```json
elessair 0:f269e3021894 47 {
elessair 0:f269e3021894 48 "1" : {"mcu": "LPC1768",
elessair 0:f269e3021894 49 "port":"COM4",
elessair 0:f269e3021894 50 "disk":"J:\\",
elessair 0:f269e3021894 51 "peripherals": []
elessair 0:f269e3021894 52 },
elessair 0:f269e3021894 53
elessair 0:f269e3021894 54 "2" : {"mcu": "KL25Z",
elessair 0:f269e3021894 55 "port":"COM39",
elessair 0:f269e3021894 56 "disk":"E:\\",
elessair 0:f269e3021894 57 "peripherals": []
elessair 0:f269e3021894 58 },
elessair 0:f269e3021894 59
elessair 0:f269e3021894 60 "3" : {"mcu": "NUCLEO_F103RB",
elessair 0:f269e3021894 61 "port":"COM11",
elessair 0:f269e3021894 62 "disk":"I:\\",
elessair 0:f269e3021894 63 "peripherals": []
elessair 0:f269e3021894 64 }
elessair 0:f269e3021894 65 }
elessair 0:f269e3021894 66 ```
elessair 0:f269e3021894 67
elessair 0:f269e3021894 68 Note: We will leave field ```peripherals``` empty for the sake of this example. We will explain it later. All you need to do now is to properly fill fields ```mcu```, ```port``` and ```disk```.
elessair 0:f269e3021894 69
elessair 0:f269e3021894 70 Note: Please make sure files muts_all.json and test_spec.json are in tools/ directory. We will assume in this example they are.
elessair 0:f269e3021894 71 Where to find ```mcu``` names? You can use option ```-S``` of ```build.py``` script (in ```mbed/tools/``` directory) to check all supported off-line MCUs names.
elessair 0:f269e3021894 72
elessair 0:f269e3021894 73 Note: If you update mbed device firmware or even disconnect / reconnect mbed device you may find that serial port / disk configuration changed. You need to update configuration file accordingly or you will face connection problems and obviously tests will not run.
elessair 0:f269e3021894 74
elessair 0:f269e3021894 75 #### Peripherals testing
elessair 0:f269e3021894 76 When using MUTs configuration file (switch ```-M```) you can define in MUTs JSON file peripherals connected to your device:
elessair 0:f269e3021894 77 ```json
elessair 0:f269e3021894 78 {
elessair 0:f269e3021894 79 "1" : {"mcu" : "KL25Z",
elessair 0:f269e3021894 80 "port" : "COM39",
elessair 0:f269e3021894 81 "disk" : "E:\\",
elessair 0:f269e3021894 82 "peripherals" : ["SD", "24LC256"]}
elessair 0:f269e3021894 83 }
elessair 0:f269e3021894 84 ```
elessair 0:f269e3021894 85 You can force test suite to run only common tests (switch ```-C```) or only peripheral tests (switch ```-P```).
elessair 0:f269e3021894 86 ```
elessair 0:f269e3021894 87 $ python singletest.py -i test_spec.json -M muts_all.json -C
elessair 0:f269e3021894 88 ```
elessair 0:f269e3021894 89 will not include tests for SD card and EEPROM 24LC256.
elessair 0:f269e3021894 90 ```
elessair 0:f269e3021894 91 $ python singletest.py -i test_spec.json -M muts_all.json -P
elessair 0:f269e3021894 92 ```
elessair 0:f269e3021894 93 will only run tests bind to SD card and EEPROM 24LC256.
elessair 0:f269e3021894 94
elessair 0:f269e3021894 95 Note: option ```-P``` is useful for example in cases when you have same platform and different shields you want to test. No need to test common part all the time (timers, RTC, RTOS etc.). You can force to test peripherals only on some devices and for example only common tests on other devices.
elessair 0:f269e3021894 96
elessair 0:f269e3021894 97 #### Additional MUTs configuration file settings
elessair 0:f269e3021894 98 You can add extra information to each MUT configuration. In particular you can specify which flashing (binary copy method) should be used, how to reset target and for example set reset timeout (used to delay test execution just after reset).
elessair 0:f269e3021894 99
elessair 0:f269e3021894 100 muts_all.json:
elessair 0:f269e3021894 101 ```json
elessair 0:f269e3021894 102 {
elessair 0:f269e3021894 103 "1" : {"mcu" : "LPC1768",
elessair 0:f269e3021894 104 "port" : "COM77",
elessair 0:f269e3021894 105 "disk" : "G:\\",
elessair 0:f269e3021894 106 "peripherals" : ["TMP102", "digital_loop", "port_loop", "analog_loop", "SD"]},
elessair 0:f269e3021894 107
elessair 0:f269e3021894 108 "2" : {"mcu" : "KL25Z",
elessair 0:f269e3021894 109 "port" : "COM89",
elessair 0:f269e3021894 110 "disk" : "F:\\",
elessair 0:f269e3021894 111 "peripherals" : ["SD", "24LC256", "KL25Z"],
elessair 0:f269e3021894 112 "copy_method" : "copy",
elessair 0:f269e3021894 113 "reset_type" : "default",
elessair 0:f269e3021894 114 "reset_tout" : "2"},
elessair 0:f269e3021894 115
elessair 0:f269e3021894 116 "3" : {"mcu" : "LPC11U24",
elessair 0:f269e3021894 117 "port" : "COM76",
elessair 0:f269e3021894 118 "disk" : "E:\\",
elessair 0:f269e3021894 119 "peripherals" : []}
elessair 0:f269e3021894 120 }
elessair 0:f269e3021894 121 ```
elessair 0:f269e3021894 122 Please note that for MUT no. 2 few extra parameters were defined: ```copy_method```, ```reset_type``` and ```reset_tout```. Using this extra options you can tell test suite more about MUT you are using. This will allow you to be more flexible in terms of how you configure and use your MUTs.
elessair 0:f269e3021894 123
elessair 0:f269e3021894 124 * ```copy_method``` - STRING - tells test suite which binary copy method should be used.
elessair 0:f269e3021894 125 You may notice that ```singletest.py``` command line help contains description about:
elessair 0:f269e3021894 126 * Option ```-c``` (in MUTs file called ```copy_method```) with available copy methods supported by test suite plugin system.
elessair 0:f269e3021894 127 * Option ```-r``` (in MUTs file called reset_type) with available reset methods supported by test suite plugin system.
elessair 0:f269e3021894 128 * ```reset_type``` - STRING - some boards may require special reset handling, for example vendor specific command must be executed to reset device.
elessair 0:f269e3021894 129 * ```reset_tout``` - INTEGER - extra timeout just after device is reseted. May be used to wait for few seconds so device may finish booting, flashing data internally etc.
elessair 0:f269e3021894 130
elessair 0:f269e3021894 131 Part of help listing for singletest.py:
elessair 0:f269e3021894 132 ```
elessair 0:f269e3021894 133 -c COPY_METHOD, --copy-method=COPY_METHOD
elessair 0:f269e3021894 134 Select binary copy (flash) method. Default is Python's
elessair 0:f269e3021894 135 shutil.copy() method. Plugin support: copy, cp,
elessair 0:f269e3021894 136 default, eACommander, eACommander-usb, xcopy
elessair 0:f269e3021894 137 -r MUT_RESET_TYPE, --reset-type=MUT_RESET_TYPE
elessair 0:f269e3021894 138 Extra reset method used to reset MUT by host test
elessair 0:f269e3021894 139 script. Plugin support: default, eACommander,
elessair 0:f269e3021894 140 eACommander-usb
elessair 0:f269e3021894 141 ```
elessair 0:f269e3021894 142
elessair 0:f269e3021894 143 ----
elessair 0:f269e3021894 144
elessair 0:f269e3021894 145 Now we've already defined how our devices are connected to our host PC. We can continue and define which of this MUTs will be tested and which compilers we will use to compile and build Mbed SDK and tests. To do so we need to create test specification file (let's call it ```test_spec.json```) and put inside our configuration file information about which MUTs we actually want to test. We will pass this file's name to ```singletest.py``` script using ```-i``` switch.
elessair 0:f269e3021894 146
elessair 0:f269e3021894 147 Below we can see how sample ```test_spec.json``` file content could look like. (I've included all possible toolchains, we will change it in a moment):
elessair 0:f269e3021894 148 ```json
elessair 0:f269e3021894 149 {
elessair 0:f269e3021894 150 "targets": {
elessair 0:f269e3021894 151 "LPC1768" : ["ARM", "uARM", "GCC_ARM", "GCC_CR", "IAR"],
elessair 0:f269e3021894 152 "KL25Z" : ["ARM", "GCC_ARM"],
elessair 0:f269e3021894 153 "NUCLEO_F103RB" : ["ARM", "uARM"]
elessair 0:f269e3021894 154 }
elessair 0:f269e3021894 155 }
elessair 0:f269e3021894 156 ```
elessair 0:f269e3021894 157 Above example configuration will force tests for LPC1768, KL25Z, NUCLEO_F103RB platforms and:
elessair 0:f269e3021894 158
elessair 0:f269e3021894 159 * Compilers: ```ARM```, ```uARM```, ```GCC_ARM```, ```GCC_CR``` and ```IAR``` will be used to compile tests for NXP's ```LPC1768```.
elessair 0:f269e3021894 160 * Compilers: ```ARM``` and ```GCC_ARM``` will be used for Freescales' ```KL25Z``` platform.
elessair 0:f269e3021894 161 * Compilers: ```ARM``` and ```uARM``` will be used for STMicro's ```NUCLEO_F103RB``` platform.
elessair 0:f269e3021894 162
elessair 0:f269e3021894 163 For our example purposes let's assume we only have Keil ARM compiler, so let's change configuration in ```test_spec.json``` file and reduce number of compiler to those we actually have:
elessair 0:f269e3021894 164 ```json
elessair 0:f269e3021894 165 {
elessair 0:f269e3021894 166 "targets": {
elessair 0:f269e3021894 167 "LPC1768" : ["ARM", "uARM"],
elessair 0:f269e3021894 168 "KL25Z" : ["ARM"],
elessair 0:f269e3021894 169 "NUCLEO_F103RB" : ["ARM", "uARM"]
elessair 0:f269e3021894 170 }
elessair 0:f269e3021894 171 }
elessair 0:f269e3021894 172 ```
elessair 0:f269e3021894 173 #### Run your tests
elessair 0:f269e3021894 174
elessair 0:f269e3021894 175 After you configure all your MUTs and compilers you are ready to run tests. Make sure your devices are connected and your configuration files reflect your current configuration (serial ports, devices). Go to tools directory in your mbed location.
elessair 0:f269e3021894 176 ```
elessair 0:f269e3021894 177 $ cd tools/
elessair 0:f269e3021894 178 ```
elessair 0:f269e3021894 179 and execute test suite script.
elessair 0:f269e3021894 180 ```
elessair 0:f269e3021894 181 $ python singletest.py -i test_spec.json -M muts_all.json
elessair 0:f269e3021894 182 ```
elessair 0:f269e3021894 183 To check your configuration before test execution please use ```--config``` switch:
elessair 0:f269e3021894 184 ```
elessair 0:f269e3021894 185 $ python singletest.py -i test_spec.json -M muts_all.json --config
elessair 0:f269e3021894 186 MUTs configuration in m.json:
elessair 0:f269e3021894 187 +-------+-------------+---------------+------+-------+
elessair 0:f269e3021894 188 | index | peripherals | mcu | disk | port |
elessair 0:f269e3021894 189 +-------+-------------+---------------+------+-------+
elessair 0:f269e3021894 190 | 1 | | LPC1768 | J:\ | COM4 |
elessair 0:f269e3021894 191 | 3 | | NUCLEO_F103RB | I:\ | COM11 |
elessair 0:f269e3021894 192 | 2 | | KL25Z | E:\ | COM39 |
elessair 0:f269e3021894 193 +-------+-------------+---------------+------+-------+
elessair 0:f269e3021894 194
elessair 0:f269e3021894 195 Test specification in t.json:
elessair 0:f269e3021894 196 +---------------+-----+------+
elessair 0:f269e3021894 197 | mcu | ARM | uARM |
elessair 0:f269e3021894 198 +---------------+-----+------+
elessair 0:f269e3021894 199 | NUCLEO_F103RB | Yes | Yes |
elessair 0:f269e3021894 200 | KL25Z | Yes | - |
elessair 0:f269e3021894 201 | LPC1768 | Yes | Yes |
elessair 0:f269e3021894 202 +---------------+-----+------+
elessair 0:f269e3021894 203 ```
elessair 0:f269e3021894 204 It should help you localize basic problems with configuration files and toolchain configuration.
elessair 0:f269e3021894 205 Note: Configurations with issues will be marked with ```*``` sign.
elessair 0:f269e3021894 206
elessair 0:f269e3021894 207 Having multiple configuration files allows you to manage your test scenarios in more flexible manner. You can:
elessair 0:f269e3021894 208
elessair 0:f269e3021894 209 * Set up all platforms and toolchains used during testing.
elessair 0:f269e3021894 210 * Define (using script's ```-n``` switch) which tests you want to run during testing.
elessair 0:f269e3021894 211 * Just run regression (all tests). Regression is default setting for test script.
elessair 0:f269e3021894 212
elessair 0:f269e3021894 213 You can also force ```singletest.py``` script to:
elessair 0:f269e3021894 214 * Run only peripherals' tests (switch ```-P```) or
elessair 0:f269e3021894 215 * Just skip peripherals' tests (switch ```-C```).
elessair 0:f269e3021894 216 * Build mbed SDK, libraries and corresponding tests with multiple cores, just use ```-j X``` option where ```X``` is number of cores you want to use for compilation.
elessair 0:f269e3021894 217 ```
elessair 0:f269e3021894 218 $ python singletest.py -i test_spec.json -M muts_all.json -j 8
elessair 0:f269e3021894 219 ```
elessair 0:f269e3021894 220 * Print test cases console output using ```-V``` option.
elessair 0:f269e3021894 221 * Only build mbed SDK, tests and dependant libraries with switch ```-O```:
elessair 0:f269e3021894 222 ```
elessair 0:f269e3021894 223 $ python singletest.py -i test_spec.json -M muts_all.json -j 8 -O
elessair 0:f269e3021894 224 ```
elessair 0:f269e3021894 225 * Execute each test case multiple times with ```--global-loops X``` option, where ```X``` number of repeats. Additionally use option ```-W``` to continue repeating test cases execution only if they continue to fail.
elessair 0:f269e3021894 226 ```
elessair 0:f269e3021894 227 $ python singletest.py -i test_spec.json -M muts_all.json --global-loops 3 -W
elessair 0:f269e3021894 228 ```
elessair 0:f269e3021894 229 * Option ```--loops``` can be used to overwrite global loop count and redefine loop count for particular tests. Define test loops as ```TEST_ID=X``` where ```X``` is integer and separate loops count definitions by comma if necessary. E.g. ```TEST_1=5,TEST_2=20,TEST_3=2```.
elessair 0:f269e3021894 230 ```
elessair 0:f269e3021894 231 $ python singletest.py -i test_spec.json -M muts_all.json RTOS_1=10,RTOS_2=5
elessair 0:f269e3021894 232 ```
elessair 0:f269e3021894 233 This will execute test ```RTOS_1``` ten (10) times and test ```RTOS_2``` five (5) times.
elessair 0:f269e3021894 234 * Force non default copy method. Note that mbed platforms can be flashed with just binary drag&drop. We simply copy file onto mbed's disk and interface chip flashes target MCU with given binary. Force non standard (Python specific) copy method by using option ```-c COPY_METHOD``` where ```COPY_METHOD``` can be shell, command line copy command like: ```cp```, ```copy````, ```xcopy``` etc. Make sure those commands are available from command line!
elessair 0:f269e3021894 235 ```
elessair 0:f269e3021894 236 $ python singletest.py -i test_spec.json -M muts_all.json -c cp
elessair 0:f269e3021894 237 ```
elessair 0:f269e3021894 238 * Run only selected tests. You can select which tests should be executed when you run test suite. Use ```-n``` switch to define tests by their ids you want to execute. Use comma to separate test ids:
elessair 0:f269e3021894 239 ```
elessair 0:f269e3021894 240 $ python singletest.py -i test_spec.json -M muts_all.json -n RTOS_1,RTOS_2,RTOS_3,MBED_10,MBED_16,MBED_11
elessair 0:f269e3021894 241 ```
elessair 0:f269e3021894 242 * Set common output binary name for all tests. In some cases you would like to have the same name for all tests. You can use switch ```--firmware-name``` to specify (without extension) build script output binary name.
elessair 0:f269e3021894 243 In below example we would like to have all test binaries called ```firmware.bin`` (or with other extension like .elf, .hex depending on target accepted format).
elessair 0:f269e3021894 244 ```
elessair 0:f269e3021894 245 $ python singletest.py -i test_spec.json -M muts_all.json --firmware-name firmware
elessair 0:f269e3021894 246 ```
elessair 0:f269e3021894 247 * Where to find test list? Tests are defined in file ```tests.py``` in ```mbed/tools/``` directory. ```singletest.py``` uses test metadata in ```tests.py``` to resolve libraries dependencies and build tests for proper platforms and peripherals. Option ```-R``` can be used to get test names and direct path and test configuration.
elessair 0:f269e3021894 248 ```
elessair 0:f269e3021894 249 $ python singletest.py -R
elessair 0:f269e3021894 250 +-------------+-----------+---------------------------------------+--------------+-------------------+----------+--------------------------------------------------------+
elessair 0:f269e3021894 251 | id | automated | description | peripherals | host_test | duration | source_dir |
elessair 0:f269e3021894 252 +-------------+-----------+---------------------------------------+--------------+-------------------+----------+--------------------------------------------------------+
elessair 0:f269e3021894 253 | MBED_1 | False | I2C SRF08 | SRF08 | host_test | 10 | C:\Work\mbed\libraries\tests\mbed\i2c_SRF08 |
elessair 0:f269e3021894 254 | MBED_10 | True | Hello World | - | host_test | 10 | C:\Work\mbed\libraries\tests\mbed\hello |
elessair 0:f269e3021894 255 | MBED_11 | True | Ticker Int | - | host_test | 20 | C:\Work\mbed\libraries\tests\mbed\ticker |
elessair 0:f269e3021894 256 | MBED_12 | True | C++ | - | host_test | 10 | C:\Work\mbed\libraries\tests\mbed\cpp |
elessair 0:f269e3021894 257 | MBED_13 | False | Heap & Stack | - | host_test | 10 | C:\Work\mbed\libraries\tests\mbed\heap_and_stack |
elessair 0:f269e3021894 258 | MBED_14 | False | Serial Interrupt | - | host_test | 10 | C:\Work\mbed\libraries\tests\mbed\serial_interrupt |
elessair 0:f269e3021894 259 | MBED_15 | False | RPC | - | host_test | 10 | C:\Work\mbed\libraries\tests\mbed\rpc |
elessair 0:f269e3021894 260 | MBED_16 | True | RTC | - | host_test | 15 | C:\Work\mbed\libraries\tests\mbed\rtc |
elessair 0:f269e3021894 261 | MBED_17 | False | Serial Interrupt 2 | - | host_test | 10 | C:\Work\mbed\libraries\tests\mbed\serial_interrupt_2 |
elessair 0:f269e3021894 262 | MBED_18 | False | Local FS Directory | - | host_test | 10 | C:\Work\mbed\libraries\tests\mbed\dir |
elessair 0:f269e3021894 263 ...
elessair 0:f269e3021894 264 ```
elessair 0:f269e3021894 265 Note: you can filter tests by ```id``` column, just use ```-f``` option and give test name or regular expression:
elessair 0:f269e3021894 266 ```
elessair 0:f269e3021894 267 $ python singletest.py -R -f RTOS
elessair 0:f269e3021894 268 +--------------+-----------+-------------------------+-------------+-----------+----------+---------------------------------------------------+
elessair 0:f269e3021894 269 | id | automated | description | peripherals | host_test | duration | source_dir |
elessair 0:f269e3021894 270 +--------------+-----------+-------------------------+-------------+-----------+----------+---------------------------------------------------+
elessair 0:f269e3021894 271 | CMSIS_RTOS_1 | False | Basic | - | host_test | 10 | C:\Work\mbed\libraries\tests\rtos\cmsis\basic |
elessair 0:f269e3021894 272 | CMSIS_RTOS_2 | False | Mutex | - | host_test | 20 | C:\Work\mbed\libraries\tests\rtos\cmsis\mutex |
elessair 0:f269e3021894 273 | CMSIS_RTOS_3 | False | Semaphore | - | host_test | 20 | C:\Work\mbed\libraries\tests\rtos\cmsis\semaphore |
elessair 0:f269e3021894 274 | CMSIS_RTOS_4 | False | Signals | - | host_test | 10 | C:\Work\mbed\libraries\tests\rtos\cmsis\signals |
elessair 0:f269e3021894 275 | CMSIS_RTOS_5 | False | Queue | - | host_test | 20 | C:\Work\mbed\libraries\tests\rtos\cmsis\queue |
elessair 0:f269e3021894 276 | CMSIS_RTOS_6 | False | Mail | - | host_test | 20 | C:\Work\mbed\libraries\tests\rtos\cmsis\mail |
elessair 0:f269e3021894 277 | CMSIS_RTOS_7 | False | Timer | - | host_test | 10 | C:\Work\mbed\libraries\tests\rtos\cmsis\timer |
elessair 0:f269e3021894 278 | CMSIS_RTOS_8 | False | ISR | - | host_test | 10 | C:\Work\mbed\libraries\tests\rtos\cmsis\isr |
elessair 0:f269e3021894 279 | RTOS_1 | True | Basic thread | - | host_test | 15 | C:\Work\mbed\libraries\tests\rtos\mbed\basic |
elessair 0:f269e3021894 280 | RTOS_2 | True | Mutex resource lock | - | host_test | 20 | C:\Work\mbed\libraries\tests\rtos\mbed\mutex |
elessair 0:f269e3021894 281 | RTOS_3 | True | Semaphore resource lock | - | host_test | 20 | C:\Work\mbed\libraries\tests\rtos\mbed\semaphore |
elessair 0:f269e3021894 282 | RTOS_4 | True | Signals messaging | - | host_test | 10 | C:\Work\mbed\libraries\tests\rtos\mbed\signals |
elessair 0:f269e3021894 283 | RTOS_5 | True | Queue messaging | - | host_test | 10 | C:\Work\mbed\libraries\tests\rtos\mbed\queue |
elessair 0:f269e3021894 284 | RTOS_6 | True | Mail messaging | - | host_test | 10 | C:\Work\mbed\libraries\tests\rtos\mbed\mail |
elessair 0:f269e3021894 285 | RTOS_7 | True | Timer | - | host_test | 15 | C:\Work\mbed\libraries\tests\rtos\mbed\timer |
elessair 0:f269e3021894 286 | RTOS_8 | True | ISR (Queue) | - | host_test | 10 | C:\Work\mbed\libraries\tests\rtos\mbed\isr |
elessair 0:f269e3021894 287 | RTOS_9 | True | SD File write-read | SD | host_test | 10 | C:\Work\mbed\libraries\tests\rtos\mbed\file |
elessair 0:f269e3021894 288 +--------------+-----------+-------------------------+-------------+-----------+----------+---------------------------------------------------+
elessair 0:f269e3021894 289 ```
elessair 0:f269e3021894 290
elessair 0:f269e3021894 291 * Shuffle your tests. We strongly encourage you to shuffle your test order each time you execute test suite script.
elessair 0:f269e3021894 292 Rationale: It is probable that tests executed in one particular order will pass and in other will fail. To shuffle your tests’ order please use option ```-u``` (or ```--shuffle```):
elessair 0:f269e3021894 293 ```
elessair 0:f269e3021894 294 $ python singletest.py -i test_spec.json -M muts_all.json --shuffle
elessair 0:f269e3021894 295 ```
elessair 0:f269e3021894 296 Above command will force test script to randomly generate shuffle seed and shuffle test order execution. Note: Shuffle seed is float in ```[0.0, 1.0)```.
elessair 0:f269e3021894 297
elessair 0:f269e3021894 298 You can always recreate particular test order by forcing shuffle (```-u``` or ```--shuffle``` switch) and passing shuffle seed back to test suite using ```--shuffle-seed``` switch:
elessair 0:f269e3021894 299 ```
elessair 0:f269e3021894 300 $ python singletest.py -i test_spec.json -M muts_all.json --shuffle --shuffle-seed 0.4041028336
elessair 0:f269e3021894 301 ```
elessair 0:f269e3021894 302 Note: You can also supply your own randomly generated shuffle seed to drive particular test execution order scenarios. Just make sure shuffle seed is float in ```[0.0, 1.0)```.
elessair 0:f269e3021894 303 You can find test shuffle seed in test summary:
elessair 0:f269e3021894 304 ```
elessair 0:f269e3021894 305 ...
elessair 0:f269e3021894 306 | OK | LPC1768 | ARM | MBED_A9 | Serial Echo at 115200 | 2.84 | 10 | 1/1 |
elessair 0:f269e3021894 307 +--------+---------+-----------+-----------+-----------------------------+--------------------+---------------+-------+
elessair 0:f269e3021894 308 Result: 1 FAIL / 22 OK
elessair 0:f269e3021894 309 Shuffle Seed: 0.4041028336
elessair 0:f269e3021894 310
elessair 0:f269e3021894 311 Completed in 234.85 sec
elessair 0:f269e3021894 312 ```
elessair 0:f269e3021894 313
elessair 0:f269e3021894 314 ### Example of device configuration (one device connected to host computer)
elessair 0:f269e3021894 315
elessair 0:f269e3021894 316 This example will show you how to configure single device, run general tests or only peripheral tests. We will also show some real test result examples.
elessair 0:f269e3021894 317
elessair 0:f269e3021894 318 1. We will test only one board STMIcro Nucleo ```F334R8``` board connected to our PC (port ```COM46``` and disk is ```E:```).
elessair 0:f269e3021894 319 2. We will also connect EEPROM ```24LC256``` to SDA, SCL pins of our Nucleo board and define 24LC256 peripheral to make sure our test suite will run all available tests for ```24LC256```.
elessair 0:f269e3021894 320
elessair 0:f269e3021894 321 Let's configure our one MUT and set uARM as the only compiler we will use to compiler Mbed SDK and tests.
elessair 0:f269e3021894 322 We also need to create two configuration files ```muts_all.json``` and ```test_spec.json``` to pass our small testbed configuration to test script.
elessair 0:f269e3021894 323
elessair 0:f269e3021894 324 muts_all.json:
elessair 0:f269e3021894 325 ```json
elessair 0:f269e3021894 326 {
elessair 0:f269e3021894 327 "1" : {
elessair 0:f269e3021894 328 "mcu": "NUCLEO_F334R8",
elessair 0:f269e3021894 329 "port":"COM46",
elessair 0:f269e3021894 330 "disk":"E:\\",
elessair 0:f269e3021894 331 "peripherals": ["24LC256"]
elessair 0:f269e3021894 332 }
elessair 0:f269e3021894 333 }
elessair 0:f269e3021894 334 ```
elessair 0:f269e3021894 335 Note: By defining ```"peripherals": ["24LC256"]``` we are passing to test suite information that this particular board has EEPROM 24LC256 connected to our board.
elessair 0:f269e3021894 336
elessair 0:f269e3021894 337 test_spec.json:
elessair 0:f269e3021894 338 ```json
elessair 0:f269e3021894 339 {
elessair 0:f269e3021894 340 "targets": {
elessair 0:f269e3021894 341 "NUCLEO_F334R8" : ["uARM"]
elessair 0:f269e3021894 342 }
elessair 0:f269e3021894 343 }
elessair 0:f269e3021894 344 ```
elessair 0:f269e3021894 345 Note:
elessair 0:f269e3021894 346 * Please make sure device is connected before we will start running tests.
elessair 0:f269e3021894 347 * Please make sure files ```muts_all.json``` and ```test_spec.json``` are in ```mbed/tools/``` directory.
elessair 0:f269e3021894 348 Now you can call test suite and execute tests:
elessair 0:f269e3021894 349 ```
elessair 0:f269e3021894 350 $ python singletest.py -i test_spec.json -M muts_all.json
elessair 0:f269e3021894 351 ...
elessair 0:f269e3021894 352 Test summary:
elessair 0:f269e3021894 353 +--------+---------------+-----------+-----------+---------------------------------+--------------------+---------------+
elessair 0:f269e3021894 354 | Result | Target | Toolchain | Test ID | Test Description | Elapsed Time (sec) | Timeout (sec) |
elessair 0:f269e3021894 355 +--------+---------------+-----------+-----------+---------------------------------+--------------------+---------------+
elessair 0:f269e3021894 356 | OK | NUCLEO_F334R8 | uARM | MBED_A25 | I2C EEPROM line read/write test | 12.41 | 15 |
elessair 0:f269e3021894 357 | OK | NUCLEO_F334R8 | uARM | MBED_A1 | Basic | 3.42 | 10 |
elessair 0:f269e3021894 358 | OK | NUCLEO_F334R8 | uARM | EXAMPLE_1 | /dev/null | 3.42 | 10 |
elessair 0:f269e3021894 359 | OK | NUCLEO_F334R8 | uARM | MBED_24 | Timeout Int us | 11.47 | 15 |
elessair 0:f269e3021894 360 | OK | NUCLEO_F334R8 | uARM | MBED_25 | Time us | 11.43 | 15 |
elessair 0:f269e3021894 361 | OK | NUCLEO_F334R8 | uARM | MBED_26 | Integer constant division | 3.37 | 10 |
elessair 0:f269e3021894 362 | OK | NUCLEO_F334R8 | uARM | MBED_23 | Ticker Int us | 12.43 | 15 |
elessair 0:f269e3021894 363 | OK | NUCLEO_F334R8 | uARM | MBED_A19 | I2C EEPROM read/write test | 11.42 | 15 |
elessair 0:f269e3021894 364 | OK | NUCLEO_F334R8 | uARM | MBED_11 | Ticker Int | 12.43 | 20 |
elessair 0:f269e3021894 365 | OK | NUCLEO_F334R8 | uARM | MBED_10 | Hello World | 2.42 | 10 |
elessair 0:f269e3021894 366 | OK | NUCLEO_F334R8 | uARM | MBED_12 | C++ | 3.42 | 10 |
elessair 0:f269e3021894 367 | OK | NUCLEO_F334R8 | uARM | MBED_16 | RTC | 4.76 | 15 |
elessair 0:f269e3021894 368 | UNDEF | NUCLEO_F334R8 | uARM | MBED_2 | stdio | 20.42 | 20 |
elessair 0:f269e3021894 369 | UNDEF | NUCLEO_F334R8 | uARM | MBED_A9 | Serial Echo at 115200 | 10.37 | 10 |
elessair 0:f269e3021894 370 +--------+---------------+-----------+-----------+---------------------------------+--------------------+---------------+
elessair 0:f269e3021894 371 Result: 2 UNDEF / 12 OK
elessair 0:f269e3021894 372
elessair 0:f269e3021894 373 Completed in 160 sec
elessair 0:f269e3021894 374 ```
elessair 0:f269e3021894 375
elessair 0:f269e3021894 376 If we want to get additional test summary with results in separate columns please use option ```-t```.
elessair 0:f269e3021894 377 ```
elessair 0:f269e3021894 378 $ python singletest.py -i test_spec.json -M muts_all.json -t
elessair 0:f269e3021894 379 ...
elessair 0:f269e3021894 380 Test summary:
elessair 0:f269e3021894 381 +---------------+-----------+---------------------------------+-------+
elessair 0:f269e3021894 382 | Target | Test ID | Test Description | uARM |
elessair 0:f269e3021894 383 +---------------+-----------+---------------------------------+-------+
elessair 0:f269e3021894 384 | NUCLEO_F334R8 | EXAMPLE_1 | /dev/null | OK |
elessair 0:f269e3021894 385 | NUCLEO_F334R8 | MBED_10 | Hello World | OK |
elessair 0:f269e3021894 386 | NUCLEO_F334R8 | MBED_11 | Ticker Int | OK |
elessair 0:f269e3021894 387 | NUCLEO_F334R8 | MBED_12 | C++ | OK |
elessair 0:f269e3021894 388 | NUCLEO_F334R8 | MBED_16 | RTC | OK |
elessair 0:f269e3021894 389 | NUCLEO_F334R8 | MBED_2 | stdio | UNDEF |
elessair 0:f269e3021894 390 | NUCLEO_F334R8 | MBED_23 | Ticker Int us | OK |
elessair 0:f269e3021894 391 | NUCLEO_F334R8 | MBED_24 | Timeout Int us | OK |
elessair 0:f269e3021894 392 | NUCLEO_F334R8 | MBED_25 | Time us | OK |
elessair 0:f269e3021894 393 | NUCLEO_F334R8 | MBED_26 | Integer constant division | OK |
elessair 0:f269e3021894 394 | NUCLEO_F334R8 | MBED_A1 | Basic | OK |
elessair 0:f269e3021894 395 | NUCLEO_F334R8 | MBED_A19 | I2C EEPROM read/write test | OK |
elessair 0:f269e3021894 396 | NUCLEO_F334R8 | MBED_A25 | I2C EEPROM line read/write test | OK |
elessair 0:f269e3021894 397 | NUCLEO_F334R8 | MBED_A9 | Serial Echo at 115200 | UNDEF |
elessair 0:f269e3021894 398 +---------------+-----------+---------------------------------+-------+
elessair 0:f269e3021894 399 ```
elessair 0:f269e3021894 400 ----
elessair 0:f269e3021894 401 Please do not forget you can combine few options together to get result you want. For example you want to repeat few tests multiple number of times, shuffle test ids execution order and select only tests which are critical for you at this point. You can do it using switch -n, --global-loops with --loops and --shuffle:
elessair 0:f269e3021894 402
elessair 0:f269e3021894 403 Execute above command to:
elessair 0:f269e3021894 404
elessair 0:f269e3021894 405 * Run only tests: ```RTOS_1```, ```RTOS_2```, ```RTOS_3```, ```MBED_10```, ```MBED_16```, ```MBED_11```.
elessair 0:f269e3021894 406 * Shuffle test execution order. Note tests in loops will not be shuffled.
elessair 0:f269e3021894 407 * Set global loop count to 3 - each test will repeated 3 times.
elessair 0:f269e3021894 408 * Overwrite global loop count (set above to 3) and:
elessair 0:f269e3021894 409 * Force to loop test RTOS_1 to execute 3 times.
elessair 0:f269e3021894 410 * Force to loop test RTOS_2 to execute 4 times.
elessair 0:f269e3021894 411 * Force to loop test RTOS_3 to execute 5 times.
elessair 0:f269e3021894 412 * Force to loop test MBED_11 to execute 5 times.
elessair 0:f269e3021894 413
elessair 0:f269e3021894 414 ```
elessair 0:f269e3021894 415 $ python singletest.py -i test_spec.json -M muts_all.json -n RTOS_1,RTOS_2,RTOS_3,MBED_10,MBED_16,MBED_11 --shuffle --global-loops 3 --loops RTOS_1=3,RTOS_2=4,RTOS_3=5,MBED_11=5
elessair 0:f269e3021894 416 ```
elessair 0:f269e3021894 417
elessair 0:f269e3021894 418 # CppUTest unit test library support
elessair 0:f269e3021894 419 ## CppUTest in Mbed SDK testing introduction
elessair 0:f269e3021894 420 [CppUTest](http://cpputest.github.io/) is a C / C++ based unit xUnit test framework for unit testing and for test-driving your code. It is written in C++ but is used in C and C++ projects and frequently used in embedded systems but it works for any C / C++ project.
elessair 0:f269e3021894 421
elessair 0:f269e3021894 422 Mbed SDK test suite supports writing tests using CppUTest. All you need to do it to provide CppUTest sources and includes with Mbed SDK port. This is already done for you so all you need to do it to get proper sources in your project directory.
elessair 0:f269e3021894 423 CppUTest’s core design principles are:
elessair 0:f269e3021894 424 * Simple in design and simple in use.
elessair 0:f269e3021894 425 * Portable to old and new platforms.
elessair 0:f269e3021894 426 * Build with Test-driven Development in mind.
elessair 0:f269e3021894 427
elessair 0:f269e3021894 428 ## From where you can get more help about CppUTest library and unit testing
elessair 0:f269e3021894 429 • You can read [CppUTest manual](http://cpputest.github.io/manual.html)
elessair 0:f269e3021894 430 * [CppUTest forum](https://groups.google.com/forum/?fromgroups#!forum/cpputest)
elessair 0:f269e3021894 431 * [CppUTest on GitHub](https://github.com/cpputest/cpputest)
elessair 0:f269e3021894 432 * Finally, if you think unit testing is new concept for you, you can have a grasp of it on Wikipedia pages about [unit testing](http://en.wikipedia.org/wiki/Unit_testing) and continue from there.
elessair 0:f269e3021894 433
elessair 0:f269e3021894 434 ## How to add CppUTest to your current Mbed SDK installation
elessair 0:f269e3021894 435
elessair 0:f269e3021894 436 ### Do I need CppUTest port for Mbed SDK?
elessair 0:f269e3021894 437 Yes, you do. If you want to use CppUTest with Mbed SDK you need to have CppUTest version with ARMCC compiler (only ARM flavor for now) port and Mbed SDK console port (if you want to have output on serial port). All is already prepared by Mbed engineers and you can get it for example here: http://mbed.org/users/rgrover1/code/CppUTest/
elessair 0:f269e3021894 438
elessair 0:f269e3021894 439 ### Prerequisites
elessair 0:f269e3021894 440 * Installed [git client](http://git-scm.com/downloads/).
elessair 0:f269e3021894 441 * Installed [Mercurial client](http://mercurial.selenic.com/).
elessair 0:f269e3021894 442
elessair 0:f269e3021894 443 ### How / where to install
elessair 0:f269e3021894 444 We want to create directory structure similar to one below:
elessair 0:f269e3021894 445 ```
elessair 0:f269e3021894 446 \your_project_directory
elessair 0:f269e3021894 447 │
elessair 0:f269e3021894 448 ├───cpputest
elessair 0:f269e3021894 449 │ ├───include
elessair 0:f269e3021894 450 │ └───src
elessair 0:f269e3021894 451 └───mbed
elessair 0:f269e3021894 452 ├───libraries
elessair 0:f269e3021894 453 ├───travis
elessair 0:f269e3021894 454 └───tools
elessair 0:f269e3021894 455 ```
elessair 0:f269e3021894 456
elessair 0:f269e3021894 457 Please go to directory with your project. For example it could be c:\Projects\Project.
elessair 0:f269e3021894 458 ```
elessair 0:f269e3021894 459 $ cd c:\Projects\Project
elessair 0:f269e3021894 460 ```
elessair 0:f269e3021894 461 If your project directory already has your mbed SDK repository included just execute below command (Mercurial console client). It should download CppUTest with Mbed SDK port.
elessair 0:f269e3021894 462 ```
elessair 0:f269e3021894 463 $ hg clone https://mbed.org/users/rgrover1/code/cpputest/
elessair 0:f269e3021894 464 ```
elessair 0:f269e3021894 465
elessair 0:f269e3021894 466 You should see something like this after you execute Mercurial clone command:
elessair 0:f269e3021894 467 ```
elessair 0:f269e3021894 468 $ hg clone https://mbed.org/users/rgrover1/code/cpputest/
elessair 0:f269e3021894 469 destination directory: cpputest
elessair 0:f269e3021894 470 requesting all changes
elessair 0:f269e3021894 471 adding changesets
elessair 0:f269e3021894 472 adding manifests
elessair 0:f269e3021894 473 adding file changes
elessair 0:f269e3021894 474 added 3 changesets with 69 changes to 42 files
elessair 0:f269e3021894 475 updating to branch default
elessair 0:f269e3021894 476 41 files updated, 0 files merged, 0 files removed, 0 files unresolved
elessair 0:f269e3021894 477 ```
elessair 0:f269e3021894 478
elessair 0:f269e3021894 479 Confirm your project structure. It should look more or less like this:
elessair 0:f269e3021894 480 ```
elessair 0:f269e3021894 481 $ ls
elessair 0:f269e3021894 482 cpputest mbed
elessair 0:f269e3021894 483 ```
elessair 0:f269e3021894 484 From now on CppUTest is in correct path. Each time you want to compile unit tests for CppUTest build script will always look for CppUTest library in the same directory where mbed library is.
elessair 0:f269e3021894 485
elessair 0:f269e3021894 486 ## New off-line mbed SDK project with CppUTest support
elessair 0:f269e3021894 487
elessair 0:f269e3021894 488 If you are creating new mbed SDK project and you want to use CppUTest with it you need to download both mbed SDK and CppUTest with mbed port to the same directory. You can do it like this:
elessair 0:f269e3021894 489 ```
elessair 0:f269e3021894 490 $ cd c:\Projects\Project
elessair 0:f269e3021894 491 $ git clone https://github.com/mbedmicro/mbed.git
elessair 0:f269e3021894 492 $ hg clone https://mbed.org/users/rgrover1/code/cpputest/
elessair 0:f269e3021894 493 ```
elessair 0:f269e3021894 494
elessair 0:f269e3021894 495 After above three steps you should have proper directory structure. All you need to do now is to configure your ```mbed_settings.py``` in ```mbed``` directory. Please refer to mbed SDK build script documentation for details.
elessair 0:f269e3021894 496
elessair 0:f269e3021894 497 ## CppUTest with mbed port
elessair 0:f269e3021894 498 To make sure you actualy have CppUTest library with mbed SDK port you can go to CppUTest ```armcc``` platform directory:
elessair 0:f269e3021894 499 ```
elessair 0:f269e3021894 500 $ cd c:/Projects/Project/cpputest/src/Platforms/armcc/
elessair 0:f269e3021894 501 ```
elessair 0:f269e3021894 502 And open file ```UtestPlatform.cpp```.
elessair 0:f269e3021894 503
elessair 0:f269e3021894 504 You should find part of code responsible for porting console on default serial port of the mbed device:
elessair 0:f269e3021894 505 ```c++
elessair 0:f269e3021894 506 #include "Serial.h"
elessair 0:f269e3021894 507 using namespace mbed;
elessair 0:f269e3021894 508
elessair 0:f269e3021894 509 int PlatformSpecificPutchar(int c)
elessair 0:f269e3021894 510 {
elessair 0:f269e3021894 511 /* Please modify this block for test results to be reported as
elessair 0:f269e3021894 512 * console output. The following is a sample implementation using a
elessair 0:f269e3021894 513 * Serial object connected to the console. */
elessair 0:f269e3021894 514 #define NEED_TEST_REPORT_AS_CONSOLE_OUTPUT 1
elessair 0:f269e3021894 515 #if NEED_TEST_REPORT_AS_CONSOLE_OUTPUT
elessair 0:f269e3021894 516 extern Serial console;
elessair 0:f269e3021894 517
elessair 0:f269e3021894 518 #define NEED_LINE_FEED_IN_ADDITION_TO_NEWLINE 1
elessair 0:f269e3021894 519 #if NEED_LINE_FEED_IN_ADDITION_TO_NEWLINE
elessair 0:f269e3021894 520 /* CppUTest emits \n line terminators in its reports; some terminals
elessair 0:f269e3021894 521 * need the linefeed (\r) in addition. */
elessair 0:f269e3021894 522 if (c == '\n') {
elessair 0:f269e3021894 523 console.putc('\r');
elessair 0:f269e3021894 524 }
elessair 0:f269e3021894 525 #endif /* #if NEED_LINE_FEED_IN_ADDITION_TO_NEWLINE */
elessair 0:f269e3021894 526
elessair 0:f269e3021894 527 return (console.putc(c));
elessair 0:f269e3021894 528 #else /* NEED_TEST_REPORT_AS_CONSOLE_OUTPUT */
elessair 0:f269e3021894 529 return (0);
elessair 0:f269e3021894 530 #endif /* NEED_TEST_REPORT_AS_CONSOLE_OUTPUT */
elessair 0:f269e3021894 531 }
elessair 0:f269e3021894 532 ```
elessair 0:f269e3021894 533
elessair 0:f269e3021894 534 You can find cpputest UT test runner main function in mbed sources: ```c:/Projects/Project/mbed/libraries/tests/utest/testrunner/testrunner.cpp```. Test runner code (in ```testrunner.cpp```) only defined console object and executes all unit tests:
elessair 0:f269e3021894 535 ```c++
elessair 0:f269e3021894 536 #include "CommandLineTestRunner.h"
elessair 0:f269e3021894 537 #include <stdio.h>
elessair 0:f269e3021894 538 #include "mbed.h"
elessair 0:f269e3021894 539 #include "testrunner.h"
elessair 0:f269e3021894 540 #include "test_env.h"
elessair 0:f269e3021894 541
elessair 0:f269e3021894 542 /**
elessair 0:f269e3021894 543 Object 'mbed_cpputest_console' is used to show prints on console.
elessair 0:f269e3021894 544 It is declared in \cpputest\src\Platforms\armcc\UtestPlatform.cpp
elessair 0:f269e3021894 545 */
elessair 0:f269e3021894 546 Serial mbed_cpputest_console(STDIO_UART_TX, STDIO_UART_RX);
elessair 0:f269e3021894 547
elessair 0:f269e3021894 548 int main(int ac, char** av) {
elessair 0:f269e3021894 549 MBED_HOSTTEST_TIMEOUT(20);
elessair 0:f269e3021894 550 MBED_HOSTTEST_SELECT(default_auto);
elessair 0:f269e3021894 551 MBED_HOSTTEST_DESCRIPTION(Unit test);
elessair 0:f269e3021894 552 MBED_HOSTTEST_START("UT");
elessair 0:f269e3021894 553
elessair 0:f269e3021894 554 unsigned failureCount = 0;
elessair 0:f269e3021894 555 {
elessair 0:f269e3021894 556 // Some compilers may not pass ac, av so we need to supply them ourselves
elessair 0:f269e3021894 557 int ac = 2;
elessair 0:f269e3021894 558 char* av[] = {__FILE__, "-v"};
elessair 0:f269e3021894 559 failureCount = CommandLineTestRunner::RunAllTests(ac, av);
elessair 0:f269e3021894 560 }
elessair 0:f269e3021894 561
elessair 0:f269e3021894 562 MBED_HOSTTEST_RESULT(failureCount == 0);
elessair 0:f269e3021894 563 return failureCount;
elessair 0:f269e3021894 564 }
elessair 0:f269e3021894 565 ```
elessair 0:f269e3021894 566
elessair 0:f269e3021894 567 ## Unit test location
elessair 0:f269e3021894 568 Unit tests source code is located in below directory: ```c:/Projects/Project/mbed/libraries/tests/utest/```
elessair 0:f269e3021894 569
elessair 0:f269e3021894 570 Each sub directory except testrunner contains compilable unit test source files with test groups and test cases. You can see utest structure below. Please note this is just example and in the future this directory will contain many sub directories with unit tests.
elessair 0:f269e3021894 571 ```
elessair 0:f269e3021894 572 $ c:\Projects\Project\mbed\libraries\tests\utest> tree
elessair 0:f269e3021894 573 utest
elessair 0:f269e3021894 574 ├───basic
elessair 0:f269e3021894 575 ├───semihost_fs
elessair 0:f269e3021894 576 └───testrunner
elessair 0:f269e3021894 577 ```
elessair 0:f269e3021894 578
elessair 0:f269e3021894 579 ## Define unit tests in mbed SDK test suite structure
elessair 0:f269e3021894 580 All tests defined in test suite are described in ```mbed/tools/tests.py``` file. This file stores data structure ```TESTS``` which is a list of simple structures describing each test. Below you can find example of ```TESTS``` structure which is configuring one of the unit tests.
elessair 0:f269e3021894 581 ```
elessair 0:f269e3021894 582 .
elessair 0:f269e3021894 583 .
elessair 0:f269e3021894 584 .
elessair 0:f269e3021894 585 {
elessair 0:f269e3021894 586 "id": "UT_2", "description": "Semihost file system",
elessair 0:f269e3021894 587 "source_dir": join(TEST_DIR, "utest", "file"),
elessair 0:f269e3021894 588 "dependencies": [MBED_LIBRARIES, TEST_MBED_LIB, CPPUTEST_LIBRARY],
elessair 0:f269e3021894 589 "automated": False,
elessair 0:f269e3021894 590 "mcu": ["LPC1768", "LPC2368", "LPC11U24"]
elessair 0:f269e3021894 591 },
elessair 0:f269e3021894 592 .
elessair 0:f269e3021894 593 .
elessair 0:f269e3021894 594 .
elessair 0:f269e3021894 595 ```
elessair 0:f269e3021894 596 Note: In dependency section we've added library ```CPPUTEST_LIBRARY``` which is pointing build script to CppUTest library with mbed port. This is a must for unit tests to be compiled with CppUTest library.
elessair 0:f269e3021894 597
elessair 0:f269e3021894 598 ### Tests are now divided into two types:
elessair 0:f269e3021894 599 #### 'Hello world' tests
elessair 0:f269e3021894 600 First type of test cases we call 'hello world' tests. They do not dependent on CppUTest library and are monolithic programs usually composed of one main function. You can find this tests in below example directories:
elessair 0:f269e3021894 601
elessair 0:f269e3021894 602 * ```mbed/libraries/tests/mbed/```
elessair 0:f269e3021894 603 * ```mbed/libraries/tests/net/```
elessair 0:f269e3021894 604 * ```mbed/libraries/tests/rtos/```
elessair 0:f269e3021894 605 * ```mbed/libraries/tests/usb/```
elessair 0:f269e3021894 606
elessair 0:f269e3021894 607 Usually ‘hello world’ test cases are using ```test_env.cpp``` and ```test_env.h``` files which implement simple test framework used to communicate with host test and help test framework instrument your tests.
elessair 0:f269e3021894 608
elessair 0:f269e3021894 609 Below you can see listing of ```test_env.h``` file which contains simple macro definitions used to communicate (via serial port printouts) between test case (on hardware) and host test script (on host computer).
elessair 0:f269e3021894 610 Each use case should print on console basic information like:
elessair 0:f269e3021894 611 * Default test case timeout.
elessair 0:f269e3021894 612 * Which host test should be used to supervise test case execution.
elessair 0:f269e3021894 613 * Test description and test case ID (short identifier).
elessair 0:f269e3021894 614
elessair 0:f269e3021894 615 ```c++
elessair 0:f269e3021894 616 .
elessair 0:f269e3021894 617 .
elessair 0:f269e3021894 618 .
elessair 0:f269e3021894 619 // Test result related notification functions
elessair 0:f269e3021894 620 void notify_start();
elessair 0:f269e3021894 621 void notify_completion(bool success);
elessair 0:f269e3021894 622 bool notify_completion_str(bool success, char* buffer);
elessair 0:f269e3021894 623 void notify_performance_coefficient(const char* measurement_name, const int value);
elessair 0:f269e3021894 624 void notify_performance_coefficient(const char* measurement_name, const unsigned int value);
elessair 0:f269e3021894 625 void notify_performance_coefficient(const char* measurement_name, const double value);
elessair 0:f269e3021894 626
elessair 0:f269e3021894 627 // Host test auto-detection API
elessair 0:f269e3021894 628 void notify_host_test_name(const char *host_test);
elessair 0:f269e3021894 629 void notify_timeout(int timeout);
elessair 0:f269e3021894 630 void notify_test_id(const char *test_id);
elessair 0:f269e3021894 631 void notify_test_description(const char *description);
elessair 0:f269e3021894 632
elessair 0:f269e3021894 633 // Host test auto-detection API
elessair 0:f269e3021894 634 #define MBED_HOSTTEST_START(TESTID) notify_test_id(TESTID); notify_start()
elessair 0:f269e3021894 635 #define MBED_HOSTTEST_SELECT(NAME) notify_host_test_name(#NAME)
elessair 0:f269e3021894 636 #define MBED_HOSTTEST_TIMEOUT(SECONDS) notify_timeout(SECONDS)
elessair 0:f269e3021894 637 #define MBED_HOSTTEST_DESCRIPTION(DESC) notify_test_description(#DESC)
elessair 0:f269e3021894 638 #define MBED_HOSTTEST_RESULT(RESULT) notify_completion(RESULT)
elessair 0:f269e3021894 639
elessair 0:f269e3021894 640 /**
elessair 0:f269e3021894 641 Test auto-detection preamble example:
elessair 0:f269e3021894 642 main() {
elessair 0:f269e3021894 643 MBED_HOSTTEST_TIMEOUT(10);
elessair 0:f269e3021894 644 MBED_HOSTTEST_SELECT( host_test );
elessair 0:f269e3021894 645 MBED_HOSTTEST_DESCRIPTION(Hello World);
elessair 0:f269e3021894 646 MBED_HOSTTEST_START("MBED_10");
elessair 0:f269e3021894 647 // Proper 'host_test.py' should take over supervising of this test
elessair 0:f269e3021894 648
elessair 0:f269e3021894 649 // Test code
elessair 0:f269e3021894 650 bool result = ...;
elessair 0:f269e3021894 651
elessair 0:f269e3021894 652 MBED_HOSTTEST_RESULT(result);
elessair 0:f269e3021894 653 }
elessair 0:f269e3021894 654 */
elessair 0:f269e3021894 655 .
elessair 0:f269e3021894 656 .
elessair 0:f269e3021894 657 .
elessair 0:f269e3021894 658 ```
elessair 0:f269e3021894 659
elessair 0:f269e3021894 660 Example of 'hello world' test:
elessair 0:f269e3021894 661 ```c++
elessair 0:f269e3021894 662 #include "mbed.h"
elessair 0:f269e3021894 663 #include "test_env.h"
elessair 0:f269e3021894 664
elessair 0:f269e3021894 665 #define CUSTOM_TIME 1256729737
elessair 0:f269e3021894 666
elessair 0:f269e3021894 667 int main() {
elessair 0:f269e3021894 668 MBED_HOSTTEST_TIMEOUT(20);
elessair 0:f269e3021894 669 MBED_HOSTTEST_SELECT(rtc_auto);
elessair 0:f269e3021894 670 MBED_HOSTTEST_DESCRIPTION(RTC);
elessair 0:f269e3021894 671 MBED_HOSTTEST_START("MBED_16");
elessair 0:f269e3021894 672
elessair 0:f269e3021894 673 char buffer[32] = {0};
elessair 0:f269e3021894 674 set_time(CUSTOM_TIME); // Set RTC time to Wed, 28 Oct 2009 11:35:37
elessair 0:f269e3021894 675 while(1) {
elessair 0:f269e3021894 676 time_t seconds = time(NULL);
elessair 0:f269e3021894 677 strftime(buffer, 32, "%Y-%m-%d %H:%M:%S %p", localtime(&seconds));
elessair 0:f269e3021894 678 printf("MBED: [%ld] [%s]\r\n", seconds, buffer);
elessair 0:f269e3021894 679 wait(1);
elessair 0:f269e3021894 680 }
elessair 0:f269e3021894 681 }
elessair 0:f269e3021894 682 ```
elessair 0:f269e3021894 683
elessair 0:f269e3021894 684 #### 'Unit test' test cases
elessair 0:f269e3021894 685 Second group of tests are unit tests. They are using CppUTest library and require you to write ```TEST_GROUP```s and ```TEST```s in your test files. Test suite will add test runner sources to your test automatically so you can concentrate on writing tests.
elessair 0:f269e3021894 686
elessair 0:f269e3021894 687 Example of unit test:
elessair 0:f269e3021894 688 ```c++
elessair 0:f269e3021894 689 #include "TestHarness.h"
elessair 0:f269e3021894 690 #include <utility>
elessair 0:f269e3021894 691 #include "mbed.h"
elessair 0:f269e3021894 692
elessair 0:f269e3021894 693 TEST_GROUP(BusOut_mask)
elessair 0:f269e3021894 694 {
elessair 0:f269e3021894 695 };
elessair 0:f269e3021894 696
elessair 0:f269e3021894 697 TEST(BusOut_mask, led_1_2_3)
elessair 0:f269e3021894 698 {
elessair 0:f269e3021894 699 BusOut bus_data(LED1, LED2, LED3);
elessair 0:f269e3021894 700 CHECK_EQUAL(0x07, bus_data.mask());
elessair 0:f269e3021894 701 }
elessair 0:f269e3021894 702
elessair 0:f269e3021894 703 TEST(BusOut_mask, led_nc_nc_nc_nc)
elessair 0:f269e3021894 704 {
elessair 0:f269e3021894 705 BusOut bus_data(NC, NC, NC, NC);
elessair 0:f269e3021894 706 CHECK_EQUAL(0x00, bus_data.mask());
elessair 0:f269e3021894 707 }
elessair 0:f269e3021894 708
elessair 0:f269e3021894 709 TEST(BusOut_mask, led_1_2_3_nc_nc)
elessair 0:f269e3021894 710 {
elessair 0:f269e3021894 711 BusOut bus_data(LED1, LED2, LED3, NC, NC);
elessair 0:f269e3021894 712 CHECK_EQUAL(0x07, bus_data.mask());
elessair 0:f269e3021894 713 }
elessair 0:f269e3021894 714
elessair 0:f269e3021894 715 TEST(BusOut_mask, led_1_nc_2_nc_nc_3)
elessair 0:f269e3021894 716 {
elessair 0:f269e3021894 717 BusOut bus_data(LED1, NC, LED2, NC, NC, LED3);
elessair 0:f269e3021894 718 CHECK_EQUAL(0x25, bus_data.mask());
elessair 0:f269e3021894 719 }
elessair 0:f269e3021894 720 ```
elessair 0:f269e3021894 721
elessair 0:f269e3021894 722 ## Example
elessair 0:f269e3021894 723 In below example we will run two example unit tests that are now available. tests ```UT_1``` and ```UT_2``` are unit tests used for now only to check if mbed SDK works with CppUTest library and if tests are being executed. In future number of unit tests will increase, nothing is also stopping you from writing and executing your own unit tests!
elessair 0:f269e3021894 724
elessair 0:f269e3021894 725 ### Example configuration
elessair 0:f269e3021894 726 By default unit tests ```UT_1``` and ```UT_2``` are not automated - simply test suite will ignore them. Also we do not want to create dependency to CppUTest library each time someone executes automation.
elessair 0:f269e3021894 727
elessair 0:f269e3021894 728 Note: To compile ```UT_1``` and ```UT_2``` tests CppUTest library described above, installation is needed and not all users wish to add UT libs to their project. Also new to mbed users may find it difficult. This is why unit testing is an extra feature active only after you deliberately install and enable needed components.
elessair 0:f269e3021894 729
elessair 0:f269e3021894 730 Bellow snippet shows how to modify 'automated' flag so test suite will consider unit tests ```UT_1``` and ```UT_2``` as part of "automated test portfolio". Just change flag 'automated' from ```False``` to ```True```.
elessair 0:f269e3021894 731
elessair 0:f269e3021894 732 ```tests.py``` listing related to ```UT_1``` and ```UT_2```:
elessair 0:f269e3021894 733 ```python
elessair 0:f269e3021894 734 .
elessair 0:f269e3021894 735 .
elessair 0:f269e3021894 736 .
elessair 0:f269e3021894 737 # CPPUTEST Library provides Unit testing Framework
elessair 0:f269e3021894 738 #
elessair 0:f269e3021894 739 # To write TESTs and TEST_GROUPs please add CPPUTEST_LIBRARY to 'dependencies'
elessair 0:f269e3021894 740 #
elessair 0:f269e3021894 741 # This will also include:
elessair 0:f269e3021894 742 # 1. test runner - main function with call to CommandLineTestRunner::RunAllTests(ac, av)
elessair 0:f269e3021894 743 # 2. Serial console object to print test result on serial port console
elessair 0:f269e3021894 744 #
elessair 0:f269e3021894 745
elessair 0:f269e3021894 746 # Unit testing with cpputest library
elessair 0:f269e3021894 747 {
elessair 0:f269e3021894 748 "id": "UT_1", "description": "Basic",
elessair 0:f269e3021894 749 "source_dir": join(TEST_DIR, "utest", "basic"),
elessair 0:f269e3021894 750 "dependencies": [MBED_LIBRARIES, TEST_MBED_LIB, CPPUTEST_LIBRARY],
elessair 0:f269e3021894 751 "automated": True,
elessair 0:f269e3021894 752 },
elessair 0:f269e3021894 753 {
elessair 0:f269e3021894 754 "id": "UT_2", "description": "Semihost file system",
elessair 0:f269e3021894 755 "source_dir": join(TEST_DIR, "utest", "semihost_fs"),
elessair 0:f269e3021894 756 "dependencies": [MBED_LIBRARIES, TEST_MBED_LIB, CPPUTEST_LIBRARY],
elessair 0:f269e3021894 757 "automated": True,
elessair 0:f269e3021894 758 "mcu": ["LPC1768", "LPC2368", "LPC11U24"]
elessair 0:f269e3021894 759 },
elessair 0:f269e3021894 760 .
elessair 0:f269e3021894 761 .
elessair 0:f269e3021894 762 .
elessair 0:f269e3021894 763 ```
elessair 0:f269e3021894 764
elessair 0:f269e3021894 765 ### Execute tests
elessair 0:f269e3021894 766 In my test I will use common [LPC1768](http://developer.mbed.org/platforms/mbed-LPC1768/) mbed-enabled board because unit test ```UT_2``` is checking semi-host functionality which is available on this board and handful of others.
elessair 0:f269e3021894 767
elessair 0:f269e3021894 768 Configure your ```test_spec.json``` and ```muts_all.json``` files (refer to test suite build script and automation description) and set mbed disk and serial port.
elessair 0:f269e3021894 769
elessair 0:f269e3021894 770 ```
elessair 0:f269e3021894 771 $ singletest.py -i test_spec.json -M muts_all.json -n UT_1,UT_2 -V
elessair 0:f269e3021894 772 Building library CMSIS (LPC1768, ARM)
elessair 0:f269e3021894 773 Building library MBED (LPC1768, ARM)
elessair 0:f269e3021894 774 Building library CPPUTEST (LPC1768, ARM)
elessair 0:f269e3021894 775 Building project BASIC (LPC1768, ARM)
elessair 0:f269e3021894 776 Executing 'python host_test.py -p COM77 -d E:\ -t 10'
elessair 0:f269e3021894 777 Test::Output::Start
elessair 0:f269e3021894 778 Host test instrumentation on port: "COM77" and disk: "E:\"
elessair 0:f269e3021894 779 TEST(FirstTestGroup, FirstTest) - 0 ms
elessair 0:f269e3021894 780
elessair 0:f269e3021894 781 OK (1 tests, 1 ran, 3 checks, 0 ignored, 0 filtered out, 3 ms)
elessair 0:f269e3021894 782
elessair 0:f269e3021894 783 {{success}}
elessair 0:f269e3021894 784 {{end}}
elessair 0:f269e3021894 785 Test::Output::Finish
elessair 0:f269e3021894 786 TargetTest::LPC1768::ARM::UT_1::Basic [OK] in 2.43 of 10 sec
elessair 0:f269e3021894 787 Building library CPPUTEST (LPC1768, ARM)
elessair 0:f269e3021894 788 Building project SEMIHOST_FS (LPC1768, ARM)
elessair 0:f269e3021894 789 Executing 'python host_test.py -p COM77 -d E:\ -t 10'
elessair 0:f269e3021894 790 Test::Output::Start
elessair 0:f269e3021894 791 Host test instrumentation on port: "COM77" and disk: "E:\"
elessair 0:f269e3021894 792 TEST(FirstTestGroup, FirstTest) - 9 ms
elessair 0:f269e3021894 793
elessair 0:f269e3021894 794 OK (1 tests, 1 ran, 10 checks, 0 ignored, 0 filtered out, 10 ms)
elessair 0:f269e3021894 795
elessair 0:f269e3021894 796 {{success}}
elessair 0:f269e3021894 797 {{end}}
elessair 0:f269e3021894 798 Test::Output::Finish
elessair 0:f269e3021894 799 TargetTest::LPC1768::ARM::UT_2::Semihost file system [OK] in 2.43 of 10 sec
elessair 0:f269e3021894 800 Test summary:
elessair 0:f269e3021894 801 +--------+---------+-----------+---------+----------------------+--------------------+---------------+-------+
elessair 0:f269e3021894 802 | Result | Target | Toolchain | Test ID | Test Description | Elapsed Time (sec) | Timeout (sec) | Loops |
elessair 0:f269e3021894 803 +--------+---------+-----------+---------+----------------------+--------------------+---------------+-------+
elessair 0:f269e3021894 804 | OK | LPC1768 | ARM | UT_1 | Basic | 2.43 | 10 | 1/1 |
elessair 0:f269e3021894 805 | OK | LPC1768 | ARM | UT_2 | Semihost file system | 2.43 | 10 | 1/1 |
elessair 0:f269e3021894 806 +--------+---------+-----------+---------+----------------------+--------------------+---------------+-------+
elessair 0:f269e3021894 807 Result: 2 OK
elessair 0:f269e3021894 808
elessair 0:f269e3021894 809 Completed in 12.02 sec
elessair 0:f269e3021894 810 ```
elessair 0:f269e3021894 811
elessair 0:f269e3021894 812 You can compile unit tests using various number of supported compilers, below just few examples with working solutions:
elessair 0:f269e3021894 813 ```
elessair 0:f269e3021894 814 Test summary:
elessair 0:f269e3021894 815 +--------+---------+-----------+---------+----------------------+--------------------+---------------+-------+
elessair 0:f269e3021894 816 | Result | Target | Toolchain | Test ID | Test Description | Elapsed Time (sec) | Timeout (sec) | Loops |
elessair 0:f269e3021894 817 +--------+---------+-----------+---------+----------------------+--------------------+---------------+-------+
elessair 0:f269e3021894 818 | OK | LPC1768 | ARM | UT_1 | Basic | 2.43 | 10 | 1/1 |
elessair 0:f269e3021894 819 | OK | LPC1768 | ARM | UT_2 | Semihost file system | 2.43 | 10 | 1/1 |
elessair 0:f269e3021894 820 | OK | LPC1768 | uARM | UT_1 | Basic | 2.43 | 10 | 1/1 |
elessair 0:f269e3021894 821 | OK | LPC1768 | uARM | UT_2 | Semihost file system | 2.43 | 10 | 1/1 |
elessair 0:f269e3021894 822 | OK | LPC1768 | GCC_ARM | UT_1 | Basic | 2.43 | 10 | 1/1 |
elessair 0:f269e3021894 823 | OK | LPC1768 | GCC_ARM | UT_2 | Semihost file system | 2.43 | 10 | 1/1 |
elessair 0:f269e3021894 824 | OK | LPC1768 | GCC_CR | UT_1 | Basic | 3.44 | 10 | 1/1 |
elessair 0:f269e3021894 825 | OK | LPC1768 | GCC_CR | UT_2 | Semihost file system | 3.43 | 10 | 1/1 |
elessair 0:f269e3021894 826 +--------+---------+-----------+---------+----------------------+--------------------+---------------+-------+
elessair 0:f269e3021894 827 Result: 8 OK
elessair 0:f269e3021894 828
elessair 0:f269e3021894 829 Completed in 55.85 sec
elessair 0:f269e3021894 830 ```