Maxim nexpaq / nexpaq_dev

Preliminary main mbed library for nexpaq development

docs/TESTING.md@0:6c56fb4bc5f0, 2016-11-04 (annotated)

Committer:
nexpaq
Date:
Fri Nov 04 20:27:58 2016 +0000
Revision:
0:6c56fb4bc5f0
Moving to library for sharing updates

Who changed what in which revision?

UserRevisionLine numberNew contents of line
nexpaq 0:6c56fb4bc5f0 1 # Mbed SDK automated test suite
nexpaq 0:6c56fb4bc5f0 2 ## Introduction
nexpaq 0:6c56fb4bc5f0 3
nexpaq 0:6c56fb4bc5f0 4 Test suit allows users to run locally on their machines Mbed SDK’s tests included in Mbed SDK repository. It also allows users to create their own tests and for example add new tests to test set as they progress with their project. If test is generic enough it could be included into official Mbed SDK test pool just do it via normal pull request!
nexpaq 0:6c56fb4bc5f0 5
nexpaq 0:6c56fb4bc5f0 6 Each test is supervised by python script called “host test” which will at least Test suite is using build script API to compile and build test source together with required by test libraries like CMSIS, Mbed, Ethernet, USB etc.
nexpaq 0:6c56fb4bc5f0 7
nexpaq 0:6c56fb4bc5f0 8 ## What is host test?
nexpaq 0:6c56fb4bc5f0 9 Test suite supports test supervisor concept. This concept is realized by separate Python script called ```host test```. Host tests can be found in ```mbed/tools/host_tests/``` directory. Note: In newer mbed versions (mbed OS) host tests will be separate library.
nexpaq 0:6c56fb4bc5f0 10
nexpaq 0:6c56fb4bc5f0 11 Host test script is executed in parallel with test runner to monitor test execution. Basic host test just monitors device's default serial port for test results returned by test runner. Simple tests will print test result on serial port. In other cases host tests can for example judge by test results returned by test runner if test passed or failed. It all depends on test itself.
nexpaq 0:6c56fb4bc5f0 12
nexpaq 0:6c56fb4bc5f0 13 In some cases host test can be TCP server echoing packets from test runner and judging packet loss. In other cases it can just check if values returned from accelerometer are actually valid (sane).
nexpaq 0:6c56fb4bc5f0 14
nexpaq 0:6c56fb4bc5f0 15 ## Test suite core: singletest.py script
nexpaq 0:6c56fb4bc5f0 16
nexpaq 0:6c56fb4bc5f0 17 ```singletest.py``` script located in ```mbed/tools/``` is a test suite script which allows users to compile, build tests and test runners (also supports CppUTest unit test library). Script also is responsible for test execution on devices selected by configuration files.
nexpaq 0:6c56fb4bc5f0 18
nexpaq 0:6c56fb4bc5f0 19 ### Parameters of singletest.py
nexpaq 0:6c56fb4bc5f0 20
nexpaq 0:6c56fb4bc5f0 21 Test execution script ```singletest.py``` is a fairly powerful tool to run tests for mbed SDK platform. It is flexible and allows users to configure test execution process and define which mbed platforms will be tested.
nexpaq 0:6c56fb4bc5f0 22
nexpaq 0:6c56fb4bc5f0 23 By specifying external configuration files (in JSON format) you can gain flexibility and prepare many different test scenarios. Just pass configuration file names to your script and run it.
nexpaq 0:6c56fb4bc5f0 24
nexpaq 0:6c56fb4bc5f0 25 #### MUTs Specification
nexpaq 0:6c56fb4bc5f0 26 You can easily configure your MUTs (Mbed Under Test) by creating configuration file with MUTs description.
nexpaq 0:6c56fb4bc5f0 27 Note: This configuration file must be in [JSON format](http://www.w3schools.com/json/).
nexpaq 0:6c56fb4bc5f0 28 Note: Unfortunately JSON format is not allowing you to have comments inside JSON code.
nexpaq 0:6c56fb4bc5f0 29
nexpaq 0:6c56fb4bc5f0 30 Let’s see some example and let's try to configure small "test farm" with three devices connected to your host computer. In this example no peripherals (like SD card or EEPROM) are connected to our Mbed boards. We will use three platforms in this example:
nexpaq 0:6c56fb4bc5f0 31 * [NXP LPC1768](https://mbed.org/platforms/mbed-LPC1768) board.
nexpaq 0:6c56fb4bc5f0 32 * \[Freescale KL25Z](https://mbed.org/platforms/KL25Z) board and
nexpaq 0:6c56fb4bc5f0 33 * [STMicro Nucleo F103RB](https://mbed.org/platforms/ST-Nucleo-F103RB) board.
nexpaq 0:6c56fb4bc5f0 34 After connecting boards to our host machine (PC) we can check which serial ports and disks they occupy. For our example let's assume that:
nexpaq 0:6c56fb4bc5f0 35 * ```LPC1768``` serial port is on ```COM4``` and disk drive is ```J:```.
nexpaq 0:6c56fb4bc5f0 36 * ```KL25Z``` serial port is on ```COM39``` and disk drive is ```E:```.
nexpaq 0:6c56fb4bc5f0 37 * ```NUCLEO_F103RB``` serial port is on ```COM11``` and disk drive is ```I:```.
nexpaq 0:6c56fb4bc5f0 38 If you are working under Linux your port and disk could look like /dev/ttyACM5 and /media/usb5.
nexpaq 0:6c56fb4bc5f0 39
nexpaq 0:6c56fb4bc5f0 40 This information is needed to create ```muts_all.json``` configuration file. You can create it in ```mbed/tools/``` directory:
nexpaq 0:6c56fb4bc5f0 41 ```
nexpaq 0:6c56fb4bc5f0 42 $ touch muts_all.json
nexpaq 0:6c56fb4bc5f0 43 ```
nexpaq 0:6c56fb4bc5f0 44
nexpaq 0:6c56fb4bc5f0 45 Its name will be passed to ```singletest.py``` script after ```-M``` (MUTs specification) switch. Let’s see how this file's content would look like in our example below:
nexpaq 0:6c56fb4bc5f0 46 ```json
nexpaq 0:6c56fb4bc5f0 47 {
nexpaq 0:6c56fb4bc5f0 48 "1" : {"mcu": "LPC1768",
nexpaq 0:6c56fb4bc5f0 49 "port":"COM4",
nexpaq 0:6c56fb4bc5f0 50 "disk":"J:\\",
nexpaq 0:6c56fb4bc5f0 51 "peripherals": []
nexpaq 0:6c56fb4bc5f0 52 },
nexpaq 0:6c56fb4bc5f0 53
nexpaq 0:6c56fb4bc5f0 54 "2" : {"mcu": "KL25Z",
nexpaq 0:6c56fb4bc5f0 55 "port":"COM39",
nexpaq 0:6c56fb4bc5f0 56 "disk":"E:\\",
nexpaq 0:6c56fb4bc5f0 57 "peripherals": []
nexpaq 0:6c56fb4bc5f0 58 },
nexpaq 0:6c56fb4bc5f0 59
nexpaq 0:6c56fb4bc5f0 60 "3" : {"mcu": "NUCLEO_F103RB",
nexpaq 0:6c56fb4bc5f0 61 "port":"COM11",
nexpaq 0:6c56fb4bc5f0 62 "disk":"I:\\",
nexpaq 0:6c56fb4bc5f0 63 "peripherals": []
nexpaq 0:6c56fb4bc5f0 64 }
nexpaq 0:6c56fb4bc5f0 65 }
nexpaq 0:6c56fb4bc5f0 66 ```
nexpaq 0:6c56fb4bc5f0 67
nexpaq 0:6c56fb4bc5f0 68 Note: We will leave field ```peripherals``` empty for the sake of this example. We will explain it later. All you need to do now is to properly fill fields ```mcu```, ```port``` and ```disk```.
nexpaq 0:6c56fb4bc5f0 69
nexpaq 0:6c56fb4bc5f0 70 Note: Please make sure files muts_all.json and test_spec.json are in tools/ directory. We will assume in this example they are.
nexpaq 0:6c56fb4bc5f0 71 Where to find ```mcu``` names? You can use option ```-S``` of ```build.py``` script (in ```mbed/tools/``` directory) to check all supported off-line MCUs names.
nexpaq 0:6c56fb4bc5f0 72
nexpaq 0:6c56fb4bc5f0 73 Note: If you update mbed device firmware or even disconnect / reconnect mbed device you may find that serial port / disk configuration changed. You need to update configuration file accordingly or you will face connection problems and obviously tests will not run.
nexpaq 0:6c56fb4bc5f0 74
nexpaq 0:6c56fb4bc5f0 75 #### Peripherals testing
nexpaq 0:6c56fb4bc5f0 76 When using MUTs configuration file (switch ```-M```) you can define in MUTs JSON file peripherals connected to your device:
nexpaq 0:6c56fb4bc5f0 77 ```json
nexpaq 0:6c56fb4bc5f0 78 {
nexpaq 0:6c56fb4bc5f0 79 "1" : {"mcu" : "KL25Z",
nexpaq 0:6c56fb4bc5f0 80 "port" : "COM39",
nexpaq 0:6c56fb4bc5f0 81 "disk" : "E:\\",
nexpaq 0:6c56fb4bc5f0 82 "peripherals" : ["SD", "24LC256"]}
nexpaq 0:6c56fb4bc5f0 83 }
nexpaq 0:6c56fb4bc5f0 84 ```
nexpaq 0:6c56fb4bc5f0 85 You can force test suite to run only common tests (switch ```-C```) or only peripheral tests (switch ```-P```).
nexpaq 0:6c56fb4bc5f0 86 ```
nexpaq 0:6c56fb4bc5f0 87 $ python singletest.py -i test_spec.json -M muts_all.json -C
nexpaq 0:6c56fb4bc5f0 88 ```
nexpaq 0:6c56fb4bc5f0 89 will not include tests for SD card and EEPROM 24LC256.
nexpaq 0:6c56fb4bc5f0 90 ```
nexpaq 0:6c56fb4bc5f0 91 $ python singletest.py -i test_spec.json -M muts_all.json -P
nexpaq 0:6c56fb4bc5f0 92 ```
nexpaq 0:6c56fb4bc5f0 93 will only run tests bind to SD card and EEPROM 24LC256.
nexpaq 0:6c56fb4bc5f0 94
nexpaq 0:6c56fb4bc5f0 95 Note: option ```-P``` is useful for example in cases when you have same platform and different shields you want to test. No need to test common part all the time (timers, RTC, RTOS etc.). You can force to test peripherals only on some devices and for example only common tests on other devices.
nexpaq 0:6c56fb4bc5f0 96
nexpaq 0:6c56fb4bc5f0 97 #### Additional MUTs configuration file settings
nexpaq 0:6c56fb4bc5f0 98 You can add extra information to each MUT configuration. In particular you can specify which flashing (binary copy method) should be used, how to reset target and for example set reset timeout (used to delay test execution just after reset).
nexpaq 0:6c56fb4bc5f0 99
nexpaq 0:6c56fb4bc5f0 100 muts_all.json:
nexpaq 0:6c56fb4bc5f0 101 ```json
nexpaq 0:6c56fb4bc5f0 102 {
nexpaq 0:6c56fb4bc5f0 103 "1" : {"mcu" : "LPC1768",
nexpaq 0:6c56fb4bc5f0 104 "port" : "COM77",
nexpaq 0:6c56fb4bc5f0 105 "disk" : "G:\\",
nexpaq 0:6c56fb4bc5f0 106 "peripherals" : ["TMP102", "digital_loop", "port_loop", "analog_loop", "SD"]},
nexpaq 0:6c56fb4bc5f0 107
nexpaq 0:6c56fb4bc5f0 108 "2" : {"mcu" : "KL25Z",
nexpaq 0:6c56fb4bc5f0 109 "port" : "COM89",
nexpaq 0:6c56fb4bc5f0 110 "disk" : "F:\\",
nexpaq 0:6c56fb4bc5f0 111 "peripherals" : ["SD", "24LC256", "KL25Z"],
nexpaq 0:6c56fb4bc5f0 112 "copy_method" : "copy",
nexpaq 0:6c56fb4bc5f0 113 "reset_type" : "default",
nexpaq 0:6c56fb4bc5f0 114 "reset_tout" : "2"},
nexpaq 0:6c56fb4bc5f0 115
nexpaq 0:6c56fb4bc5f0 116 "3" : {"mcu" : "LPC11U24",
nexpaq 0:6c56fb4bc5f0 117 "port" : "COM76",
nexpaq 0:6c56fb4bc5f0 118 "disk" : "E:\\",
nexpaq 0:6c56fb4bc5f0 119 "peripherals" : []}
nexpaq 0:6c56fb4bc5f0 120 }
nexpaq 0:6c56fb4bc5f0 121 ```
nexpaq 0:6c56fb4bc5f0 122 Please note that for MUT no. 2 few extra parameters were defined: ```copy_method```, ```reset_type``` and ```reset_tout```. Using this extra options you can tell test suite more about MUT you are using. This will allow you to be more flexible in terms of how you configure and use your MUTs.
nexpaq 0:6c56fb4bc5f0 123
nexpaq 0:6c56fb4bc5f0 124 * ```copy_method``` - STRING - tells test suite which binary copy method should be used.
nexpaq 0:6c56fb4bc5f0 125 You may notice that ```singletest.py``` command line help contains description about:
nexpaq 0:6c56fb4bc5f0 126 * Option ```-c``` (in MUTs file called ```copy_method```) with available copy methods supported by test suite plugin system.
nexpaq 0:6c56fb4bc5f0 127 * Option ```-r``` (in MUTs file called reset_type) with available reset methods supported by test suite plugin system.
nexpaq 0:6c56fb4bc5f0 128 * ```reset_type``` - STRING - some boards may require special reset handling, for example vendor specific command must be executed to reset device.
nexpaq 0:6c56fb4bc5f0 129 * ```reset_tout``` - INTEGER - extra timeout just after device is reseted. May be used to wait for few seconds so device may finish booting, flashing data internally etc.
nexpaq 0:6c56fb4bc5f0 130
nexpaq 0:6c56fb4bc5f0 131 Part of help listing for singletest.py:
nexpaq 0:6c56fb4bc5f0 132 ```
nexpaq 0:6c56fb4bc5f0 133 -c COPY_METHOD, --copy-method=COPY_METHOD
nexpaq 0:6c56fb4bc5f0 134 Select binary copy (flash) method. Default is Python's
nexpaq 0:6c56fb4bc5f0 135 shutil.copy() method. Plugin support: copy, cp,
nexpaq 0:6c56fb4bc5f0 136 default, eACommander, eACommander-usb, xcopy
nexpaq 0:6c56fb4bc5f0 137 -r MUT_RESET_TYPE, --reset-type=MUT_RESET_TYPE
nexpaq 0:6c56fb4bc5f0 138 Extra reset method used to reset MUT by host test
nexpaq 0:6c56fb4bc5f0 139 script. Plugin support: default, eACommander,
nexpaq 0:6c56fb4bc5f0 140 eACommander-usb
nexpaq 0:6c56fb4bc5f0 141 ```
nexpaq 0:6c56fb4bc5f0 142
nexpaq 0:6c56fb4bc5f0 143 ----
nexpaq 0:6c56fb4bc5f0 144
nexpaq 0:6c56fb4bc5f0 145 Now we've already defined how our devices are connected to our host PC. We can continue and define which of this MUTs will be tested and which compilers we will use to compile and build Mbed SDK and tests. To do so we need to create test specification file (let's call it ```test_spec.json```) and put inside our configuration file information about which MUTs we actually want to test. We will pass this file's name to ```singletest.py``` script using ```-i``` switch.
nexpaq 0:6c56fb4bc5f0 146
nexpaq 0:6c56fb4bc5f0 147 Below we can see how sample ```test_spec.json``` file content could look like. (I've included all possible toolchains, we will change it in a moment):
nexpaq 0:6c56fb4bc5f0 148 ```json
nexpaq 0:6c56fb4bc5f0 149 {
nexpaq 0:6c56fb4bc5f0 150 "targets": {
nexpaq 0:6c56fb4bc5f0 151 "LPC1768" : ["ARM", "uARM", "GCC_ARM", "GCC_CR", "IAR"],
nexpaq 0:6c56fb4bc5f0 152 "KL25Z" : ["ARM", "GCC_ARM"],
nexpaq 0:6c56fb4bc5f0 153 "NUCLEO_F103RB" : ["ARM", "uARM"]
nexpaq 0:6c56fb4bc5f0 154 }
nexpaq 0:6c56fb4bc5f0 155 }
nexpaq 0:6c56fb4bc5f0 156 ```
nexpaq 0:6c56fb4bc5f0 157 Above example configuration will force tests for LPC1768, KL25Z, NUCLEO_F103RB platforms and:
nexpaq 0:6c56fb4bc5f0 158
nexpaq 0:6c56fb4bc5f0 159 * Compilers: ```ARM```, ```uARM```, ```GCC_ARM```, ```GCC_CR``` and ```IAR``` will be used to compile tests for NXP's ```LPC1768```.
nexpaq 0:6c56fb4bc5f0 160 * Compilers: ```ARM``` and ```GCC_ARM``` will be used for Freescales' ```KL25Z``` platform.
nexpaq 0:6c56fb4bc5f0 161 * Compilers: ```ARM``` and ```uARM``` will be used for STMicro's ```NUCLEO_F103RB``` platform.
nexpaq 0:6c56fb4bc5f0 162
nexpaq 0:6c56fb4bc5f0 163 For our example purposes let's assume we only have Keil ARM compiler, so let's change configuration in ```test_spec.json``` file and reduce number of compiler to those we actually have:
nexpaq 0:6c56fb4bc5f0 164 ```json
nexpaq 0:6c56fb4bc5f0 165 {
nexpaq 0:6c56fb4bc5f0 166 "targets": {
nexpaq 0:6c56fb4bc5f0 167 "LPC1768" : ["ARM", "uARM"],
nexpaq 0:6c56fb4bc5f0 168 "KL25Z" : ["ARM"],
nexpaq 0:6c56fb4bc5f0 169 "NUCLEO_F103RB" : ["ARM", "uARM"]
nexpaq 0:6c56fb4bc5f0 170 }
nexpaq 0:6c56fb4bc5f0 171 }
nexpaq 0:6c56fb4bc5f0 172 ```
nexpaq 0:6c56fb4bc5f0 173 #### Run your tests
nexpaq 0:6c56fb4bc5f0 174
nexpaq 0:6c56fb4bc5f0 175 After you configure all your MUTs and compilers you are ready to run tests. Make sure your devices are connected and your configuration files reflect your current configuration (serial ports, devices). Go to tools directory in your mbed location.
nexpaq 0:6c56fb4bc5f0 176 ```
nexpaq 0:6c56fb4bc5f0 177 $ cd tools/
nexpaq 0:6c56fb4bc5f0 178 ```
nexpaq 0:6c56fb4bc5f0 179 and execute test suite script.
nexpaq 0:6c56fb4bc5f0 180 ```
nexpaq 0:6c56fb4bc5f0 181 $ python singletest.py -i test_spec.json -M muts_all.json
nexpaq 0:6c56fb4bc5f0 182 ```
nexpaq 0:6c56fb4bc5f0 183 To check your configuration before test execution please use ```--config``` switch:
nexpaq 0:6c56fb4bc5f0 184 ```
nexpaq 0:6c56fb4bc5f0 185 $ python singletest.py -i test_spec.json -M muts_all.json --config
nexpaq 0:6c56fb4bc5f0 186 MUTs configuration in m.json:
nexpaq 0:6c56fb4bc5f0 187 +-------+-------------+---------------+------+-------+
nexpaq 0:6c56fb4bc5f0 188 | index | peripherals | mcu | disk | port |
nexpaq 0:6c56fb4bc5f0 189 +-------+-------------+---------------+------+-------+
nexpaq 0:6c56fb4bc5f0 190 | 1 | | LPC1768 | J:\ | COM4 |
nexpaq 0:6c56fb4bc5f0 191 | 3 | | NUCLEO_F103RB | I:\ | COM11 |
nexpaq 0:6c56fb4bc5f0 192 | 2 | | KL25Z | E:\ | COM39 |
nexpaq 0:6c56fb4bc5f0 193 +-------+-------------+---------------+------+-------+
nexpaq 0:6c56fb4bc5f0 194
nexpaq 0:6c56fb4bc5f0 195 Test specification in t.json:
nexpaq 0:6c56fb4bc5f0 196 +---------------+-----+------+
nexpaq 0:6c56fb4bc5f0 197 | mcu | ARM | uARM |
nexpaq 0:6c56fb4bc5f0 198 +---------------+-----+------+
nexpaq 0:6c56fb4bc5f0 199 | NUCLEO_F103RB | Yes | Yes |
nexpaq 0:6c56fb4bc5f0 200 | KL25Z | Yes | - |
nexpaq 0:6c56fb4bc5f0 201 | LPC1768 | Yes | Yes |
nexpaq 0:6c56fb4bc5f0 202 +---------------+-----+------+
nexpaq 0:6c56fb4bc5f0 203 ```
nexpaq 0:6c56fb4bc5f0 204 It should help you localize basic problems with configuration files and toolchain configuration.
nexpaq 0:6c56fb4bc5f0 205 Note: Configurations with issues will be marked with ```*``` sign.
nexpaq 0:6c56fb4bc5f0 206
nexpaq 0:6c56fb4bc5f0 207 Having multiple configuration files allows you to manage your test scenarios in more flexible manner. You can:
nexpaq 0:6c56fb4bc5f0 208
nexpaq 0:6c56fb4bc5f0 209 * Set up all platforms and toolchains used during testing.
nexpaq 0:6c56fb4bc5f0 210 * Define (using script's ```-n``` switch) which tests you want to run during testing.
nexpaq 0:6c56fb4bc5f0 211 * Just run regression (all tests). Regression is default setting for test script.
nexpaq 0:6c56fb4bc5f0 212
nexpaq 0:6c56fb4bc5f0 213 You can also force ```singletest.py``` script to:
nexpaq 0:6c56fb4bc5f0 214 * Run only peripherals' tests (switch ```-P```) or
nexpaq 0:6c56fb4bc5f0 215 * Just skip peripherals' tests (switch ```-C```).
nexpaq 0:6c56fb4bc5f0 216 * Build mbed SDK, libraries and corresponding tests with multiple cores, just use ```-j X``` option where ```X``` is number of cores you want to use for compilation.
nexpaq 0:6c56fb4bc5f0 217 ```
nexpaq 0:6c56fb4bc5f0 218 $ python singletest.py -i test_spec.json -M muts_all.json -j 8
nexpaq 0:6c56fb4bc5f0 219 ```
nexpaq 0:6c56fb4bc5f0 220 * Print test cases console output using ```-V``` option.
nexpaq 0:6c56fb4bc5f0 221 * Only build mbed SDK, tests and dependant libraries with switch ```-O```:
nexpaq 0:6c56fb4bc5f0 222 ```
nexpaq 0:6c56fb4bc5f0 223 $ python singletest.py -i test_spec.json -M muts_all.json -j 8 -O
nexpaq 0:6c56fb4bc5f0 224 ```
nexpaq 0:6c56fb4bc5f0 225 * Execute each test case multiple times with ```--global-loops X``` option, where ```X``` number of repeats. Additionally use option ```-W``` to continue repeating test cases execution only if they continue to fail.
nexpaq 0:6c56fb4bc5f0 226 ```
nexpaq 0:6c56fb4bc5f0 227 $ python singletest.py -i test_spec.json -M muts_all.json --global-loops 3 -W
nexpaq 0:6c56fb4bc5f0 228 ```
nexpaq 0:6c56fb4bc5f0 229 * Option ```--loops``` can be used to overwrite global loop count and redefine loop count for particular tests. Define test loops as ```TEST_ID=X``` where ```X``` is integer and separate loops count definitions by comma if necessary. E.g. ```TEST_1=5,TEST_2=20,TEST_3=2```.
nexpaq 0:6c56fb4bc5f0 230 ```
nexpaq 0:6c56fb4bc5f0 231 $ python singletest.py -i test_spec.json -M muts_all.json RTOS_1=10,RTOS_2=5
nexpaq 0:6c56fb4bc5f0 232 ```
nexpaq 0:6c56fb4bc5f0 233 This will execute test ```RTOS_1``` ten (10) times and test ```RTOS_2``` five (5) times.
nexpaq 0:6c56fb4bc5f0 234 * Force non default copy method. Note that mbed platforms can be flashed with just binary drag&drop. We simply copy file onto mbed's disk and interface chip flashes target MCU with given binary. Force non standard (Python specific) copy method by using option ```-c COPY_METHOD``` where ```COPY_METHOD``` can be shell, command line copy command like: ```cp```, ```copy````, ```xcopy``` etc. Make sure those commands are available from command line!
nexpaq 0:6c56fb4bc5f0 235 ```
nexpaq 0:6c56fb4bc5f0 236 $ python singletest.py -i test_spec.json -M muts_all.json -c cp
nexpaq 0:6c56fb4bc5f0 237 ```
nexpaq 0:6c56fb4bc5f0 238 * Run only selected tests. You can select which tests should be executed when you run test suite. Use ```-n``` switch to define tests by their ids you want to execute. Use comma to separate test ids:
nexpaq 0:6c56fb4bc5f0 239 ```
nexpaq 0:6c56fb4bc5f0 240 $ python singletest.py -i test_spec.json -M muts_all.json -n RTOS_1,RTOS_2,RTOS_3,MBED_10,MBED_16,MBED_11
nexpaq 0:6c56fb4bc5f0 241 ```
nexpaq 0:6c56fb4bc5f0 242 * Set common output binary name for all tests. In some cases you would like to have the same name for all tests. You can use switch ```--firmware-name``` to specify (without extension) build script output binary name.
nexpaq 0:6c56fb4bc5f0 243 In below example we would like to have all test binaries called ```firmware.bin`` (or with other extension like .elf, .hex depending on target accepted format).
nexpaq 0:6c56fb4bc5f0 244 ```
nexpaq 0:6c56fb4bc5f0 245 $ python singletest.py -i test_spec.json -M muts_all.json --firmware-name firmware
nexpaq 0:6c56fb4bc5f0 246 ```
nexpaq 0:6c56fb4bc5f0 247 * Where to find test list? Tests are defined in file ```tests.py``` in ```mbed/tools/``` directory. ```singletest.py``` uses test metadata in ```tests.py``` to resolve libraries dependencies and build tests for proper platforms and peripherals. Option ```-R``` can be used to get test names and direct path and test configuration.
nexpaq 0:6c56fb4bc5f0 248 ```
nexpaq 0:6c56fb4bc5f0 249 $ python singletest.py -R
nexpaq 0:6c56fb4bc5f0 250 +-------------+-----------+---------------------------------------+--------------+-------------------+----------+--------------------------------------------------------+
nexpaq 0:6c56fb4bc5f0 251 | id | automated | description | peripherals | host_test | duration | source_dir |
nexpaq 0:6c56fb4bc5f0 252 +-------------+-----------+---------------------------------------+--------------+-------------------+----------+--------------------------------------------------------+
nexpaq 0:6c56fb4bc5f0 253 | MBED_1 | False | I2C SRF08 | SRF08 | host_test | 10 | C:\Work\mbed\libraries\tests\mbed\i2c_SRF08 |
nexpaq 0:6c56fb4bc5f0 254 | MBED_10 | True | Hello World | - | host_test | 10 | C:\Work\mbed\libraries\tests\mbed\hello |
nexpaq 0:6c56fb4bc5f0 255 | MBED_11 | True | Ticker Int | - | host_test | 20 | C:\Work\mbed\libraries\tests\mbed\ticker |
nexpaq 0:6c56fb4bc5f0 256 | MBED_12 | True | C++ | - | host_test | 10 | C:\Work\mbed\libraries\tests\mbed\cpp |
nexpaq 0:6c56fb4bc5f0 257 | MBED_13 | False | Heap & Stack | - | host_test | 10 | C:\Work\mbed\libraries\tests\mbed\heap_and_stack |
nexpaq 0:6c56fb4bc5f0 258 | MBED_14 | False | Serial Interrupt | - | host_test | 10 | C:\Work\mbed\libraries\tests\mbed\serial_interrupt |
nexpaq 0:6c56fb4bc5f0 259 | MBED_15 | False | RPC | - | host_test | 10 | C:\Work\mbed\libraries\tests\mbed\rpc |
nexpaq 0:6c56fb4bc5f0 260 | MBED_16 | True | RTC | - | host_test | 15 | C:\Work\mbed\libraries\tests\mbed\rtc |
nexpaq 0:6c56fb4bc5f0 261 | MBED_17 | False | Serial Interrupt 2 | - | host_test | 10 | C:\Work\mbed\libraries\tests\mbed\serial_interrupt_2 |
nexpaq 0:6c56fb4bc5f0 262 | MBED_18 | False | Local FS Directory | - | host_test | 10 | C:\Work\mbed\libraries\tests\mbed\dir |
nexpaq 0:6c56fb4bc5f0 263 ...
nexpaq 0:6c56fb4bc5f0 264 ```
nexpaq 0:6c56fb4bc5f0 265 Note: you can filter tests by ```id``` column, just use ```-f``` option and give test name or regular expression:
nexpaq 0:6c56fb4bc5f0 266 ```
nexpaq 0:6c56fb4bc5f0 267 $ python singletest.py -R -f RTOS
nexpaq 0:6c56fb4bc5f0 268 +--------------+-----------+-------------------------+-------------+-----------+----------+---------------------------------------------------+
nexpaq 0:6c56fb4bc5f0 269 | id | automated | description | peripherals | host_test | duration | source_dir |
nexpaq 0:6c56fb4bc5f0 270 +--------------+-----------+-------------------------+-------------+-----------+----------+---------------------------------------------------+
nexpaq 0:6c56fb4bc5f0 271 | CMSIS_RTOS_1 | False | Basic | - | host_test | 10 | C:\Work\mbed\libraries\tests\rtos\cmsis\basic |
nexpaq 0:6c56fb4bc5f0 272 | CMSIS_RTOS_2 | False | Mutex | - | host_test | 20 | C:\Work\mbed\libraries\tests\rtos\cmsis\mutex |
nexpaq 0:6c56fb4bc5f0 273 | CMSIS_RTOS_3 | False | Semaphore | - | host_test | 20 | C:\Work\mbed\libraries\tests\rtos\cmsis\semaphore |
nexpaq 0:6c56fb4bc5f0 274 | CMSIS_RTOS_4 | False | Signals | - | host_test | 10 | C:\Work\mbed\libraries\tests\rtos\cmsis\signals |
nexpaq 0:6c56fb4bc5f0 275 | CMSIS_RTOS_5 | False | Queue | - | host_test | 20 | C:\Work\mbed\libraries\tests\rtos\cmsis\queue |
nexpaq 0:6c56fb4bc5f0 276 | CMSIS_RTOS_6 | False | Mail | - | host_test | 20 | C:\Work\mbed\libraries\tests\rtos\cmsis\mail |
nexpaq 0:6c56fb4bc5f0 277 | CMSIS_RTOS_7 | False | Timer | - | host_test | 10 | C:\Work\mbed\libraries\tests\rtos\cmsis\timer |
nexpaq 0:6c56fb4bc5f0 278 | CMSIS_RTOS_8 | False | ISR | - | host_test | 10 | C:\Work\mbed\libraries\tests\rtos\cmsis\isr |
nexpaq 0:6c56fb4bc5f0 279 | RTOS_1 | True | Basic thread | - | host_test | 15 | C:\Work\mbed\libraries\tests\rtos\mbed\basic |
nexpaq 0:6c56fb4bc5f0 280 | RTOS_2 | True | Mutex resource lock | - | host_test | 20 | C:\Work\mbed\libraries\tests\rtos\mbed\mutex |
nexpaq 0:6c56fb4bc5f0 281 | RTOS_3 | True | Semaphore resource lock | - | host_test | 20 | C:\Work\mbed\libraries\tests\rtos\mbed\semaphore |
nexpaq 0:6c56fb4bc5f0 282 | RTOS_4 | True | Signals messaging | - | host_test | 10 | C:\Work\mbed\libraries\tests\rtos\mbed\signals |
nexpaq 0:6c56fb4bc5f0 283 | RTOS_5 | True | Queue messaging | - | host_test | 10 | C:\Work\mbed\libraries\tests\rtos\mbed\queue |
nexpaq 0:6c56fb4bc5f0 284 | RTOS_6 | True | Mail messaging | - | host_test | 10 | C:\Work\mbed\libraries\tests\rtos\mbed\mail |
nexpaq 0:6c56fb4bc5f0 285 | RTOS_7 | True | Timer | - | host_test | 15 | C:\Work\mbed\libraries\tests\rtos\mbed\timer |
nexpaq 0:6c56fb4bc5f0 286 | RTOS_8 | True | ISR (Queue) | - | host_test | 10 | C:\Work\mbed\libraries\tests\rtos\mbed\isr |
nexpaq 0:6c56fb4bc5f0 287 | RTOS_9 | True | SD File write-read | SD | host_test | 10 | C:\Work\mbed\libraries\tests\rtos\mbed\file |
nexpaq 0:6c56fb4bc5f0 288 +--------------+-----------+-------------------------+-------------+-----------+----------+---------------------------------------------------+
nexpaq 0:6c56fb4bc5f0 289 ```
nexpaq 0:6c56fb4bc5f0 290
nexpaq 0:6c56fb4bc5f0 291 * Shuffle your tests. We strongly encourage you to shuffle your test order each time you execute test suite script.
nexpaq 0:6c56fb4bc5f0 292 Rationale: It is probable that tests executed in one particular order will pass and in other will fail. To shuffle your tests’ order please use option ```-u``` (or ```--shuffle```):
nexpaq 0:6c56fb4bc5f0 293 ```
nexpaq 0:6c56fb4bc5f0 294 $ python singletest.py -i test_spec.json -M muts_all.json --shuffle
nexpaq 0:6c56fb4bc5f0 295 ```
nexpaq 0:6c56fb4bc5f0 296 Above command will force test script to randomly generate shuffle seed and shuffle test order execution. Note: Shuffle seed is float in ```[0.0, 1.0)```.
nexpaq 0:6c56fb4bc5f0 297
nexpaq 0:6c56fb4bc5f0 298 You can always recreate particular test order by forcing shuffle (```-u``` or ```--shuffle``` switch) and passing shuffle seed back to test suite using ```--shuffle-seed``` switch:
nexpaq 0:6c56fb4bc5f0 299 ```
nexpaq 0:6c56fb4bc5f0 300 $ python singletest.py -i test_spec.json -M muts_all.json --shuffle --shuffle-seed 0.4041028336
nexpaq 0:6c56fb4bc5f0 301 ```
nexpaq 0:6c56fb4bc5f0 302 Note: You can also supply your own randomly generated shuffle seed to drive particular test execution order scenarios. Just make sure shuffle seed is float in ```[0.0, 1.0)```.
nexpaq 0:6c56fb4bc5f0 303 You can find test shuffle seed in test summary:
nexpaq 0:6c56fb4bc5f0 304 ```
nexpaq 0:6c56fb4bc5f0 305 ...
nexpaq 0:6c56fb4bc5f0 306 | OK | LPC1768 | ARM | MBED_A9 | Serial Echo at 115200 | 2.84 | 10 | 1/1 |
nexpaq 0:6c56fb4bc5f0 307 +--------+---------+-----------+-----------+-----------------------------+--------------------+---------------+-------+
nexpaq 0:6c56fb4bc5f0 308 Result: 1 FAIL / 22 OK
nexpaq 0:6c56fb4bc5f0 309 Shuffle Seed: 0.4041028336
nexpaq 0:6c56fb4bc5f0 310
nexpaq 0:6c56fb4bc5f0 311 Completed in 234.85 sec
nexpaq 0:6c56fb4bc5f0 312 ```
nexpaq 0:6c56fb4bc5f0 313
nexpaq 0:6c56fb4bc5f0 314 ### Example of device configuration (one device connected to host computer)
nexpaq 0:6c56fb4bc5f0 315
nexpaq 0:6c56fb4bc5f0 316 This example will show you how to configure single device, run general tests or only peripheral tests. We will also show some real test result examples.
nexpaq 0:6c56fb4bc5f0 317
nexpaq 0:6c56fb4bc5f0 318 1. We will test only one board STMIcro Nucleo ```F334R8``` board connected to our PC (port ```COM46``` and disk is ```E:```).
nexpaq 0:6c56fb4bc5f0 319 2. We will also connect EEPROM ```24LC256``` to SDA, SCL pins of our Nucleo board and define 24LC256 peripheral to make sure our test suite will run all available tests for ```24LC256```.
nexpaq 0:6c56fb4bc5f0 320
nexpaq 0:6c56fb4bc5f0 321 Let's configure our one MUT and set uARM as the only compiler we will use to compiler Mbed SDK and tests.
nexpaq 0:6c56fb4bc5f0 322 We also need to create two configuration files ```muts_all.json``` and ```test_spec.json``` to pass our small testbed configuration to test script.
nexpaq 0:6c56fb4bc5f0 323
nexpaq 0:6c56fb4bc5f0 324 muts_all.json:
nexpaq 0:6c56fb4bc5f0 325 ```json
nexpaq 0:6c56fb4bc5f0 326 {
nexpaq 0:6c56fb4bc5f0 327 "1" : {
nexpaq 0:6c56fb4bc5f0 328 "mcu": "NUCLEO_F334R8",
nexpaq 0:6c56fb4bc5f0 329 "port":"COM46",
nexpaq 0:6c56fb4bc5f0 330 "disk":"E:\\",
nexpaq 0:6c56fb4bc5f0 331 "peripherals": ["24LC256"]
nexpaq 0:6c56fb4bc5f0 332 }
nexpaq 0:6c56fb4bc5f0 333 }
nexpaq 0:6c56fb4bc5f0 334 ```
nexpaq 0:6c56fb4bc5f0 335 Note: By defining ```"peripherals": ["24LC256"]``` we are passing to test suite information that this particular board has EEPROM 24LC256 connected to our board.
nexpaq 0:6c56fb4bc5f0 336
nexpaq 0:6c56fb4bc5f0 337 test_spec.json:
nexpaq 0:6c56fb4bc5f0 338 ```json
nexpaq 0:6c56fb4bc5f0 339 {
nexpaq 0:6c56fb4bc5f0 340 "targets": {
nexpaq 0:6c56fb4bc5f0 341 "NUCLEO_F334R8" : ["uARM"]
nexpaq 0:6c56fb4bc5f0 342 }
nexpaq 0:6c56fb4bc5f0 343 }
nexpaq 0:6c56fb4bc5f0 344 ```
nexpaq 0:6c56fb4bc5f0 345 Note:
nexpaq 0:6c56fb4bc5f0 346 * Please make sure device is connected before we will start running tests.
nexpaq 0:6c56fb4bc5f0 347 * Please make sure files ```muts_all.json``` and ```test_spec.json``` are in ```mbed/tools/``` directory.
nexpaq 0:6c56fb4bc5f0 348 Now you can call test suite and execute tests:
nexpaq 0:6c56fb4bc5f0 349 ```
nexpaq 0:6c56fb4bc5f0 350 $ python singletest.py -i test_spec.json -M muts_all.json
nexpaq 0:6c56fb4bc5f0 351 ...
nexpaq 0:6c56fb4bc5f0 352 Test summary:
nexpaq 0:6c56fb4bc5f0 353 +--------+---------------+-----------+-----------+---------------------------------+--------------------+---------------+
nexpaq 0:6c56fb4bc5f0 354 | Result | Target | Toolchain | Test ID | Test Description | Elapsed Time (sec) | Timeout (sec) |
nexpaq 0:6c56fb4bc5f0 355 +--------+---------------+-----------+-----------+---------------------------------+--------------------+---------------+
nexpaq 0:6c56fb4bc5f0 356 | OK | NUCLEO_F334R8 | uARM | MBED_A25 | I2C EEPROM line read/write test | 12.41 | 15 |
nexpaq 0:6c56fb4bc5f0 357 | OK | NUCLEO_F334R8 | uARM | MBED_A1 | Basic | 3.42 | 10 |
nexpaq 0:6c56fb4bc5f0 358 | OK | NUCLEO_F334R8 | uARM | EXAMPLE_1 | /dev/null | 3.42 | 10 |
nexpaq 0:6c56fb4bc5f0 359 | OK | NUCLEO_F334R8 | uARM | MBED_24 | Timeout Int us | 11.47 | 15 |
nexpaq 0:6c56fb4bc5f0 360 | OK | NUCLEO_F334R8 | uARM | MBED_25 | Time us | 11.43 | 15 |
nexpaq 0:6c56fb4bc5f0 361 | OK | NUCLEO_F334R8 | uARM | MBED_26 | Integer constant division | 3.37 | 10 |
nexpaq 0:6c56fb4bc5f0 362 | OK | NUCLEO_F334R8 | uARM | MBED_23 | Ticker Int us | 12.43 | 15 |
nexpaq 0:6c56fb4bc5f0 363 | OK | NUCLEO_F334R8 | uARM | MBED_A19 | I2C EEPROM read/write test | 11.42 | 15 |
nexpaq 0:6c56fb4bc5f0 364 | OK | NUCLEO_F334R8 | uARM | MBED_11 | Ticker Int | 12.43 | 20 |
nexpaq 0:6c56fb4bc5f0 365 | OK | NUCLEO_F334R8 | uARM | MBED_10 | Hello World | 2.42 | 10 |
nexpaq 0:6c56fb4bc5f0 366 | OK | NUCLEO_F334R8 | uARM | MBED_12 | C++ | 3.42 | 10 |
nexpaq 0:6c56fb4bc5f0 367 | OK | NUCLEO_F334R8 | uARM | MBED_16 | RTC | 4.76 | 15 |
nexpaq 0:6c56fb4bc5f0 368 | UNDEF | NUCLEO_F334R8 | uARM | MBED_2 | stdio | 20.42 | 20 |
nexpaq 0:6c56fb4bc5f0 369 | UNDEF | NUCLEO_F334R8 | uARM | MBED_A9 | Serial Echo at 115200 | 10.37 | 10 |
nexpaq 0:6c56fb4bc5f0 370 +--------+---------------+-----------+-----------+---------------------------------+--------------------+---------------+
nexpaq 0:6c56fb4bc5f0 371 Result: 2 UNDEF / 12 OK
nexpaq 0:6c56fb4bc5f0 372
nexpaq 0:6c56fb4bc5f0 373 Completed in 160 sec
nexpaq 0:6c56fb4bc5f0 374 ```
nexpaq 0:6c56fb4bc5f0 375
nexpaq 0:6c56fb4bc5f0 376 If we want to get additional test summary with results in separate columns please use option ```-t```.
nexpaq 0:6c56fb4bc5f0 377 ```
nexpaq 0:6c56fb4bc5f0 378 $ python singletest.py -i test_spec.json -M muts_all.json -t
nexpaq 0:6c56fb4bc5f0 379 ...
nexpaq 0:6c56fb4bc5f0 380 Test summary:
nexpaq 0:6c56fb4bc5f0 381 +---------------+-----------+---------------------------------+-------+
nexpaq 0:6c56fb4bc5f0 382 | Target | Test ID | Test Description | uARM |
nexpaq 0:6c56fb4bc5f0 383 +---------------+-----------+---------------------------------+-------+
nexpaq 0:6c56fb4bc5f0 384 | NUCLEO_F334R8 | EXAMPLE_1 | /dev/null | OK |
nexpaq 0:6c56fb4bc5f0 385 | NUCLEO_F334R8 | MBED_10 | Hello World | OK |
nexpaq 0:6c56fb4bc5f0 386 | NUCLEO_F334R8 | MBED_11 | Ticker Int | OK |
nexpaq 0:6c56fb4bc5f0 387 | NUCLEO_F334R8 | MBED_12 | C++ | OK |
nexpaq 0:6c56fb4bc5f0 388 | NUCLEO_F334R8 | MBED_16 | RTC | OK |
nexpaq 0:6c56fb4bc5f0 389 | NUCLEO_F334R8 | MBED_2 | stdio | UNDEF |
nexpaq 0:6c56fb4bc5f0 390 | NUCLEO_F334R8 | MBED_23 | Ticker Int us | OK |
nexpaq 0:6c56fb4bc5f0 391 | NUCLEO_F334R8 | MBED_24 | Timeout Int us | OK |
nexpaq 0:6c56fb4bc5f0 392 | NUCLEO_F334R8 | MBED_25 | Time us | OK |
nexpaq 0:6c56fb4bc5f0 393 | NUCLEO_F334R8 | MBED_26 | Integer constant division | OK |
nexpaq 0:6c56fb4bc5f0 394 | NUCLEO_F334R8 | MBED_A1 | Basic | OK |
nexpaq 0:6c56fb4bc5f0 395 | NUCLEO_F334R8 | MBED_A19 | I2C EEPROM read/write test | OK |
nexpaq 0:6c56fb4bc5f0 396 | NUCLEO_F334R8 | MBED_A25 | I2C EEPROM line read/write test | OK |
nexpaq 0:6c56fb4bc5f0 397 | NUCLEO_F334R8 | MBED_A9 | Serial Echo at 115200 | UNDEF |
nexpaq 0:6c56fb4bc5f0 398 +---------------+-----------+---------------------------------+-------+
nexpaq 0:6c56fb4bc5f0 399 ```
nexpaq 0:6c56fb4bc5f0 400 ----
nexpaq 0:6c56fb4bc5f0 401 Please do not forget you can combine few options together to get result you want. For example you want to repeat few tests multiple number of times, shuffle test ids execution order and select only tests which are critical for you at this point. You can do it using switch -n, --global-loops with --loops and --shuffle:
nexpaq 0:6c56fb4bc5f0 402
nexpaq 0:6c56fb4bc5f0 403 Execute above command to:
nexpaq 0:6c56fb4bc5f0 404
nexpaq 0:6c56fb4bc5f0 405 * Run only tests: ```RTOS_1```, ```RTOS_2```, ```RTOS_3```, ```MBED_10```, ```MBED_16```, ```MBED_11```.
nexpaq 0:6c56fb4bc5f0 406 * Shuffle test execution order. Note tests in loops will not be shuffled.
nexpaq 0:6c56fb4bc5f0 407 * Set global loop count to 3 - each test will repeated 3 times.
nexpaq 0:6c56fb4bc5f0 408 * Overwrite global loop count (set above to 3) and:
nexpaq 0:6c56fb4bc5f0 409 * Force to loop test RTOS_1 to execute 3 times.
nexpaq 0:6c56fb4bc5f0 410 * Force to loop test RTOS_2 to execute 4 times.
nexpaq 0:6c56fb4bc5f0 411 * Force to loop test RTOS_3 to execute 5 times.
nexpaq 0:6c56fb4bc5f0 412 * Force to loop test MBED_11 to execute 5 times.
nexpaq 0:6c56fb4bc5f0 413
nexpaq 0:6c56fb4bc5f0 414 ```
nexpaq 0:6c56fb4bc5f0 415 $ python singletest.py -i test_spec.json -M muts_all.json -n RTOS_1,RTOS_2,RTOS_3,MBED_10,MBED_16,MBED_11 --shuffle --global-loops 3 --loops RTOS_1=3,RTOS_2=4,RTOS_3=5,MBED_11=5
nexpaq 0:6c56fb4bc5f0 416 ```
nexpaq 0:6c56fb4bc5f0 417
nexpaq 0:6c56fb4bc5f0 418 # CppUTest unit test library support
nexpaq 0:6c56fb4bc5f0 419 ## CppUTest in Mbed SDK testing introduction
nexpaq 0:6c56fb4bc5f0 420 [CppUTest](http://cpputest.github.io/) is a C / C++ based unit xUnit test framework for unit testing and for test-driving your code. It is written in C++ but is used in C and C++ projects and frequently used in embedded systems but it works for any C / C++ project.
nexpaq 0:6c56fb4bc5f0 421
nexpaq 0:6c56fb4bc5f0 422 Mbed SDK test suite supports writing tests using CppUTest. All you need to do it to provide CppUTest sources and includes with Mbed SDK port. This is already done for you so all you need to do it to get proper sources in your project directory.
nexpaq 0:6c56fb4bc5f0 423 CppUTest’s core design principles are:
nexpaq 0:6c56fb4bc5f0 424 * Simple in design and simple in use.
nexpaq 0:6c56fb4bc5f0 425 * Portable to old and new platforms.
nexpaq 0:6c56fb4bc5f0 426 * Build with Test-driven Development in mind.
nexpaq 0:6c56fb4bc5f0 427
nexpaq 0:6c56fb4bc5f0 428 ## From where you can get more help about CppUTest library and unit testing
nexpaq 0:6c56fb4bc5f0 429 • You can read [CppUTest manual](http://cpputest.github.io/manual.html)
nexpaq 0:6c56fb4bc5f0 430 * [CppUTest forum](https://groups.google.com/forum/?fromgroups#!forum/cpputest)
nexpaq 0:6c56fb4bc5f0 431 * [CppUTest on GitHub](https://github.com/cpputest/cpputest)
nexpaq 0:6c56fb4bc5f0 432 * Finally, if you think unit testing is new concept for you, you can have a grasp of it on Wikipedia pages about [unit testing](http://en.wikipedia.org/wiki/Unit_testing) and continue from there.
nexpaq 0:6c56fb4bc5f0 433
nexpaq 0:6c56fb4bc5f0 434 ## How to add CppUTest to your current Mbed SDK installation
nexpaq 0:6c56fb4bc5f0 435
nexpaq 0:6c56fb4bc5f0 436 ### Do I need CppUTest port for Mbed SDK?
nexpaq 0:6c56fb4bc5f0 437 Yes, you do. If you want to use CppUTest with Mbed SDK you need to have CppUTest version with ARMCC compiler (only ARM flavor for now) port and Mbed SDK console port (if you want to have output on serial port). All is already prepared by Mbed engineers and you can get it for example here: http://mbed.org/users/rgrover1/code/CppUTest/
nexpaq 0:6c56fb4bc5f0 438
nexpaq 0:6c56fb4bc5f0 439 ### Prerequisites
nexpaq 0:6c56fb4bc5f0 440 * Installed [git client](http://git-scm.com/downloads/).
nexpaq 0:6c56fb4bc5f0 441 * Installed [Mercurial client](http://mercurial.selenic.com/).
nexpaq 0:6c56fb4bc5f0 442
nexpaq 0:6c56fb4bc5f0 443 ### How / where to install
nexpaq 0:6c56fb4bc5f0 444 We want to create directory structure similar to one below:
nexpaq 0:6c56fb4bc5f0 445 ```
nexpaq 0:6c56fb4bc5f0 446 \your_project_directory
nexpaq 0:6c56fb4bc5f0 447 │
nexpaq 0:6c56fb4bc5f0 448 ├───cpputest
nexpaq 0:6c56fb4bc5f0 449 │ ├───include
nexpaq 0:6c56fb4bc5f0 450 │ └───src
nexpaq 0:6c56fb4bc5f0 451 └───mbed
nexpaq 0:6c56fb4bc5f0 452 ├───libraries
nexpaq 0:6c56fb4bc5f0 453 ├───travis
nexpaq 0:6c56fb4bc5f0 454 └───tools
nexpaq 0:6c56fb4bc5f0 455 ```
nexpaq 0:6c56fb4bc5f0 456
nexpaq 0:6c56fb4bc5f0 457 Please go to directory with your project. For example it could be c:\Projects\Project.
nexpaq 0:6c56fb4bc5f0 458 ```
nexpaq 0:6c56fb4bc5f0 459 $ cd c:\Projects\Project
nexpaq 0:6c56fb4bc5f0 460 ```
nexpaq 0:6c56fb4bc5f0 461 If your project directory already has your mbed SDK repository included just execute below command (Mercurial console client). It should download CppUTest with Mbed SDK port.
nexpaq 0:6c56fb4bc5f0 462 ```
nexpaq 0:6c56fb4bc5f0 463 $ hg clone https://mbed.org/users/rgrover1/code/cpputest/
nexpaq 0:6c56fb4bc5f0 464 ```
nexpaq 0:6c56fb4bc5f0 465
nexpaq 0:6c56fb4bc5f0 466 You should see something like this after you execute Mercurial clone command:
nexpaq 0:6c56fb4bc5f0 467 ```
nexpaq 0:6c56fb4bc5f0 468 $ hg clone https://mbed.org/users/rgrover1/code/cpputest/
nexpaq 0:6c56fb4bc5f0 469 destination directory: cpputest
nexpaq 0:6c56fb4bc5f0 470 requesting all changes
nexpaq 0:6c56fb4bc5f0 471 adding changesets
nexpaq 0:6c56fb4bc5f0 472 adding manifests
nexpaq 0:6c56fb4bc5f0 473 adding file changes
nexpaq 0:6c56fb4bc5f0 474 added 3 changesets with 69 changes to 42 files
nexpaq 0:6c56fb4bc5f0 475 updating to branch default
nexpaq 0:6c56fb4bc5f0 476 41 files updated, 0 files merged, 0 files removed, 0 files unresolved
nexpaq 0:6c56fb4bc5f0 477 ```
nexpaq 0:6c56fb4bc5f0 478
nexpaq 0:6c56fb4bc5f0 479 Confirm your project structure. It should look more or less like this:
nexpaq 0:6c56fb4bc5f0 480 ```
nexpaq 0:6c56fb4bc5f0 481 $ ls
nexpaq 0:6c56fb4bc5f0 482 cpputest mbed
nexpaq 0:6c56fb4bc5f0 483 ```
nexpaq 0:6c56fb4bc5f0 484 From now on CppUTest is in correct path. Each time you want to compile unit tests for CppUTest build script will always look for CppUTest library in the same directory where mbed library is.
nexpaq 0:6c56fb4bc5f0 485
nexpaq 0:6c56fb4bc5f0 486 ## New off-line mbed SDK project with CppUTest support
nexpaq 0:6c56fb4bc5f0 487
nexpaq 0:6c56fb4bc5f0 488 If you are creating new mbed SDK project and you want to use CppUTest with it you need to download both mbed SDK and CppUTest with mbed port to the same directory. You can do it like this:
nexpaq 0:6c56fb4bc5f0 489 ```
nexpaq 0:6c56fb4bc5f0 490 $ cd c:\Projects\Project
nexpaq 0:6c56fb4bc5f0 491 $ git clone https://github.com/mbedmicro/mbed.git
nexpaq 0:6c56fb4bc5f0 492 $ hg clone https://mbed.org/users/rgrover1/code/cpputest/
nexpaq 0:6c56fb4bc5f0 493 ```
nexpaq 0:6c56fb4bc5f0 494
nexpaq 0:6c56fb4bc5f0 495 After above three steps you should have proper directory structure. All you need to do now is to configure your ```mbed_settings.py``` in ```mbed``` directory. Please refer to mbed SDK build script documentation for details.
nexpaq 0:6c56fb4bc5f0 496
nexpaq 0:6c56fb4bc5f0 497 ## CppUTest with mbed port
nexpaq 0:6c56fb4bc5f0 498 To make sure you actualy have CppUTest library with mbed SDK port you can go to CppUTest ```armcc``` platform directory:
nexpaq 0:6c56fb4bc5f0 499 ```
nexpaq 0:6c56fb4bc5f0 500 $ cd c:/Projects/Project/cpputest/src/Platforms/armcc/
nexpaq 0:6c56fb4bc5f0 501 ```
nexpaq 0:6c56fb4bc5f0 502 And open file ```UtestPlatform.cpp```.
nexpaq 0:6c56fb4bc5f0 503
nexpaq 0:6c56fb4bc5f0 504 You should find part of code responsible for porting console on default serial port of the mbed device:
nexpaq 0:6c56fb4bc5f0 505 ```c++
nexpaq 0:6c56fb4bc5f0 506 #include "Serial.h"
nexpaq 0:6c56fb4bc5f0 507 using namespace mbed;
nexpaq 0:6c56fb4bc5f0 508
nexpaq 0:6c56fb4bc5f0 509 int PlatformSpecificPutchar(int c)
nexpaq 0:6c56fb4bc5f0 510 {
nexpaq 0:6c56fb4bc5f0 511 /* Please modify this block for test results to be reported as
nexpaq 0:6c56fb4bc5f0 512 * console output. The following is a sample implementation using a
nexpaq 0:6c56fb4bc5f0 513 * Serial object connected to the console. */
nexpaq 0:6c56fb4bc5f0 514 #define NEED_TEST_REPORT_AS_CONSOLE_OUTPUT 1
nexpaq 0:6c56fb4bc5f0 515 #if NEED_TEST_REPORT_AS_CONSOLE_OUTPUT
nexpaq 0:6c56fb4bc5f0 516 extern Serial console;
nexpaq 0:6c56fb4bc5f0 517
nexpaq 0:6c56fb4bc5f0 518 #define NEED_LINE_FEED_IN_ADDITION_TO_NEWLINE 1
nexpaq 0:6c56fb4bc5f0 519 #if NEED_LINE_FEED_IN_ADDITION_TO_NEWLINE
nexpaq 0:6c56fb4bc5f0 520 /* CppUTest emits \n line terminators in its reports; some terminals
nexpaq 0:6c56fb4bc5f0 521 * need the linefeed (\r) in addition. */
nexpaq 0:6c56fb4bc5f0 522 if (c == '\n') {
nexpaq 0:6c56fb4bc5f0 523 console.putc('\r');
nexpaq 0:6c56fb4bc5f0 524 }
nexpaq 0:6c56fb4bc5f0 525 #endif /* #if NEED_LINE_FEED_IN_ADDITION_TO_NEWLINE */
nexpaq 0:6c56fb4bc5f0 526
nexpaq 0:6c56fb4bc5f0 527 return (console.putc(c));
nexpaq 0:6c56fb4bc5f0 528 #else /* NEED_TEST_REPORT_AS_CONSOLE_OUTPUT */
nexpaq 0:6c56fb4bc5f0 529 return (0);
nexpaq 0:6c56fb4bc5f0 530 #endif /* NEED_TEST_REPORT_AS_CONSOLE_OUTPUT */
nexpaq 0:6c56fb4bc5f0 531 }
nexpaq 0:6c56fb4bc5f0 532 ```
nexpaq 0:6c56fb4bc5f0 533
nexpaq 0:6c56fb4bc5f0 534 You can find cpputest UT test runner main function in mbed sources: ```c:/Projects/Project/mbed/libraries/tests/utest/testrunner/testrunner.cpp```. Test runner code (in ```testrunner.cpp```) only defined console object and executes all unit tests:
nexpaq 0:6c56fb4bc5f0 535 ```c++
nexpaq 0:6c56fb4bc5f0 536 #include "CommandLineTestRunner.h"
nexpaq 0:6c56fb4bc5f0 537 #include <stdio.h>
nexpaq 0:6c56fb4bc5f0 538 #include "mbed.h"
nexpaq 0:6c56fb4bc5f0 539 #include "testrunner.h"
nexpaq 0:6c56fb4bc5f0 540 #include "test_env.h"
nexpaq 0:6c56fb4bc5f0 541
nexpaq 0:6c56fb4bc5f0 542 /**
nexpaq 0:6c56fb4bc5f0 543 Object 'mbed_cpputest_console' is used to show prints on console.
nexpaq 0:6c56fb4bc5f0 544 It is declared in \cpputest\src\Platforms\armcc\UtestPlatform.cpp
nexpaq 0:6c56fb4bc5f0 545 */
nexpaq 0:6c56fb4bc5f0 546 Serial mbed_cpputest_console(STDIO_UART_TX, STDIO_UART_RX);
nexpaq 0:6c56fb4bc5f0 547
nexpaq 0:6c56fb4bc5f0 548 int main(int ac, char** av) {
nexpaq 0:6c56fb4bc5f0 549 MBED_HOSTTEST_TIMEOUT(20);
nexpaq 0:6c56fb4bc5f0 550 MBED_HOSTTEST_SELECT(default_auto);
nexpaq 0:6c56fb4bc5f0 551 MBED_HOSTTEST_DESCRIPTION(Unit test);
nexpaq 0:6c56fb4bc5f0 552 MBED_HOSTTEST_START("UT");
nexpaq 0:6c56fb4bc5f0 553
nexpaq 0:6c56fb4bc5f0 554 unsigned failureCount = 0;
nexpaq 0:6c56fb4bc5f0 555 {
nexpaq 0:6c56fb4bc5f0 556 // Some compilers may not pass ac, av so we need to supply them ourselves
nexpaq 0:6c56fb4bc5f0 557 int ac = 2;
nexpaq 0:6c56fb4bc5f0 558 char* av[] = {__FILE__, "-v"};
nexpaq 0:6c56fb4bc5f0 559 failureCount = CommandLineTestRunner::RunAllTests(ac, av);
nexpaq 0:6c56fb4bc5f0 560 }
nexpaq 0:6c56fb4bc5f0 561
nexpaq 0:6c56fb4bc5f0 562 MBED_HOSTTEST_RESULT(failureCount == 0);
nexpaq 0:6c56fb4bc5f0 563 return failureCount;
nexpaq 0:6c56fb4bc5f0 564 }
nexpaq 0:6c56fb4bc5f0 565 ```
nexpaq 0:6c56fb4bc5f0 566
nexpaq 0:6c56fb4bc5f0 567 ## Unit test location
nexpaq 0:6c56fb4bc5f0 568 Unit tests source code is located in below directory: ```c:/Projects/Project/mbed/libraries/tests/utest/```
nexpaq 0:6c56fb4bc5f0 569
nexpaq 0:6c56fb4bc5f0 570 Each sub directory except testrunner contains compilable unit test source files with test groups and test cases. You can see utest structure below. Please note this is just example and in the future this directory will contain many sub directories with unit tests.
nexpaq 0:6c56fb4bc5f0 571 ```
nexpaq 0:6c56fb4bc5f0 572 $ c:\Projects\Project\mbed\libraries\tests\utest> tree
nexpaq 0:6c56fb4bc5f0 573 utest
nexpaq 0:6c56fb4bc5f0 574 ├───basic
nexpaq 0:6c56fb4bc5f0 575 ├───semihost_fs
nexpaq 0:6c56fb4bc5f0 576 └───testrunner
nexpaq 0:6c56fb4bc5f0 577 ```
nexpaq 0:6c56fb4bc5f0 578
nexpaq 0:6c56fb4bc5f0 579 ## Define unit tests in mbed SDK test suite structure
nexpaq 0:6c56fb4bc5f0 580 All tests defined in test suite are described in ```mbed/tools/tests.py``` file. This file stores data structure ```TESTS``` which is a list of simple structures describing each test. Below you can find example of ```TESTS``` structure which is configuring one of the unit tests.
nexpaq 0:6c56fb4bc5f0 581 ```
nexpaq 0:6c56fb4bc5f0 582 .
nexpaq 0:6c56fb4bc5f0 583 .
nexpaq 0:6c56fb4bc5f0 584 .
nexpaq 0:6c56fb4bc5f0 585 {
nexpaq 0:6c56fb4bc5f0 586 "id": "UT_2", "description": "Semihost file system",
nexpaq 0:6c56fb4bc5f0 587 "source_dir": join(TEST_DIR, "utest", "file"),
nexpaq 0:6c56fb4bc5f0 588 "dependencies": [MBED_LIBRARIES, TEST_MBED_LIB, CPPUTEST_LIBRARY],
nexpaq 0:6c56fb4bc5f0 589 "automated": False,
nexpaq 0:6c56fb4bc5f0 590 "mcu": ["LPC1768", "LPC2368", "LPC11U24"]
nexpaq 0:6c56fb4bc5f0 591 },
nexpaq 0:6c56fb4bc5f0 592 .
nexpaq 0:6c56fb4bc5f0 593 .
nexpaq 0:6c56fb4bc5f0 594 .
nexpaq 0:6c56fb4bc5f0 595 ```
nexpaq 0:6c56fb4bc5f0 596 Note: In dependency section we've added library ```CPPUTEST_LIBRARY``` which is pointing build script to CppUTest library with mbed port. This is a must for unit tests to be compiled with CppUTest library.
nexpaq 0:6c56fb4bc5f0 597
nexpaq 0:6c56fb4bc5f0 598 ### Tests are now divided into two types:
nexpaq 0:6c56fb4bc5f0 599 #### 'Hello world' tests
nexpaq 0:6c56fb4bc5f0 600 First type of test cases we call 'hello world' tests. They do not dependent on CppUTest library and are monolithic programs usually composed of one main function. You can find this tests in below example directories:
nexpaq 0:6c56fb4bc5f0 601
nexpaq 0:6c56fb4bc5f0 602 * ```mbed/libraries/tests/mbed/```
nexpaq 0:6c56fb4bc5f0 603 * ```mbed/libraries/tests/net/```
nexpaq 0:6c56fb4bc5f0 604 * ```mbed/libraries/tests/rtos/```
nexpaq 0:6c56fb4bc5f0 605 * ```mbed/libraries/tests/usb/```
nexpaq 0:6c56fb4bc5f0 606
nexpaq 0:6c56fb4bc5f0 607 Usually ‘hello world’ test cases are using ```test_env.cpp``` and ```test_env.h``` files which implement simple test framework used to communicate with host test and help test framework instrument your tests.
nexpaq 0:6c56fb4bc5f0 608
nexpaq 0:6c56fb4bc5f0 609 Below you can see listing of ```test_env.h``` file which contains simple macro definitions used to communicate (via serial port printouts) between test case (on hardware) and host test script (on host computer).
nexpaq 0:6c56fb4bc5f0 610 Each use case should print on console basic information like:
nexpaq 0:6c56fb4bc5f0 611 * Default test case timeout.
nexpaq 0:6c56fb4bc5f0 612 * Which host test should be used to supervise test case execution.
nexpaq 0:6c56fb4bc5f0 613 * Test description and test case ID (short identifier).
nexpaq 0:6c56fb4bc5f0 614
nexpaq 0:6c56fb4bc5f0 615 ```c++
nexpaq 0:6c56fb4bc5f0 616 .
nexpaq 0:6c56fb4bc5f0 617 .
nexpaq 0:6c56fb4bc5f0 618 .
nexpaq 0:6c56fb4bc5f0 619 // Test result related notification functions
nexpaq 0:6c56fb4bc5f0 620 void notify_start();
nexpaq 0:6c56fb4bc5f0 621 void notify_completion(bool success);
nexpaq 0:6c56fb4bc5f0 622 bool notify_completion_str(bool success, char* buffer);
nexpaq 0:6c56fb4bc5f0 623 void notify_performance_coefficient(const char* measurement_name, const int value);
nexpaq 0:6c56fb4bc5f0 624 void notify_performance_coefficient(const char* measurement_name, const unsigned int value);
nexpaq 0:6c56fb4bc5f0 625 void notify_performance_coefficient(const char* measurement_name, const double value);
nexpaq 0:6c56fb4bc5f0 626
nexpaq 0:6c56fb4bc5f0 627 // Host test auto-detection API
nexpaq 0:6c56fb4bc5f0 628 void notify_host_test_name(const char *host_test);
nexpaq 0:6c56fb4bc5f0 629 void notify_timeout(int timeout);
nexpaq 0:6c56fb4bc5f0 630 void notify_test_id(const char *test_id);
nexpaq 0:6c56fb4bc5f0 631 void notify_test_description(const char *description);
nexpaq 0:6c56fb4bc5f0 632
nexpaq 0:6c56fb4bc5f0 633 // Host test auto-detection API
nexpaq 0:6c56fb4bc5f0 634 #define MBED_HOSTTEST_START(TESTID) notify_test_id(TESTID); notify_start()
nexpaq 0:6c56fb4bc5f0 635 #define MBED_HOSTTEST_SELECT(NAME) notify_host_test_name(#NAME)
nexpaq 0:6c56fb4bc5f0 636 #define MBED_HOSTTEST_TIMEOUT(SECONDS) notify_timeout(SECONDS)
nexpaq 0:6c56fb4bc5f0 637 #define MBED_HOSTTEST_DESCRIPTION(DESC) notify_test_description(#DESC)
nexpaq 0:6c56fb4bc5f0 638 #define MBED_HOSTTEST_RESULT(RESULT) notify_completion(RESULT)
nexpaq 0:6c56fb4bc5f0 639
nexpaq 0:6c56fb4bc5f0 640 /**
nexpaq 0:6c56fb4bc5f0 641 Test auto-detection preamble example:
nexpaq 0:6c56fb4bc5f0 642 main() {
nexpaq 0:6c56fb4bc5f0 643 MBED_HOSTTEST_TIMEOUT(10);
nexpaq 0:6c56fb4bc5f0 644 MBED_HOSTTEST_SELECT( host_test );
nexpaq 0:6c56fb4bc5f0 645 MBED_HOSTTEST_DESCRIPTION(Hello World);
nexpaq 0:6c56fb4bc5f0 646 MBED_HOSTTEST_START("MBED_10");
nexpaq 0:6c56fb4bc5f0 647 // Proper 'host_test.py' should take over supervising of this test
nexpaq 0:6c56fb4bc5f0 648
nexpaq 0:6c56fb4bc5f0 649 // Test code
nexpaq 0:6c56fb4bc5f0 650 bool result = ...;
nexpaq 0:6c56fb4bc5f0 651
nexpaq 0:6c56fb4bc5f0 652 MBED_HOSTTEST_RESULT(result);
nexpaq 0:6c56fb4bc5f0 653 }
nexpaq 0:6c56fb4bc5f0 654 */
nexpaq 0:6c56fb4bc5f0 655 .
nexpaq 0:6c56fb4bc5f0 656 .
nexpaq 0:6c56fb4bc5f0 657 .
nexpaq 0:6c56fb4bc5f0 658 ```
nexpaq 0:6c56fb4bc5f0 659
nexpaq 0:6c56fb4bc5f0 660 Example of 'hello world' test:
nexpaq 0:6c56fb4bc5f0 661 ```c++
nexpaq 0:6c56fb4bc5f0 662 #include "mbed.h"
nexpaq 0:6c56fb4bc5f0 663 #include "test_env.h"
nexpaq 0:6c56fb4bc5f0 664
nexpaq 0:6c56fb4bc5f0 665 #define CUSTOM_TIME 1256729737
nexpaq 0:6c56fb4bc5f0 666
nexpaq 0:6c56fb4bc5f0 667 int main() {
nexpaq 0:6c56fb4bc5f0 668 MBED_HOSTTEST_TIMEOUT(20);
nexpaq 0:6c56fb4bc5f0 669 MBED_HOSTTEST_SELECT(rtc_auto);
nexpaq 0:6c56fb4bc5f0 670 MBED_HOSTTEST_DESCRIPTION(RTC);
nexpaq 0:6c56fb4bc5f0 671 MBED_HOSTTEST_START("MBED_16");
nexpaq 0:6c56fb4bc5f0 672
nexpaq 0:6c56fb4bc5f0 673 char buffer[32] = {0};
nexpaq 0:6c56fb4bc5f0 674 set_time(CUSTOM_TIME); // Set RTC time to Wed, 28 Oct 2009 11:35:37
nexpaq 0:6c56fb4bc5f0 675 while(1) {
nexpaq 0:6c56fb4bc5f0 676 time_t seconds = time(NULL);
nexpaq 0:6c56fb4bc5f0 677 strftime(buffer, 32, "%Y-%m-%d %H:%M:%S %p", localtime(&seconds));
nexpaq 0:6c56fb4bc5f0 678 printf("MBED: [%ld] [%s]\r\n", seconds, buffer);
nexpaq 0:6c56fb4bc5f0 679 wait(1);
nexpaq 0:6c56fb4bc5f0 680 }
nexpaq 0:6c56fb4bc5f0 681 }
nexpaq 0:6c56fb4bc5f0 682 ```
nexpaq 0:6c56fb4bc5f0 683
nexpaq 0:6c56fb4bc5f0 684 #### 'Unit test' test cases
nexpaq 0:6c56fb4bc5f0 685 Second group of tests are unit tests. They are using CppUTest library and require you to write ```TEST_GROUP```s and ```TEST```s in your test files. Test suite will add test runner sources to your test automatically so you can concentrate on writing tests.
nexpaq 0:6c56fb4bc5f0 686
nexpaq 0:6c56fb4bc5f0 687 Example of unit test:
nexpaq 0:6c56fb4bc5f0 688 ```c++
nexpaq 0:6c56fb4bc5f0 689 #include "TestHarness.h"
nexpaq 0:6c56fb4bc5f0 690 #include <utility>
nexpaq 0:6c56fb4bc5f0 691 #include "mbed.h"
nexpaq 0:6c56fb4bc5f0 692
nexpaq 0:6c56fb4bc5f0 693 TEST_GROUP(BusOut_mask)
nexpaq 0:6c56fb4bc5f0 694 {
nexpaq 0:6c56fb4bc5f0 695 };
nexpaq 0:6c56fb4bc5f0 696
nexpaq 0:6c56fb4bc5f0 697 TEST(BusOut_mask, led_1_2_3)
nexpaq 0:6c56fb4bc5f0 698 {
nexpaq 0:6c56fb4bc5f0 699 BusOut bus_data(LED1, LED2, LED3);
nexpaq 0:6c56fb4bc5f0 700 CHECK_EQUAL(0x07, bus_data.mask());
nexpaq 0:6c56fb4bc5f0 701 }
nexpaq 0:6c56fb4bc5f0 702
nexpaq 0:6c56fb4bc5f0 703 TEST(BusOut_mask, led_nc_nc_nc_nc)
nexpaq 0:6c56fb4bc5f0 704 {
nexpaq 0:6c56fb4bc5f0 705 BusOut bus_data(NC, NC, NC, NC);
nexpaq 0:6c56fb4bc5f0 706 CHECK_EQUAL(0x00, bus_data.mask());
nexpaq 0:6c56fb4bc5f0 707 }
nexpaq 0:6c56fb4bc5f0 708
nexpaq 0:6c56fb4bc5f0 709 TEST(BusOut_mask, led_1_2_3_nc_nc)
nexpaq 0:6c56fb4bc5f0 710 {
nexpaq 0:6c56fb4bc5f0 711 BusOut bus_data(LED1, LED2, LED3, NC, NC);
nexpaq 0:6c56fb4bc5f0 712 CHECK_EQUAL(0x07, bus_data.mask());
nexpaq 0:6c56fb4bc5f0 713 }
nexpaq 0:6c56fb4bc5f0 714
nexpaq 0:6c56fb4bc5f0 715 TEST(BusOut_mask, led_1_nc_2_nc_nc_3)
nexpaq 0:6c56fb4bc5f0 716 {
nexpaq 0:6c56fb4bc5f0 717 BusOut bus_data(LED1, NC, LED2, NC, NC, LED3);
nexpaq 0:6c56fb4bc5f0 718 CHECK_EQUAL(0x25, bus_data.mask());
nexpaq 0:6c56fb4bc5f0 719 }
nexpaq 0:6c56fb4bc5f0 720 ```
nexpaq 0:6c56fb4bc5f0 721
nexpaq 0:6c56fb4bc5f0 722 ## Example
nexpaq 0:6c56fb4bc5f0 723 In below example we will run two example unit tests that are now available. tests ```UT_1``` and ```UT_2``` are unit tests used for now only to check if mbed SDK works with CppUTest library and if tests are being executed. In future number of unit tests will increase, nothing is also stopping you from writing and executing your own unit tests!
nexpaq 0:6c56fb4bc5f0 724
nexpaq 0:6c56fb4bc5f0 725 ### Example configuration
nexpaq 0:6c56fb4bc5f0 726 By default unit tests ```UT_1``` and ```UT_2``` are not automated - simply test suite will ignore them. Also we do not want to create dependency to CppUTest library each time someone executes automation.
nexpaq 0:6c56fb4bc5f0 727
nexpaq 0:6c56fb4bc5f0 728 Note: To compile ```UT_1``` and ```UT_2``` tests CppUTest library described above, installation is needed and not all users wish to add UT libs to their project. Also new to mbed users may find it difficult. This is why unit testing is an extra feature active only after you deliberately install and enable needed components.
nexpaq 0:6c56fb4bc5f0 729
nexpaq 0:6c56fb4bc5f0 730 Bellow snippet shows how to modify 'automated' flag so test suite will consider unit tests ```UT_1``` and ```UT_2``` as part of "automated test portfolio". Just change flag 'automated' from ```False``` to ```True```.
nexpaq 0:6c56fb4bc5f0 731
nexpaq 0:6c56fb4bc5f0 732 ```tests.py``` listing related to ```UT_1``` and ```UT_2```:
nexpaq 0:6c56fb4bc5f0 733 ```python
nexpaq 0:6c56fb4bc5f0 734 .
nexpaq 0:6c56fb4bc5f0 735 .
nexpaq 0:6c56fb4bc5f0 736 .
nexpaq 0:6c56fb4bc5f0 737 # CPPUTEST Library provides Unit testing Framework
nexpaq 0:6c56fb4bc5f0 738 #
nexpaq 0:6c56fb4bc5f0 739 # To write TESTs and TEST_GROUPs please add CPPUTEST_LIBRARY to 'dependencies'
nexpaq 0:6c56fb4bc5f0 740 #
nexpaq 0:6c56fb4bc5f0 741 # This will also include:
nexpaq 0:6c56fb4bc5f0 742 # 1. test runner - main function with call to CommandLineTestRunner::RunAllTests(ac, av)
nexpaq 0:6c56fb4bc5f0 743 # 2. Serial console object to print test result on serial port console
nexpaq 0:6c56fb4bc5f0 744 #
nexpaq 0:6c56fb4bc5f0 745
nexpaq 0:6c56fb4bc5f0 746 # Unit testing with cpputest library
nexpaq 0:6c56fb4bc5f0 747 {
nexpaq 0:6c56fb4bc5f0 748 "id": "UT_1", "description": "Basic",
nexpaq 0:6c56fb4bc5f0 749 "source_dir": join(TEST_DIR, "utest", "basic"),
nexpaq 0:6c56fb4bc5f0 750 "dependencies": [MBED_LIBRARIES, TEST_MBED_LIB, CPPUTEST_LIBRARY],
nexpaq 0:6c56fb4bc5f0 751 "automated": True,
nexpaq 0:6c56fb4bc5f0 752 },
nexpaq 0:6c56fb4bc5f0 753 {
nexpaq 0:6c56fb4bc5f0 754 "id": "UT_2", "description": "Semihost file system",
nexpaq 0:6c56fb4bc5f0 755 "source_dir": join(TEST_DIR, "utest", "semihost_fs"),
nexpaq 0:6c56fb4bc5f0 756 "dependencies": [MBED_LIBRARIES, TEST_MBED_LIB, CPPUTEST_LIBRARY],
nexpaq 0:6c56fb4bc5f0 757 "automated": True,
nexpaq 0:6c56fb4bc5f0 758 "mcu": ["LPC1768", "LPC2368", "LPC11U24"]
nexpaq 0:6c56fb4bc5f0 759 },
nexpaq 0:6c56fb4bc5f0 760 .
nexpaq 0:6c56fb4bc5f0 761 .
nexpaq 0:6c56fb4bc5f0 762 .
nexpaq 0:6c56fb4bc5f0 763 ```
nexpaq 0:6c56fb4bc5f0 764
nexpaq 0:6c56fb4bc5f0 765 ### Execute tests
nexpaq 0:6c56fb4bc5f0 766 In my test I will use common [LPC1768](http://developer.mbed.org/platforms/mbed-LPC1768/) mbed-enabled board because unit test ```UT_2``` is checking semi-host functionality which is available on this board and handful of others.
nexpaq 0:6c56fb4bc5f0 767
nexpaq 0:6c56fb4bc5f0 768 Configure your ```test_spec.json``` and ```muts_all.json``` files (refer to test suite build script and automation description) and set mbed disk and serial port.
nexpaq 0:6c56fb4bc5f0 769
nexpaq 0:6c56fb4bc5f0 770 ```
nexpaq 0:6c56fb4bc5f0 771 $ singletest.py -i test_spec.json -M muts_all.json -n UT_1,UT_2 -V
nexpaq 0:6c56fb4bc5f0 772 Building library CMSIS (LPC1768, ARM)
nexpaq 0:6c56fb4bc5f0 773 Building library MBED (LPC1768, ARM)
nexpaq 0:6c56fb4bc5f0 774 Building library CPPUTEST (LPC1768, ARM)
nexpaq 0:6c56fb4bc5f0 775 Building project BASIC (LPC1768, ARM)
nexpaq 0:6c56fb4bc5f0 776 Executing 'python host_test.py -p COM77 -d E:\ -t 10'
nexpaq 0:6c56fb4bc5f0 777 Test::Output::Start
nexpaq 0:6c56fb4bc5f0 778 Host test instrumentation on port: "COM77" and disk: "E:\"
nexpaq 0:6c56fb4bc5f0 779 TEST(FirstTestGroup, FirstTest) - 0 ms
nexpaq 0:6c56fb4bc5f0 780
nexpaq 0:6c56fb4bc5f0 781 OK (1 tests, 1 ran, 3 checks, 0 ignored, 0 filtered out, 3 ms)
nexpaq 0:6c56fb4bc5f0 782
nexpaq 0:6c56fb4bc5f0 783 {{success}}
nexpaq 0:6c56fb4bc5f0 784 {{end}}
nexpaq 0:6c56fb4bc5f0 785 Test::Output::Finish
nexpaq 0:6c56fb4bc5f0 786 TargetTest::LPC1768::ARM::UT_1::Basic [OK] in 2.43 of 10 sec
nexpaq 0:6c56fb4bc5f0 787 Building library CPPUTEST (LPC1768, ARM)
nexpaq 0:6c56fb4bc5f0 788 Building project SEMIHOST_FS (LPC1768, ARM)
nexpaq 0:6c56fb4bc5f0 789 Executing 'python host_test.py -p COM77 -d E:\ -t 10'
nexpaq 0:6c56fb4bc5f0 790 Test::Output::Start
nexpaq 0:6c56fb4bc5f0 791 Host test instrumentation on port: "COM77" and disk: "E:\"
nexpaq 0:6c56fb4bc5f0 792 TEST(FirstTestGroup, FirstTest) - 9 ms
nexpaq 0:6c56fb4bc5f0 793
nexpaq 0:6c56fb4bc5f0 794 OK (1 tests, 1 ran, 10 checks, 0 ignored, 0 filtered out, 10 ms)
nexpaq 0:6c56fb4bc5f0 795
nexpaq 0:6c56fb4bc5f0 796 {{success}}
nexpaq 0:6c56fb4bc5f0 797 {{end}}
nexpaq 0:6c56fb4bc5f0 798 Test::Output::Finish
nexpaq 0:6c56fb4bc5f0 799 TargetTest::LPC1768::ARM::UT_2::Semihost file system [OK] in 2.43 of 10 sec
nexpaq 0:6c56fb4bc5f0 800 Test summary:
nexpaq 0:6c56fb4bc5f0 801 +--------+---------+-----------+---------+----------------------+--------------------+---------------+-------+
nexpaq 0:6c56fb4bc5f0 802 | Result | Target | Toolchain | Test ID | Test Description | Elapsed Time (sec) | Timeout (sec) | Loops |
nexpaq 0:6c56fb4bc5f0 803 +--------+---------+-----------+---------+----------------------+--------------------+---------------+-------+
nexpaq 0:6c56fb4bc5f0 804 | OK | LPC1768 | ARM | UT_1 | Basic | 2.43 | 10 | 1/1 |
nexpaq 0:6c56fb4bc5f0 805 | OK | LPC1768 | ARM | UT_2 | Semihost file system | 2.43 | 10 | 1/1 |
nexpaq 0:6c56fb4bc5f0 806 +--------+---------+-----------+---------+----------------------+--------------------+---------------+-------+
nexpaq 0:6c56fb4bc5f0 807 Result: 2 OK
nexpaq 0:6c56fb4bc5f0 808
nexpaq 0:6c56fb4bc5f0 809 Completed in 12.02 sec
nexpaq 0:6c56fb4bc5f0 810 ```
nexpaq 0:6c56fb4bc5f0 811
nexpaq 0:6c56fb4bc5f0 812 You can compile unit tests using various number of supported compilers, below just few examples with working solutions:
nexpaq 0:6c56fb4bc5f0 813 ```
nexpaq 0:6c56fb4bc5f0 814 Test summary:
nexpaq 0:6c56fb4bc5f0 815 +--------+---------+-----------+---------+----------------------+--------------------+---------------+-------+
nexpaq 0:6c56fb4bc5f0 816 | Result | Target | Toolchain | Test ID | Test Description | Elapsed Time (sec) | Timeout (sec) | Loops |
nexpaq 0:6c56fb4bc5f0 817 +--------+---------+-----------+---------+----------------------+--------------------+---------------+-------+
nexpaq 0:6c56fb4bc5f0 818 | OK | LPC1768 | ARM | UT_1 | Basic | 2.43 | 10 | 1/1 |
nexpaq 0:6c56fb4bc5f0 819 | OK | LPC1768 | ARM | UT_2 | Semihost file system | 2.43 | 10 | 1/1 |
nexpaq 0:6c56fb4bc5f0 820 | OK | LPC1768 | uARM | UT_1 | Basic | 2.43 | 10 | 1/1 |
nexpaq 0:6c56fb4bc5f0 821 | OK | LPC1768 | uARM | UT_2 | Semihost file system | 2.43 | 10 | 1/1 |
nexpaq 0:6c56fb4bc5f0 822 | OK | LPC1768 | GCC_ARM | UT_1 | Basic | 2.43 | 10 | 1/1 |
nexpaq 0:6c56fb4bc5f0 823 | OK | LPC1768 | GCC_ARM | UT_2 | Semihost file system | 2.43 | 10 | 1/1 |
nexpaq 0:6c56fb4bc5f0 824 | OK | LPC1768 | GCC_CR | UT_1 | Basic | 3.44 | 10 | 1/1 |
nexpaq 0:6c56fb4bc5f0 825 | OK | LPC1768 | GCC_CR | UT_2 | Semihost file system | 3.43 | 10 | 1/1 |
nexpaq 0:6c56fb4bc5f0 826 +--------+---------+-----------+---------+----------------------+--------------------+---------------+-------+
nexpaq 0:6c56fb4bc5f0 827 Result: 8 OK
nexpaq 0:6c56fb4bc5f0 828
nexpaq 0:6c56fb4bc5f0 829 Completed in 55.85 sec
nexpaq 0:6c56fb4bc5f0 830 ```