Magnificent7 / Hackathon

Includes library modifications to allow access to AIN_4 (AIN_0 / 5)

mbd_os/docs/TESTING.md@0:eafc3fd41f75, 2016-09-20 (annotated)

Committer:
bryantaylor
Date:
Tue Sep 20 21:26:12 2016 +0000
Revision:
0:eafc3fd41f75
hackathon

Who changed what in which revision?

UserRevisionLine numberNew contents of line
bryantaylor 0:eafc3fd41f75 1 # Mbed SDK automated test suite
bryantaylor 0:eafc3fd41f75 2 ## Introduction
bryantaylor 0:eafc3fd41f75 3
bryantaylor 0:eafc3fd41f75 4 Test suit allows users to run locally on their machines Mbed SDK’s tests included in Mbed SDK repository. It also allows users to create their own tests and for example add new tests to test set as they progress with their project. If test is generic enough it could be included into official Mbed SDK test pool just do it via normal pull request!
bryantaylor 0:eafc3fd41f75 5
bryantaylor 0:eafc3fd41f75 6 Each test is supervised by python script called “host test” which will at least Test suite is using build script API to compile and build test source together with required by test libraries like CMSIS, Mbed, Ethernet, USB etc.
bryantaylor 0:eafc3fd41f75 7
bryantaylor 0:eafc3fd41f75 8 ## What is host test?
bryantaylor 0:eafc3fd41f75 9 Test suite supports test supervisor concept. This concept is realized by separate Python script called ```host test```. Host tests can be found in ```mbed/tools/host_tests/``` directory. Note: In newer mbed versions (mbed OS) host tests will be separate library.
bryantaylor 0:eafc3fd41f75 10
bryantaylor 0:eafc3fd41f75 11 Host test script is executed in parallel with test runner to monitor test execution. Basic host test just monitors device's default serial port for test results returned by test runner. Simple tests will print test result on serial port. In other cases host tests can for example judge by test results returned by test runner if test passed or failed. It all depends on test itself.
bryantaylor 0:eafc3fd41f75 12
bryantaylor 0:eafc3fd41f75 13 In some cases host test can be TCP server echoing packets from test runner and judging packet loss. In other cases it can just check if values returned from accelerometer are actually valid (sane).
bryantaylor 0:eafc3fd41f75 14
bryantaylor 0:eafc3fd41f75 15 ## Test suite core: singletest.py script
bryantaylor 0:eafc3fd41f75 16
bryantaylor 0:eafc3fd41f75 17 ```singletest.py``` script located in ```mbed/tools/``` is a test suite script which allows users to compile, build tests and test runners (also supports CppUTest unit test library). Script also is responsible for test execution on devices selected by configuration files.
bryantaylor 0:eafc3fd41f75 18
bryantaylor 0:eafc3fd41f75 19 ### Parameters of singletest.py
bryantaylor 0:eafc3fd41f75 20
bryantaylor 0:eafc3fd41f75 21 Test execution script ```singletest.py``` is a fairly powerful tool to run tests for mbed SDK platform. It is flexible and allows users to configure test execution process and define which mbed platforms will be tested.
bryantaylor 0:eafc3fd41f75 22
bryantaylor 0:eafc3fd41f75 23 By specifying external configuration files (in JSON format) you can gain flexibility and prepare many different test scenarios. Just pass configuration file names to your script and run it.
bryantaylor 0:eafc3fd41f75 24
bryantaylor 0:eafc3fd41f75 25 #### MUTs Specification
bryantaylor 0:eafc3fd41f75 26 You can easily configure your MUTs (Mbed Under Test) by creating configuration file with MUTs description.
bryantaylor 0:eafc3fd41f75 27 Note: This configuration file must be in [JSON format](http://www.w3schools.com/json/).
bryantaylor 0:eafc3fd41f75 28 Note: Unfortunately JSON format is not allowing you to have comments inside JSON code.
bryantaylor 0:eafc3fd41f75 29
bryantaylor 0:eafc3fd41f75 30 Let’s see some example and let's try to configure small "test farm" with three devices connected to your host computer. In this example no peripherals (like SD card or EEPROM) are connected to our Mbed boards. We will use three platforms in this example:
bryantaylor 0:eafc3fd41f75 31 * [NXP LPC1768](https://mbed.org/platforms/mbed-LPC1768) board.
bryantaylor 0:eafc3fd41f75 32 * \[Freescale KL25Z](https://mbed.org/platforms/KL25Z) board and
bryantaylor 0:eafc3fd41f75 33 * [STMicro Nucleo F103RB](https://mbed.org/platforms/ST-Nucleo-F103RB) board.
bryantaylor 0:eafc3fd41f75 34 After connecting boards to our host machine (PC) we can check which serial ports and disks they occupy. For our example let's assume that:
bryantaylor 0:eafc3fd41f75 35 * ```LPC1768``` serial port is on ```COM4``` and disk drive is ```J:```.
bryantaylor 0:eafc3fd41f75 36 * ```KL25Z``` serial port is on ```COM39``` and disk drive is ```E:```.
bryantaylor 0:eafc3fd41f75 37 * ```NUCLEO_F103RB``` serial port is on ```COM11``` and disk drive is ```I:```.
bryantaylor 0:eafc3fd41f75 38 If you are working under Linux your port and disk could look like /dev/ttyACM5 and /media/usb5.
bryantaylor 0:eafc3fd41f75 39
bryantaylor 0:eafc3fd41f75 40 This information is needed to create ```muts_all.json``` configuration file. You can create it in ```mbed/tools/``` directory:
bryantaylor 0:eafc3fd41f75 41 ```
bryantaylor 0:eafc3fd41f75 42 $ touch muts_all.json
bryantaylor 0:eafc3fd41f75 43 ```
bryantaylor 0:eafc3fd41f75 44
bryantaylor 0:eafc3fd41f75 45 Its name will be passed to ```singletest.py``` script after ```-M``` (MUTs specification) switch. Let’s see how this file's content would look like in our example below:
bryantaylor 0:eafc3fd41f75 46 ```json
bryantaylor 0:eafc3fd41f75 47 {
bryantaylor 0:eafc3fd41f75 48 "1" : {"mcu": "LPC1768",
bryantaylor 0:eafc3fd41f75 49 "port":"COM4",
bryantaylor 0:eafc3fd41f75 50 "disk":"J:\\",
bryantaylor 0:eafc3fd41f75 51 "peripherals": []
bryantaylor 0:eafc3fd41f75 52 },
bryantaylor 0:eafc3fd41f75 53
bryantaylor 0:eafc3fd41f75 54 "2" : {"mcu": "KL25Z",
bryantaylor 0:eafc3fd41f75 55 "port":"COM39",
bryantaylor 0:eafc3fd41f75 56 "disk":"E:\\",
bryantaylor 0:eafc3fd41f75 57 "peripherals": []
bryantaylor 0:eafc3fd41f75 58 },
bryantaylor 0:eafc3fd41f75 59
bryantaylor 0:eafc3fd41f75 60 "3" : {"mcu": "NUCLEO_F103RB",
bryantaylor 0:eafc3fd41f75 61 "port":"COM11",
bryantaylor 0:eafc3fd41f75 62 "disk":"I:\\",
bryantaylor 0:eafc3fd41f75 63 "peripherals": []
bryantaylor 0:eafc3fd41f75 64 }
bryantaylor 0:eafc3fd41f75 65 }
bryantaylor 0:eafc3fd41f75 66 ```
bryantaylor 0:eafc3fd41f75 67
bryantaylor 0:eafc3fd41f75 68 Note: We will leave field ```peripherals``` empty for the sake of this example. We will explain it later. All you need to do now is to properly fill fields ```mcu```, ```port``` and ```disk```.
bryantaylor 0:eafc3fd41f75 69
bryantaylor 0:eafc3fd41f75 70 Note: Please make sure files muts_all.json and test_spec.json are in tools/ directory. We will assume in this example they are.
bryantaylor 0:eafc3fd41f75 71 Where to find ```mcu``` names? You can use option ```-S``` of ```build.py``` script (in ```mbed/tools/``` directory) to check all supported off-line MCUs names.
bryantaylor 0:eafc3fd41f75 72
bryantaylor 0:eafc3fd41f75 73 Note: If you update mbed device firmware or even disconnect / reconnect mbed device you may find that serial port / disk configuration changed. You need to update configuration file accordingly or you will face connection problems and obviously tests will not run.
bryantaylor 0:eafc3fd41f75 74
bryantaylor 0:eafc3fd41f75 75 #### Peripherals testing
bryantaylor 0:eafc3fd41f75 76 When using MUTs configuration file (switch ```-M```) you can define in MUTs JSON file peripherals connected to your device:
bryantaylor 0:eafc3fd41f75 77 ```json
bryantaylor 0:eafc3fd41f75 78 {
bryantaylor 0:eafc3fd41f75 79 "1" : {"mcu" : "KL25Z",
bryantaylor 0:eafc3fd41f75 80 "port" : "COM39",
bryantaylor 0:eafc3fd41f75 81 "disk" : "E:\\",
bryantaylor 0:eafc3fd41f75 82 "peripherals" : ["SD", "24LC256"]}
bryantaylor 0:eafc3fd41f75 83 }
bryantaylor 0:eafc3fd41f75 84 ```
bryantaylor 0:eafc3fd41f75 85 You can force test suite to run only common tests (switch ```-C```) or only peripheral tests (switch ```-P```).
bryantaylor 0:eafc3fd41f75 86 ```
bryantaylor 0:eafc3fd41f75 87 $ python singletest.py -i test_spec.json -M muts_all.json -C
bryantaylor 0:eafc3fd41f75 88 ```
bryantaylor 0:eafc3fd41f75 89 will not include tests for SD card and EEPROM 24LC256.
bryantaylor 0:eafc3fd41f75 90 ```
bryantaylor 0:eafc3fd41f75 91 $ python singletest.py -i test_spec.json -M muts_all.json -P
bryantaylor 0:eafc3fd41f75 92 ```
bryantaylor 0:eafc3fd41f75 93 will only run tests bind to SD card and EEPROM 24LC256.
bryantaylor 0:eafc3fd41f75 94
bryantaylor 0:eafc3fd41f75 95 Note: option ```-P``` is useful for example in cases when you have same platform and different shields you want to test. No need to test common part all the time (timers, RTC, RTOS etc.). You can force to test peripherals only on some devices and for example only common tests on other devices.
bryantaylor 0:eafc3fd41f75 96
bryantaylor 0:eafc3fd41f75 97 #### Additional MUTs configuration file settings
bryantaylor 0:eafc3fd41f75 98 You can add extra information to each MUT configuration. In particular you can specify which flashing (binary copy method) should be used, how to reset target and for example set reset timeout (used to delay test execution just after reset).
bryantaylor 0:eafc3fd41f75 99
bryantaylor 0:eafc3fd41f75 100 muts_all.json:
bryantaylor 0:eafc3fd41f75 101 ```json
bryantaylor 0:eafc3fd41f75 102 {
bryantaylor 0:eafc3fd41f75 103 "1" : {"mcu" : "LPC1768",
bryantaylor 0:eafc3fd41f75 104 "port" : "COM77",
bryantaylor 0:eafc3fd41f75 105 "disk" : "G:\\",
bryantaylor 0:eafc3fd41f75 106 "peripherals" : ["TMP102", "digital_loop", "port_loop", "analog_loop", "SD"]},
bryantaylor 0:eafc3fd41f75 107
bryantaylor 0:eafc3fd41f75 108 "2" : {"mcu" : "KL25Z",
bryantaylor 0:eafc3fd41f75 109 "port" : "COM89",
bryantaylor 0:eafc3fd41f75 110 "disk" : "F:\\",
bryantaylor 0:eafc3fd41f75 111 "peripherals" : ["SD", "24LC256", "KL25Z"],
bryantaylor 0:eafc3fd41f75 112 "copy_method" : "copy",
bryantaylor 0:eafc3fd41f75 113 "reset_type" : "default",
bryantaylor 0:eafc3fd41f75 114 "reset_tout" : "2"},
bryantaylor 0:eafc3fd41f75 115
bryantaylor 0:eafc3fd41f75 116 "3" : {"mcu" : "LPC11U24",
bryantaylor 0:eafc3fd41f75 117 "port" : "COM76",
bryantaylor 0:eafc3fd41f75 118 "disk" : "E:\\",
bryantaylor 0:eafc3fd41f75 119 "peripherals" : []}
bryantaylor 0:eafc3fd41f75 120 }
bryantaylor 0:eafc3fd41f75 121 ```
bryantaylor 0:eafc3fd41f75 122 Please note that for MUT no. 2 few extra parameters were defined: ```copy_method```, ```reset_type``` and ```reset_tout```. Using this extra options you can tell test suite more about MUT you are using. This will allow you to be more flexible in terms of how you configure and use your MUTs.
bryantaylor 0:eafc3fd41f75 123
bryantaylor 0:eafc3fd41f75 124 * ```copy_method``` - STRING - tells test suite which binary copy method should be used.
bryantaylor 0:eafc3fd41f75 125 You may notice that ```singletest.py``` command line help contains description about:
bryantaylor 0:eafc3fd41f75 126 * Option ```-c``` (in MUTs file called ```copy_method```) with available copy methods supported by test suite plugin system.
bryantaylor 0:eafc3fd41f75 127 * Option ```-r``` (in MUTs file called reset_type) with available reset methods supported by test suite plugin system.
bryantaylor 0:eafc3fd41f75 128 * ```reset_type``` - STRING - some boards may require special reset handling, for example vendor specific command must be executed to reset device.
bryantaylor 0:eafc3fd41f75 129 * ```reset_tout``` - INTEGER - extra timeout just after device is reseted. May be used to wait for few seconds so device may finish booting, flashing data internally etc.
bryantaylor 0:eafc3fd41f75 130
bryantaylor 0:eafc3fd41f75 131 Part of help listing for singletest.py:
bryantaylor 0:eafc3fd41f75 132 ```
bryantaylor 0:eafc3fd41f75 133 -c COPY_METHOD, --copy-method=COPY_METHOD
bryantaylor 0:eafc3fd41f75 134 Select binary copy (flash) method. Default is Python's
bryantaylor 0:eafc3fd41f75 135 shutil.copy() method. Plugin support: copy, cp,
bryantaylor 0:eafc3fd41f75 136 default, eACommander, eACommander-usb, xcopy
bryantaylor 0:eafc3fd41f75 137 -r MUT_RESET_TYPE, --reset-type=MUT_RESET_TYPE
bryantaylor 0:eafc3fd41f75 138 Extra reset method used to reset MUT by host test
bryantaylor 0:eafc3fd41f75 139 script. Plugin support: default, eACommander,
bryantaylor 0:eafc3fd41f75 140 eACommander-usb
bryantaylor 0:eafc3fd41f75 141 ```
bryantaylor 0:eafc3fd41f75 142
bryantaylor 0:eafc3fd41f75 143 ----
bryantaylor 0:eafc3fd41f75 144
bryantaylor 0:eafc3fd41f75 145 Now we've already defined how our devices are connected to our host PC. We can continue and define which of this MUTs will be tested and which compilers we will use to compile and build Mbed SDK and tests. To do so we need to create test specification file (let's call it ```test_spec.json```) and put inside our configuration file information about which MUTs we actually want to test. We will pass this file's name to ```singletest.py``` script using ```-i``` switch.
bryantaylor 0:eafc3fd41f75 146
bryantaylor 0:eafc3fd41f75 147 Below we can see how sample ```test_spec.json``` file content could look like. (I've included all possible toolchains, we will change it in a moment):
bryantaylor 0:eafc3fd41f75 148 ```json
bryantaylor 0:eafc3fd41f75 149 {
bryantaylor 0:eafc3fd41f75 150 "targets": {
bryantaylor 0:eafc3fd41f75 151 "LPC1768" : ["ARM", "uARM", "GCC_ARM", "GCC_CR", "IAR"],
bryantaylor 0:eafc3fd41f75 152 "KL25Z" : ["ARM", "GCC_ARM"],
bryantaylor 0:eafc3fd41f75 153 "NUCLEO_F103RB" : ["ARM", "uARM"]
bryantaylor 0:eafc3fd41f75 154 }
bryantaylor 0:eafc3fd41f75 155 }
bryantaylor 0:eafc3fd41f75 156 ```
bryantaylor 0:eafc3fd41f75 157 Above example configuration will force tests for LPC1768, KL25Z, NUCLEO_F103RB platforms and:
bryantaylor 0:eafc3fd41f75 158
bryantaylor 0:eafc3fd41f75 159 * Compilers: ```ARM```, ```uARM```, ```GCC_ARM```, ```GCC_CR``` and ```IAR``` will be used to compile tests for NXP's ```LPC1768```.
bryantaylor 0:eafc3fd41f75 160 * Compilers: ```ARM``` and ```GCC_ARM``` will be used for Freescales' ```KL25Z``` platform.
bryantaylor 0:eafc3fd41f75 161 * Compilers: ```ARM``` and ```uARM``` will be used for STMicro's ```NUCLEO_F103RB``` platform.
bryantaylor 0:eafc3fd41f75 162
bryantaylor 0:eafc3fd41f75 163 For our example purposes let's assume we only have Keil ARM compiler, so let's change configuration in ```test_spec.json``` file and reduce number of compiler to those we actually have:
bryantaylor 0:eafc3fd41f75 164 ```json
bryantaylor 0:eafc3fd41f75 165 {
bryantaylor 0:eafc3fd41f75 166 "targets": {
bryantaylor 0:eafc3fd41f75 167 "LPC1768" : ["ARM", "uARM"],
bryantaylor 0:eafc3fd41f75 168 "KL25Z" : ["ARM"],
bryantaylor 0:eafc3fd41f75 169 "NUCLEO_F103RB" : ["ARM", "uARM"]
bryantaylor 0:eafc3fd41f75 170 }
bryantaylor 0:eafc3fd41f75 171 }
bryantaylor 0:eafc3fd41f75 172 ```
bryantaylor 0:eafc3fd41f75 173 #### Run your tests
bryantaylor 0:eafc3fd41f75 174
bryantaylor 0:eafc3fd41f75 175 After you configure all your MUTs and compilers you are ready to run tests. Make sure your devices are connected and your configuration files reflect your current configuration (serial ports, devices). Go to tools directory in your mbed location.
bryantaylor 0:eafc3fd41f75 176 ```
bryantaylor 0:eafc3fd41f75 177 $ cd tools/
bryantaylor 0:eafc3fd41f75 178 ```
bryantaylor 0:eafc3fd41f75 179 and execute test suite script.
bryantaylor 0:eafc3fd41f75 180 ```
bryantaylor 0:eafc3fd41f75 181 $ python singletest.py -i test_spec.json -M muts_all.json
bryantaylor 0:eafc3fd41f75 182 ```
bryantaylor 0:eafc3fd41f75 183 To check your configuration before test execution please use ```--config``` switch:
bryantaylor 0:eafc3fd41f75 184 ```
bryantaylor 0:eafc3fd41f75 185 $ python singletest.py -i test_spec.json -M muts_all.json --config
bryantaylor 0:eafc3fd41f75 186 MUTs configuration in m.json:
bryantaylor 0:eafc3fd41f75 187 +-------+-------------+---------------+------+-------+
bryantaylor 0:eafc3fd41f75 188 | index | peripherals | mcu | disk | port |
bryantaylor 0:eafc3fd41f75 189 +-------+-------------+---------------+------+-------+
bryantaylor 0:eafc3fd41f75 190 | 1 | | LPC1768 | J:\ | COM4 |
bryantaylor 0:eafc3fd41f75 191 | 3 | | NUCLEO_F103RB | I:\ | COM11 |
bryantaylor 0:eafc3fd41f75 192 | 2 | | KL25Z | E:\ | COM39 |
bryantaylor 0:eafc3fd41f75 193 +-------+-------------+---------------+------+-------+
bryantaylor 0:eafc3fd41f75 194
bryantaylor 0:eafc3fd41f75 195 Test specification in t.json:
bryantaylor 0:eafc3fd41f75 196 +---------------+-----+------+
bryantaylor 0:eafc3fd41f75 197 | mcu | ARM | uARM |
bryantaylor 0:eafc3fd41f75 198 +---------------+-----+------+
bryantaylor 0:eafc3fd41f75 199 | NUCLEO_F103RB | Yes | Yes |
bryantaylor 0:eafc3fd41f75 200 | KL25Z | Yes | - |
bryantaylor 0:eafc3fd41f75 201 | LPC1768 | Yes | Yes |
bryantaylor 0:eafc3fd41f75 202 +---------------+-----+------+
bryantaylor 0:eafc3fd41f75 203 ```
bryantaylor 0:eafc3fd41f75 204 It should help you localize basic problems with configuration files and toolchain configuration.
bryantaylor 0:eafc3fd41f75 205 Note: Configurations with issues will be marked with ```*``` sign.
bryantaylor 0:eafc3fd41f75 206
bryantaylor 0:eafc3fd41f75 207 Having multiple configuration files allows you to manage your test scenarios in more flexible manner. You can:
bryantaylor 0:eafc3fd41f75 208
bryantaylor 0:eafc3fd41f75 209 * Set up all platforms and toolchains used during testing.
bryantaylor 0:eafc3fd41f75 210 * Define (using script's ```-n``` switch) which tests you want to run during testing.
bryantaylor 0:eafc3fd41f75 211 * Just run regression (all tests). Regression is default setting for test script.
bryantaylor 0:eafc3fd41f75 212
bryantaylor 0:eafc3fd41f75 213 You can also force ```singletest.py``` script to:
bryantaylor 0:eafc3fd41f75 214 * Run only peripherals' tests (switch ```-P```) or
bryantaylor 0:eafc3fd41f75 215 * Just skip peripherals' tests (switch ```-C```).
bryantaylor 0:eafc3fd41f75 216 * Build mbed SDK, libraries and corresponding tests with multiple cores, just use ```-j X``` option where ```X``` is number of cores you want to use for compilation.
bryantaylor 0:eafc3fd41f75 217 ```
bryantaylor 0:eafc3fd41f75 218 $ python singletest.py -i test_spec.json -M muts_all.json -j 8
bryantaylor 0:eafc3fd41f75 219 ```
bryantaylor 0:eafc3fd41f75 220 * Print test cases console output using ```-V``` option.
bryantaylor 0:eafc3fd41f75 221 * Only build mbed SDK, tests and dependant libraries with switch ```-O```:
bryantaylor 0:eafc3fd41f75 222 ```
bryantaylor 0:eafc3fd41f75 223 $ python singletest.py -i test_spec.json -M muts_all.json -j 8 -O
bryantaylor 0:eafc3fd41f75 224 ```
bryantaylor 0:eafc3fd41f75 225 * Execute each test case multiple times with ```--global-loops X``` option, where ```X``` number of repeats. Additionally use option ```-W``` to continue repeating test cases execution only if they continue to fail.
bryantaylor 0:eafc3fd41f75 226 ```
bryantaylor 0:eafc3fd41f75 227 $ python singletest.py -i test_spec.json -M muts_all.json --global-loops 3 -W
bryantaylor 0:eafc3fd41f75 228 ```
bryantaylor 0:eafc3fd41f75 229 * Option ```--loops``` can be used to overwrite global loop count and redefine loop count for particular tests. Define test loops as ```TEST_ID=X``` where ```X``` is integer and separate loops count definitions by comma if necessary. E.g. ```TEST_1=5,TEST_2=20,TEST_3=2```.
bryantaylor 0:eafc3fd41f75 230 ```
bryantaylor 0:eafc3fd41f75 231 $ python singletest.py -i test_spec.json -M muts_all.json RTOS_1=10,RTOS_2=5
bryantaylor 0:eafc3fd41f75 232 ```
bryantaylor 0:eafc3fd41f75 233 This will execute test ```RTOS_1``` ten (10) times and test ```RTOS_2``` five (5) times.
bryantaylor 0:eafc3fd41f75 234 * Force non default copy method. Note that mbed platforms can be flashed with just binary drag&drop. We simply copy file onto mbed's disk and interface chip flashes target MCU with given binary. Force non standard (Python specific) copy method by using option ```-c COPY_METHOD``` where ```COPY_METHOD``` can be shell, command line copy command like: ```cp```, ```copy````, ```xcopy``` etc. Make sure those commands are available from command line!
bryantaylor 0:eafc3fd41f75 235 ```
bryantaylor 0:eafc3fd41f75 236 $ python singletest.py -i test_spec.json -M muts_all.json -c cp
bryantaylor 0:eafc3fd41f75 237 ```
bryantaylor 0:eafc3fd41f75 238 * Run only selected tests. You can select which tests should be executed when you run test suite. Use ```-n``` switch to define tests by their ids you want to execute. Use comma to separate test ids:
bryantaylor 0:eafc3fd41f75 239 ```
bryantaylor 0:eafc3fd41f75 240 $ python singletest.py -i test_spec.json -M muts_all.json -n RTOS_1,RTOS_2,RTOS_3,MBED_10,MBED_16,MBED_11
bryantaylor 0:eafc3fd41f75 241 ```
bryantaylor 0:eafc3fd41f75 242 * Set common output binary name for all tests. In some cases you would like to have the same name for all tests. You can use switch ```--firmware-name``` to specify (without extension) build script output binary name.
bryantaylor 0:eafc3fd41f75 243 In below example we would like to have all test binaries called ```firmware.bin`` (or with other extension like .elf, .hex depending on target accepted format).
bryantaylor 0:eafc3fd41f75 244 ```
bryantaylor 0:eafc3fd41f75 245 $ python singletest.py -i test_spec.json -M muts_all.json --firmware-name firmware
bryantaylor 0:eafc3fd41f75 246 ```
bryantaylor 0:eafc3fd41f75 247 * Where to find test list? Tests are defined in file ```tests.py``` in ```mbed/tools/``` directory. ```singletest.py``` uses test metadata in ```tests.py``` to resolve libraries dependencies and build tests for proper platforms and peripherals. Option ```-R``` can be used to get test names and direct path and test configuration.
bryantaylor 0:eafc3fd41f75 248 ```
bryantaylor 0:eafc3fd41f75 249 $ python singletest.py -R
bryantaylor 0:eafc3fd41f75 250 +-------------+-----------+---------------------------------------+--------------+-------------------+----------+--------------------------------------------------------+
bryantaylor 0:eafc3fd41f75 251 | id | automated | description | peripherals | host_test | duration | source_dir |
bryantaylor 0:eafc3fd41f75 252 +-------------+-----------+---------------------------------------+--------------+-------------------+----------+--------------------------------------------------------+
bryantaylor 0:eafc3fd41f75 253 | MBED_1 | False | I2C SRF08 | SRF08 | host_test | 10 | C:\Work\mbed\libraries\tests\mbed\i2c_SRF08 |
bryantaylor 0:eafc3fd41f75 254 | MBED_10 | True | Hello World | - | host_test | 10 | C:\Work\mbed\libraries\tests\mbed\hello |
bryantaylor 0:eafc3fd41f75 255 | MBED_11 | True | Ticker Int | - | host_test | 20 | C:\Work\mbed\libraries\tests\mbed\ticker |
bryantaylor 0:eafc3fd41f75 256 | MBED_12 | True | C++ | - | host_test | 10 | C:\Work\mbed\libraries\tests\mbed\cpp |
bryantaylor 0:eafc3fd41f75 257 | MBED_13 | False | Heap & Stack | - | host_test | 10 | C:\Work\mbed\libraries\tests\mbed\heap_and_stack |
bryantaylor 0:eafc3fd41f75 258 | MBED_14 | False | Serial Interrupt | - | host_test | 10 | C:\Work\mbed\libraries\tests\mbed\serial_interrupt |
bryantaylor 0:eafc3fd41f75 259 | MBED_15 | False | RPC | - | host_test | 10 | C:\Work\mbed\libraries\tests\mbed\rpc |
bryantaylor 0:eafc3fd41f75 260 | MBED_16 | True | RTC | - | host_test | 15 | C:\Work\mbed\libraries\tests\mbed\rtc |
bryantaylor 0:eafc3fd41f75 261 | MBED_17 | False | Serial Interrupt 2 | - | host_test | 10 | C:\Work\mbed\libraries\tests\mbed\serial_interrupt_2 |
bryantaylor 0:eafc3fd41f75 262 | MBED_18 | False | Local FS Directory | - | host_test | 10 | C:\Work\mbed\libraries\tests\mbed\dir |
bryantaylor 0:eafc3fd41f75 263 ...
bryantaylor 0:eafc3fd41f75 264 ```
bryantaylor 0:eafc3fd41f75 265 Note: you can filter tests by ```id``` column, just use ```-f``` option and give test name or regular expression:
bryantaylor 0:eafc3fd41f75 266 ```
bryantaylor 0:eafc3fd41f75 267 $ python singletest.py -R -f RTOS
bryantaylor 0:eafc3fd41f75 268 +--------------+-----------+-------------------------+-------------+-----------+----------+---------------------------------------------------+
bryantaylor 0:eafc3fd41f75 269 | id | automated | description | peripherals | host_test | duration | source_dir |
bryantaylor 0:eafc3fd41f75 270 +--------------+-----------+-------------------------+-------------+-----------+----------+---------------------------------------------------+
bryantaylor 0:eafc3fd41f75 271 | CMSIS_RTOS_1 | False | Basic | - | host_test | 10 | C:\Work\mbed\libraries\tests\rtos\cmsis\basic |
bryantaylor 0:eafc3fd41f75 272 | CMSIS_RTOS_2 | False | Mutex | - | host_test | 20 | C:\Work\mbed\libraries\tests\rtos\cmsis\mutex |
bryantaylor 0:eafc3fd41f75 273 | CMSIS_RTOS_3 | False | Semaphore | - | host_test | 20 | C:\Work\mbed\libraries\tests\rtos\cmsis\semaphore |
bryantaylor 0:eafc3fd41f75 274 | CMSIS_RTOS_4 | False | Signals | - | host_test | 10 | C:\Work\mbed\libraries\tests\rtos\cmsis\signals |
bryantaylor 0:eafc3fd41f75 275 | CMSIS_RTOS_5 | False | Queue | - | host_test | 20 | C:\Work\mbed\libraries\tests\rtos\cmsis\queue |
bryantaylor 0:eafc3fd41f75 276 | CMSIS_RTOS_6 | False | Mail | - | host_test | 20 | C:\Work\mbed\libraries\tests\rtos\cmsis\mail |
bryantaylor 0:eafc3fd41f75 277 | CMSIS_RTOS_7 | False | Timer | - | host_test | 10 | C:\Work\mbed\libraries\tests\rtos\cmsis\timer |
bryantaylor 0:eafc3fd41f75 278 | CMSIS_RTOS_8 | False | ISR | - | host_test | 10 | C:\Work\mbed\libraries\tests\rtos\cmsis\isr |
bryantaylor 0:eafc3fd41f75 279 | RTOS_1 | True | Basic thread | - | host_test | 15 | C:\Work\mbed\libraries\tests\rtos\mbed\basic |
bryantaylor 0:eafc3fd41f75 280 | RTOS_2 | True | Mutex resource lock | - | host_test | 20 | C:\Work\mbed\libraries\tests\rtos\mbed\mutex |
bryantaylor 0:eafc3fd41f75 281 | RTOS_3 | True | Semaphore resource lock | - | host_test | 20 | C:\Work\mbed\libraries\tests\rtos\mbed\semaphore |
bryantaylor 0:eafc3fd41f75 282 | RTOS_4 | True | Signals messaging | - | host_test | 10 | C:\Work\mbed\libraries\tests\rtos\mbed\signals |
bryantaylor 0:eafc3fd41f75 283 | RTOS_5 | True | Queue messaging | - | host_test | 10 | C:\Work\mbed\libraries\tests\rtos\mbed\queue |
bryantaylor 0:eafc3fd41f75 284 | RTOS_6 | True | Mail messaging | - | host_test | 10 | C:\Work\mbed\libraries\tests\rtos\mbed\mail |
bryantaylor 0:eafc3fd41f75 285 | RTOS_7 | True | Timer | - | host_test | 15 | C:\Work\mbed\libraries\tests\rtos\mbed\timer |
bryantaylor 0:eafc3fd41f75 286 | RTOS_8 | True | ISR (Queue) | - | host_test | 10 | C:\Work\mbed\libraries\tests\rtos\mbed\isr |
bryantaylor 0:eafc3fd41f75 287 | RTOS_9 | True | SD File write-read | SD | host_test | 10 | C:\Work\mbed\libraries\tests\rtos\mbed\file |
bryantaylor 0:eafc3fd41f75 288 +--------------+-----------+-------------------------+-------------+-----------+----------+---------------------------------------------------+
bryantaylor 0:eafc3fd41f75 289 ```
bryantaylor 0:eafc3fd41f75 290
bryantaylor 0:eafc3fd41f75 291 * Shuffle your tests. We strongly encourage you to shuffle your test order each time you execute test suite script.
bryantaylor 0:eafc3fd41f75 292 Rationale: It is probable that tests executed in one particular order will pass and in other will fail. To shuffle your tests’ order please use option ```-u``` (or ```--shuffle```):
bryantaylor 0:eafc3fd41f75 293 ```
bryantaylor 0:eafc3fd41f75 294 $ python singletest.py -i test_spec.json -M muts_all.json --shuffle
bryantaylor 0:eafc3fd41f75 295 ```
bryantaylor 0:eafc3fd41f75 296 Above command will force test script to randomly generate shuffle seed and shuffle test order execution. Note: Shuffle seed is float in ```[0.0, 1.0)```.
bryantaylor 0:eafc3fd41f75 297
bryantaylor 0:eafc3fd41f75 298 You can always recreate particular test order by forcing shuffle (```-u``` or ```--shuffle``` switch) and passing shuffle seed back to test suite using ```--shuffle-seed``` switch:
bryantaylor 0:eafc3fd41f75 299 ```
bryantaylor 0:eafc3fd41f75 300 $ python singletest.py -i test_spec.json -M muts_all.json --shuffle --shuffle-seed 0.4041028336
bryantaylor 0:eafc3fd41f75 301 ```
bryantaylor 0:eafc3fd41f75 302 Note: You can also supply your own randomly generated shuffle seed to drive particular test execution order scenarios. Just make sure shuffle seed is float in ```[0.0, 1.0)```.
bryantaylor 0:eafc3fd41f75 303 You can find test shuffle seed in test summary:
bryantaylor 0:eafc3fd41f75 304 ```
bryantaylor 0:eafc3fd41f75 305 ...
bryantaylor 0:eafc3fd41f75 306 | OK | LPC1768 | ARM | MBED_A9 | Serial Echo at 115200 | 2.84 | 10 | 1/1 |
bryantaylor 0:eafc3fd41f75 307 +--------+---------+-----------+-----------+-----------------------------+--------------------+---------------+-------+
bryantaylor 0:eafc3fd41f75 308 Result: 1 FAIL / 22 OK
bryantaylor 0:eafc3fd41f75 309 Shuffle Seed: 0.4041028336
bryantaylor 0:eafc3fd41f75 310
bryantaylor 0:eafc3fd41f75 311 Completed in 234.85 sec
bryantaylor 0:eafc3fd41f75 312 ```
bryantaylor 0:eafc3fd41f75 313
bryantaylor 0:eafc3fd41f75 314 ### Example of device configuration (one device connected to host computer)
bryantaylor 0:eafc3fd41f75 315
bryantaylor 0:eafc3fd41f75 316 This example will show you how to configure single device, run general tests or only peripheral tests. We will also show some real test result examples.
bryantaylor 0:eafc3fd41f75 317
bryantaylor 0:eafc3fd41f75 318 1. We will test only one board STMIcro Nucleo ```F334R8``` board connected to our PC (port ```COM46``` and disk is ```E:```).
bryantaylor 0:eafc3fd41f75 319 2. We will also connect EEPROM ```24LC256``` to SDA, SCL pins of our Nucleo board and define 24LC256 peripheral to make sure our test suite will run all available tests for ```24LC256```.
bryantaylor 0:eafc3fd41f75 320
bryantaylor 0:eafc3fd41f75 321 Let's configure our one MUT and set uARM as the only compiler we will use to compiler Mbed SDK and tests.
bryantaylor 0:eafc3fd41f75 322 We also need to create two configuration files ```muts_all.json``` and ```test_spec.json``` to pass our small testbed configuration to test script.
bryantaylor 0:eafc3fd41f75 323
bryantaylor 0:eafc3fd41f75 324 muts_all.json:
bryantaylor 0:eafc3fd41f75 325 ```json
bryantaylor 0:eafc3fd41f75 326 {
bryantaylor 0:eafc3fd41f75 327 "1" : {
bryantaylor 0:eafc3fd41f75 328 "mcu": "NUCLEO_F334R8",
bryantaylor 0:eafc3fd41f75 329 "port":"COM46",
bryantaylor 0:eafc3fd41f75 330 "disk":"E:\\",
bryantaylor 0:eafc3fd41f75 331 "peripherals": ["24LC256"]
bryantaylor 0:eafc3fd41f75 332 }
bryantaylor 0:eafc3fd41f75 333 }
bryantaylor 0:eafc3fd41f75 334 ```
bryantaylor 0:eafc3fd41f75 335 Note: By defining ```"peripherals": ["24LC256"]``` we are passing to test suite information that this particular board has EEPROM 24LC256 connected to our board.
bryantaylor 0:eafc3fd41f75 336
bryantaylor 0:eafc3fd41f75 337 test_spec.json:
bryantaylor 0:eafc3fd41f75 338 ```json
bryantaylor 0:eafc3fd41f75 339 {
bryantaylor 0:eafc3fd41f75 340 "targets": {
bryantaylor 0:eafc3fd41f75 341 "NUCLEO_F334R8" : ["uARM"]
bryantaylor 0:eafc3fd41f75 342 }
bryantaylor 0:eafc3fd41f75 343 }
bryantaylor 0:eafc3fd41f75 344 ```
bryantaylor 0:eafc3fd41f75 345 Note:
bryantaylor 0:eafc3fd41f75 346 * Please make sure device is connected before we will start running tests.
bryantaylor 0:eafc3fd41f75 347 * Please make sure files ```muts_all.json``` and ```test_spec.json``` are in ```mbed/tools/``` directory.
bryantaylor 0:eafc3fd41f75 348 Now you can call test suite and execute tests:
bryantaylor 0:eafc3fd41f75 349 ```
bryantaylor 0:eafc3fd41f75 350 $ python singletest.py -i test_spec.json -M muts_all.json
bryantaylor 0:eafc3fd41f75 351 ...
bryantaylor 0:eafc3fd41f75 352 Test summary:
bryantaylor 0:eafc3fd41f75 353 +--------+---------------+-----------+-----------+---------------------------------+--------------------+---------------+
bryantaylor 0:eafc3fd41f75 354 | Result | Target | Toolchain | Test ID | Test Description | Elapsed Time (sec) | Timeout (sec) |
bryantaylor 0:eafc3fd41f75 355 +--------+---------------+-----------+-----------+---------------------------------+--------------------+---------------+
bryantaylor 0:eafc3fd41f75 356 | OK | NUCLEO_F334R8 | uARM | MBED_A25 | I2C EEPROM line read/write test | 12.41 | 15 |
bryantaylor 0:eafc3fd41f75 357 | OK | NUCLEO_F334R8 | uARM | MBED_A1 | Basic | 3.42 | 10 |
bryantaylor 0:eafc3fd41f75 358 | OK | NUCLEO_F334R8 | uARM | EXAMPLE_1 | /dev/null | 3.42 | 10 |
bryantaylor 0:eafc3fd41f75 359 | OK | NUCLEO_F334R8 | uARM | MBED_24 | Timeout Int us | 11.47 | 15 |
bryantaylor 0:eafc3fd41f75 360 | OK | NUCLEO_F334R8 | uARM | MBED_25 | Time us | 11.43 | 15 |
bryantaylor 0:eafc3fd41f75 361 | OK | NUCLEO_F334R8 | uARM | MBED_26 | Integer constant division | 3.37 | 10 |
bryantaylor 0:eafc3fd41f75 362 | OK | NUCLEO_F334R8 | uARM | MBED_23 | Ticker Int us | 12.43 | 15 |
bryantaylor 0:eafc3fd41f75 363 | OK | NUCLEO_F334R8 | uARM | MBED_A19 | I2C EEPROM read/write test | 11.42 | 15 |
bryantaylor 0:eafc3fd41f75 364 | OK | NUCLEO_F334R8 | uARM | MBED_11 | Ticker Int | 12.43 | 20 |
bryantaylor 0:eafc3fd41f75 365 | OK | NUCLEO_F334R8 | uARM | MBED_10 | Hello World | 2.42 | 10 |
bryantaylor 0:eafc3fd41f75 366 | OK | NUCLEO_F334R8 | uARM | MBED_12 | C++ | 3.42 | 10 |
bryantaylor 0:eafc3fd41f75 367 | OK | NUCLEO_F334R8 | uARM | MBED_16 | RTC | 4.76 | 15 |
bryantaylor 0:eafc3fd41f75 368 | UNDEF | NUCLEO_F334R8 | uARM | MBED_2 | stdio | 20.42 | 20 |
bryantaylor 0:eafc3fd41f75 369 | UNDEF | NUCLEO_F334R8 | uARM | MBED_A9 | Serial Echo at 115200 | 10.37 | 10 |
bryantaylor 0:eafc3fd41f75 370 +--------+---------------+-----------+-----------+---------------------------------+--------------------+---------------+
bryantaylor 0:eafc3fd41f75 371 Result: 2 UNDEF / 12 OK
bryantaylor 0:eafc3fd41f75 372
bryantaylor 0:eafc3fd41f75 373 Completed in 160 sec
bryantaylor 0:eafc3fd41f75 374 ```
bryantaylor 0:eafc3fd41f75 375
bryantaylor 0:eafc3fd41f75 376 If we want to get additional test summary with results in separate columns please use option ```-t```.
bryantaylor 0:eafc3fd41f75 377 ```
bryantaylor 0:eafc3fd41f75 378 $ python singletest.py -i test_spec.json -M muts_all.json -t
bryantaylor 0:eafc3fd41f75 379 ...
bryantaylor 0:eafc3fd41f75 380 Test summary:
bryantaylor 0:eafc3fd41f75 381 +---------------+-----------+---------------------------------+-------+
bryantaylor 0:eafc3fd41f75 382 | Target | Test ID | Test Description | uARM |
bryantaylor 0:eafc3fd41f75 383 +---------------+-----------+---------------------------------+-------+
bryantaylor 0:eafc3fd41f75 384 | NUCLEO_F334R8 | EXAMPLE_1 | /dev/null | OK |
bryantaylor 0:eafc3fd41f75 385 | NUCLEO_F334R8 | MBED_10 | Hello World | OK |
bryantaylor 0:eafc3fd41f75 386 | NUCLEO_F334R8 | MBED_11 | Ticker Int | OK |
bryantaylor 0:eafc3fd41f75 387 | NUCLEO_F334R8 | MBED_12 | C++ | OK |
bryantaylor 0:eafc3fd41f75 388 | NUCLEO_F334R8 | MBED_16 | RTC | OK |
bryantaylor 0:eafc3fd41f75 389 | NUCLEO_F334R8 | MBED_2 | stdio | UNDEF |
bryantaylor 0:eafc3fd41f75 390 | NUCLEO_F334R8 | MBED_23 | Ticker Int us | OK |
bryantaylor 0:eafc3fd41f75 391 | NUCLEO_F334R8 | MBED_24 | Timeout Int us | OK |
bryantaylor 0:eafc3fd41f75 392 | NUCLEO_F334R8 | MBED_25 | Time us | OK |
bryantaylor 0:eafc3fd41f75 393 | NUCLEO_F334R8 | MBED_26 | Integer constant division | OK |
bryantaylor 0:eafc3fd41f75 394 | NUCLEO_F334R8 | MBED_A1 | Basic | OK |
bryantaylor 0:eafc3fd41f75 395 | NUCLEO_F334R8 | MBED_A19 | I2C EEPROM read/write test | OK |
bryantaylor 0:eafc3fd41f75 396 | NUCLEO_F334R8 | MBED_A25 | I2C EEPROM line read/write test | OK |
bryantaylor 0:eafc3fd41f75 397 | NUCLEO_F334R8 | MBED_A9 | Serial Echo at 115200 | UNDEF |
bryantaylor 0:eafc3fd41f75 398 +---------------+-----------+---------------------------------+-------+
bryantaylor 0:eafc3fd41f75 399 ```
bryantaylor 0:eafc3fd41f75 400 ----
bryantaylor 0:eafc3fd41f75 401 Please do not forget you can combine few options together to get result you want. For example you want to repeat few tests multiple number of times, shuffle test ids execution order and select only tests which are critical for you at this point. You can do it using switch -n, --global-loops with --loops and --shuffle:
bryantaylor 0:eafc3fd41f75 402
bryantaylor 0:eafc3fd41f75 403 Execute above command to:
bryantaylor 0:eafc3fd41f75 404
bryantaylor 0:eafc3fd41f75 405 * Run only tests: ```RTOS_1```, ```RTOS_2```, ```RTOS_3```, ```MBED_10```, ```MBED_16```, ```MBED_11```.
bryantaylor 0:eafc3fd41f75 406 * Shuffle test execution order. Note tests in loops will not be shuffled.
bryantaylor 0:eafc3fd41f75 407 * Set global loop count to 3 - each test will repeated 3 times.
bryantaylor 0:eafc3fd41f75 408 * Overwrite global loop count (set above to 3) and:
bryantaylor 0:eafc3fd41f75 409 * Force to loop test RTOS_1 to execute 3 times.
bryantaylor 0:eafc3fd41f75 410 * Force to loop test RTOS_2 to execute 4 times.
bryantaylor 0:eafc3fd41f75 411 * Force to loop test RTOS_3 to execute 5 times.
bryantaylor 0:eafc3fd41f75 412 * Force to loop test MBED_11 to execute 5 times.
bryantaylor 0:eafc3fd41f75 413
bryantaylor 0:eafc3fd41f75 414 ```
bryantaylor 0:eafc3fd41f75 415 $ python singletest.py -i test_spec.json -M muts_all.json -n RTOS_1,RTOS_2,RTOS_3,MBED_10,MBED_16,MBED_11 --shuffle --global-loops 3 --loops RTOS_1=3,RTOS_2=4,RTOS_3=5,MBED_11=5
bryantaylor 0:eafc3fd41f75 416 ```
bryantaylor 0:eafc3fd41f75 417
bryantaylor 0:eafc3fd41f75 418 # CppUTest unit test library support
bryantaylor 0:eafc3fd41f75 419 ## CppUTest in Mbed SDK testing introduction
bryantaylor 0:eafc3fd41f75 420 [CppUTest](http://cpputest.github.io/) is a C / C++ based unit xUnit test framework for unit testing and for test-driving your code. It is written in C++ but is used in C and C++ projects and frequently used in embedded systems but it works for any C / C++ project.
bryantaylor 0:eafc3fd41f75 421
bryantaylor 0:eafc3fd41f75 422 Mbed SDK test suite supports writing tests using CppUTest. All you need to do it to provide CppUTest sources and includes with Mbed SDK port. This is already done for you so all you need to do it to get proper sources in your project directory.
bryantaylor 0:eafc3fd41f75 423 CppUTest’s core design principles are:
bryantaylor 0:eafc3fd41f75 424 * Simple in design and simple in use.
bryantaylor 0:eafc3fd41f75 425 * Portable to old and new platforms.
bryantaylor 0:eafc3fd41f75 426 * Build with Test-driven Development in mind.
bryantaylor 0:eafc3fd41f75 427
bryantaylor 0:eafc3fd41f75 428 ## From where you can get more help about CppUTest library and unit testing
bryantaylor 0:eafc3fd41f75 429 • You can read [CppUTest manual](http://cpputest.github.io/manual.html)
bryantaylor 0:eafc3fd41f75 430 * [CppUTest forum](https://groups.google.com/forum/?fromgroups#!forum/cpputest)
bryantaylor 0:eafc3fd41f75 431 * [CppUTest on GitHub](https://github.com/cpputest/cpputest)
bryantaylor 0:eafc3fd41f75 432 * Finally, if you think unit testing is new concept for you, you can have a grasp of it on Wikipedia pages about [unit testing](http://en.wikipedia.org/wiki/Unit_testing) and continue from there.
bryantaylor 0:eafc3fd41f75 433
bryantaylor 0:eafc3fd41f75 434 ## How to add CppUTest to your current Mbed SDK installation
bryantaylor 0:eafc3fd41f75 435
bryantaylor 0:eafc3fd41f75 436 ### Do I need CppUTest port for Mbed SDK?
bryantaylor 0:eafc3fd41f75 437 Yes, you do. If you want to use CppUTest with Mbed SDK you need to have CppUTest version with ARMCC compiler (only ARM flavor for now) port and Mbed SDK console port (if you want to have output on serial port). All is already prepared by Mbed engineers and you can get it for example here: http://mbed.org/users/rgrover1/code/CppUTest/
bryantaylor 0:eafc3fd41f75 438
bryantaylor 0:eafc3fd41f75 439 ### Prerequisites
bryantaylor 0:eafc3fd41f75 440 * Installed [git client](http://git-scm.com/downloads/).
bryantaylor 0:eafc3fd41f75 441 * Installed [Mercurial client](http://mercurial.selenic.com/).
bryantaylor 0:eafc3fd41f75 442
bryantaylor 0:eafc3fd41f75 443 ### How / where to install
bryantaylor 0:eafc3fd41f75 444 We want to create directory structure similar to one below:
bryantaylor 0:eafc3fd41f75 445 ```
bryantaylor 0:eafc3fd41f75 446 \your_project_directory
bryantaylor 0:eafc3fd41f75 447 │
bryantaylor 0:eafc3fd41f75 448 ├───cpputest
bryantaylor 0:eafc3fd41f75 449 │ ├───include
bryantaylor 0:eafc3fd41f75 450 │ └───src
bryantaylor 0:eafc3fd41f75 451 └───mbed
bryantaylor 0:eafc3fd41f75 452 ├───libraries
bryantaylor 0:eafc3fd41f75 453 ├───travis
bryantaylor 0:eafc3fd41f75 454 └───tools
bryantaylor 0:eafc3fd41f75 455 ```
bryantaylor 0:eafc3fd41f75 456
bryantaylor 0:eafc3fd41f75 457 Please go to directory with your project. For example it could be c:\Projects\Project.
bryantaylor 0:eafc3fd41f75 458 ```
bryantaylor 0:eafc3fd41f75 459 $ cd c:\Projects\Project
bryantaylor 0:eafc3fd41f75 460 ```
bryantaylor 0:eafc3fd41f75 461 If your project directory already has your mbed SDK repository included just execute below command (Mercurial console client). It should download CppUTest with Mbed SDK port.
bryantaylor 0:eafc3fd41f75 462 ```
bryantaylor 0:eafc3fd41f75 463 $ hg clone https://mbed.org/users/rgrover1/code/cpputest/
bryantaylor 0:eafc3fd41f75 464 ```
bryantaylor 0:eafc3fd41f75 465
bryantaylor 0:eafc3fd41f75 466 You should see something like this after you execute Mercurial clone command:
bryantaylor 0:eafc3fd41f75 467 ```
bryantaylor 0:eafc3fd41f75 468 $ hg clone https://mbed.org/users/rgrover1/code/cpputest/
bryantaylor 0:eafc3fd41f75 469 destination directory: cpputest
bryantaylor 0:eafc3fd41f75 470 requesting all changes
bryantaylor 0:eafc3fd41f75 471 adding changesets
bryantaylor 0:eafc3fd41f75 472 adding manifests
bryantaylor 0:eafc3fd41f75 473 adding file changes
bryantaylor 0:eafc3fd41f75 474 added 3 changesets with 69 changes to 42 files
bryantaylor 0:eafc3fd41f75 475 updating to branch default
bryantaylor 0:eafc3fd41f75 476 41 files updated, 0 files merged, 0 files removed, 0 files unresolved
bryantaylor 0:eafc3fd41f75 477 ```
bryantaylor 0:eafc3fd41f75 478
bryantaylor 0:eafc3fd41f75 479 Confirm your project structure. It should look more or less like this:
bryantaylor 0:eafc3fd41f75 480 ```
bryantaylor 0:eafc3fd41f75 481 $ ls
bryantaylor 0:eafc3fd41f75 482 cpputest mbed
bryantaylor 0:eafc3fd41f75 483 ```
bryantaylor 0:eafc3fd41f75 484 From now on CppUTest is in correct path. Each time you want to compile unit tests for CppUTest build script will always look for CppUTest library in the same directory where mbed library is.
bryantaylor 0:eafc3fd41f75 485
bryantaylor 0:eafc3fd41f75 486 ## New off-line mbed SDK project with CppUTest support
bryantaylor 0:eafc3fd41f75 487
bryantaylor 0:eafc3fd41f75 488 If you are creating new mbed SDK project and you want to use CppUTest with it you need to download both mbed SDK and CppUTest with mbed port to the same directory. You can do it like this:
bryantaylor 0:eafc3fd41f75 489 ```
bryantaylor 0:eafc3fd41f75 490 $ cd c:\Projects\Project
bryantaylor 0:eafc3fd41f75 491 $ git clone https://github.com/mbedmicro/mbed.git
bryantaylor 0:eafc3fd41f75 492 $ hg clone https://mbed.org/users/rgrover1/code/cpputest/
bryantaylor 0:eafc3fd41f75 493 ```
bryantaylor 0:eafc3fd41f75 494
bryantaylor 0:eafc3fd41f75 495 After above three steps you should have proper directory structure. All you need to do now is to configure your ```mbed_settings.py``` in ```mbed``` directory. Please refer to mbed SDK build script documentation for details.
bryantaylor 0:eafc3fd41f75 496
bryantaylor 0:eafc3fd41f75 497 ## CppUTest with mbed port
bryantaylor 0:eafc3fd41f75 498 To make sure you actualy have CppUTest library with mbed SDK port you can go to CppUTest ```armcc``` platform directory:
bryantaylor 0:eafc3fd41f75 499 ```
bryantaylor 0:eafc3fd41f75 500 $ cd c:/Projects/Project/cpputest/src/Platforms/armcc/
bryantaylor 0:eafc3fd41f75 501 ```
bryantaylor 0:eafc3fd41f75 502 And open file ```UtestPlatform.cpp```.
bryantaylor 0:eafc3fd41f75 503
bryantaylor 0:eafc3fd41f75 504 You should find part of code responsible for porting console on default serial port of the mbed device:
bryantaylor 0:eafc3fd41f75 505 ```c++
bryantaylor 0:eafc3fd41f75 506 #include "Serial.h"
bryantaylor 0:eafc3fd41f75 507 using namespace mbed;
bryantaylor 0:eafc3fd41f75 508
bryantaylor 0:eafc3fd41f75 509 int PlatformSpecificPutchar(int c)
bryantaylor 0:eafc3fd41f75 510 {
bryantaylor 0:eafc3fd41f75 511 /* Please modify this block for test results to be reported as
bryantaylor 0:eafc3fd41f75 512 * console output. The following is a sample implementation using a
bryantaylor 0:eafc3fd41f75 513 * Serial object connected to the console. */
bryantaylor 0:eafc3fd41f75 514 #define NEED_TEST_REPORT_AS_CONSOLE_OUTPUT 1
bryantaylor 0:eafc3fd41f75 515 #if NEED_TEST_REPORT_AS_CONSOLE_OUTPUT
bryantaylor 0:eafc3fd41f75 516 extern Serial console;
bryantaylor 0:eafc3fd41f75 517
bryantaylor 0:eafc3fd41f75 518 #define NEED_LINE_FEED_IN_ADDITION_TO_NEWLINE 1
bryantaylor 0:eafc3fd41f75 519 #if NEED_LINE_FEED_IN_ADDITION_TO_NEWLINE
bryantaylor 0:eafc3fd41f75 520 /* CppUTest emits \n line terminators in its reports; some terminals
bryantaylor 0:eafc3fd41f75 521 * need the linefeed (\r) in addition. */
bryantaylor 0:eafc3fd41f75 522 if (c == '\n') {
bryantaylor 0:eafc3fd41f75 523 console.putc('\r');
bryantaylor 0:eafc3fd41f75 524 }
bryantaylor 0:eafc3fd41f75 525 #endif /* #if NEED_LINE_FEED_IN_ADDITION_TO_NEWLINE */
bryantaylor 0:eafc3fd41f75 526
bryantaylor 0:eafc3fd41f75 527 return (console.putc(c));
bryantaylor 0:eafc3fd41f75 528 #else /* NEED_TEST_REPORT_AS_CONSOLE_OUTPUT */
bryantaylor 0:eafc3fd41f75 529 return (0);
bryantaylor 0:eafc3fd41f75 530 #endif /* NEED_TEST_REPORT_AS_CONSOLE_OUTPUT */
bryantaylor 0:eafc3fd41f75 531 }
bryantaylor 0:eafc3fd41f75 532 ```
bryantaylor 0:eafc3fd41f75 533
bryantaylor 0:eafc3fd41f75 534 You can find cpputest UT test runner main function in mbed sources: ```c:/Projects/Project/mbed/libraries/tests/utest/testrunner/testrunner.cpp```. Test runner code (in ```testrunner.cpp```) only defined console object and executes all unit tests:
bryantaylor 0:eafc3fd41f75 535 ```c++
bryantaylor 0:eafc3fd41f75 536 #include "CommandLineTestRunner.h"
bryantaylor 0:eafc3fd41f75 537 #include <stdio.h>
bryantaylor 0:eafc3fd41f75 538 #include "mbed.h"
bryantaylor 0:eafc3fd41f75 539 #include "testrunner.h"
bryantaylor 0:eafc3fd41f75 540 #include "test_env.h"
bryantaylor 0:eafc3fd41f75 541
bryantaylor 0:eafc3fd41f75 542 /**
bryantaylor 0:eafc3fd41f75 543 Object 'mbed_cpputest_console' is used to show prints on console.
bryantaylor 0:eafc3fd41f75 544 It is declared in \cpputest\src\Platforms\armcc\UtestPlatform.cpp
bryantaylor 0:eafc3fd41f75 545 */
bryantaylor 0:eafc3fd41f75 546 Serial mbed_cpputest_console(STDIO_UART_TX, STDIO_UART_RX);
bryantaylor 0:eafc3fd41f75 547
bryantaylor 0:eafc3fd41f75 548 int main(int ac, char** av) {
bryantaylor 0:eafc3fd41f75 549 MBED_HOSTTEST_TIMEOUT(20);
bryantaylor 0:eafc3fd41f75 550 MBED_HOSTTEST_SELECT(default_auto);
bryantaylor 0:eafc3fd41f75 551 MBED_HOSTTEST_DESCRIPTION(Unit test);
bryantaylor 0:eafc3fd41f75 552 MBED_HOSTTEST_START("UT");
bryantaylor 0:eafc3fd41f75 553
bryantaylor 0:eafc3fd41f75 554 unsigned failureCount = 0;
bryantaylor 0:eafc3fd41f75 555 {
bryantaylor 0:eafc3fd41f75 556 // Some compilers may not pass ac, av so we need to supply them ourselves
bryantaylor 0:eafc3fd41f75 557 int ac = 2;
bryantaylor 0:eafc3fd41f75 558 char* av[] = {__FILE__, "-v"};
bryantaylor 0:eafc3fd41f75 559 failureCount = CommandLineTestRunner::RunAllTests(ac, av);
bryantaylor 0:eafc3fd41f75 560 }
bryantaylor 0:eafc3fd41f75 561
bryantaylor 0:eafc3fd41f75 562 MBED_HOSTTEST_RESULT(failureCount == 0);
bryantaylor 0:eafc3fd41f75 563 return failureCount;
bryantaylor 0:eafc3fd41f75 564 }
bryantaylor 0:eafc3fd41f75 565 ```
bryantaylor 0:eafc3fd41f75 566
bryantaylor 0:eafc3fd41f75 567 ## Unit test location
bryantaylor 0:eafc3fd41f75 568 Unit tests source code is located in below directory: ```c:/Projects/Project/mbed/libraries/tests/utest/```
bryantaylor 0:eafc3fd41f75 569
bryantaylor 0:eafc3fd41f75 570 Each sub directory except testrunner contains compilable unit test source files with test groups and test cases. You can see utest structure below. Please note this is just example and in the future this directory will contain many sub directories with unit tests.
bryantaylor 0:eafc3fd41f75 571 ```
bryantaylor 0:eafc3fd41f75 572 $ c:\Projects\Project\mbed\libraries\tests\utest> tree
bryantaylor 0:eafc3fd41f75 573 utest
bryantaylor 0:eafc3fd41f75 574 ├───basic
bryantaylor 0:eafc3fd41f75 575 ├───semihost_fs
bryantaylor 0:eafc3fd41f75 576 └───testrunner
bryantaylor 0:eafc3fd41f75 577 ```
bryantaylor 0:eafc3fd41f75 578
bryantaylor 0:eafc3fd41f75 579 ## Define unit tests in mbed SDK test suite structure
bryantaylor 0:eafc3fd41f75 580 All tests defined in test suite are described in ```mbed/tools/tests.py``` file. This file stores data structure ```TESTS``` which is a list of simple structures describing each test. Below you can find example of ```TESTS``` structure which is configuring one of the unit tests.
bryantaylor 0:eafc3fd41f75 581 ```
bryantaylor 0:eafc3fd41f75 582 .
bryantaylor 0:eafc3fd41f75 583 .
bryantaylor 0:eafc3fd41f75 584 .
bryantaylor 0:eafc3fd41f75 585 {
bryantaylor 0:eafc3fd41f75 586 "id": "UT_2", "description": "Semihost file system",
bryantaylor 0:eafc3fd41f75 587 "source_dir": join(TEST_DIR, "utest", "file"),
bryantaylor 0:eafc3fd41f75 588 "dependencies": [MBED_LIBRARIES, TEST_MBED_LIB, CPPUTEST_LIBRARY],
bryantaylor 0:eafc3fd41f75 589 "automated": False,
bryantaylor 0:eafc3fd41f75 590 "mcu": ["LPC1768", "LPC2368", "LPC11U24"]
bryantaylor 0:eafc3fd41f75 591 },
bryantaylor 0:eafc3fd41f75 592 .
bryantaylor 0:eafc3fd41f75 593 .
bryantaylor 0:eafc3fd41f75 594 .
bryantaylor 0:eafc3fd41f75 595 ```
bryantaylor 0:eafc3fd41f75 596 Note: In dependency section we've added library ```CPPUTEST_LIBRARY``` which is pointing build script to CppUTest library with mbed port. This is a must for unit tests to be compiled with CppUTest library.
bryantaylor 0:eafc3fd41f75 597
bryantaylor 0:eafc3fd41f75 598 ### Tests are now divided into two types:
bryantaylor 0:eafc3fd41f75 599 #### 'Hello world' tests
bryantaylor 0:eafc3fd41f75 600 First type of test cases we call 'hello world' tests. They do not dependent on CppUTest library and are monolithic programs usually composed of one main function. You can find this tests in below example directories:
bryantaylor 0:eafc3fd41f75 601
bryantaylor 0:eafc3fd41f75 602 * ```mbed/libraries/tests/mbed/```
bryantaylor 0:eafc3fd41f75 603 * ```mbed/libraries/tests/net/```
bryantaylor 0:eafc3fd41f75 604 * ```mbed/libraries/tests/rtos/```
bryantaylor 0:eafc3fd41f75 605 * ```mbed/libraries/tests/usb/```
bryantaylor 0:eafc3fd41f75 606
bryantaylor 0:eafc3fd41f75 607 Usually ‘hello world’ test cases are using ```test_env.cpp``` and ```test_env.h``` files which implement simple test framework used to communicate with host test and help test framework instrument your tests.
bryantaylor 0:eafc3fd41f75 608
bryantaylor 0:eafc3fd41f75 609 Below you can see listing of ```test_env.h``` file which contains simple macro definitions used to communicate (via serial port printouts) between test case (on hardware) and host test script (on host computer).
bryantaylor 0:eafc3fd41f75 610 Each use case should print on console basic information like:
bryantaylor 0:eafc3fd41f75 611 * Default test case timeout.
bryantaylor 0:eafc3fd41f75 612 * Which host test should be used to supervise test case execution.
bryantaylor 0:eafc3fd41f75 613 * Test description and test case ID (short identifier).
bryantaylor 0:eafc3fd41f75 614
bryantaylor 0:eafc3fd41f75 615 ```c++
bryantaylor 0:eafc3fd41f75 616 .
bryantaylor 0:eafc3fd41f75 617 .
bryantaylor 0:eafc3fd41f75 618 .
bryantaylor 0:eafc3fd41f75 619 // Test result related notification functions
bryantaylor 0:eafc3fd41f75 620 void notify_start();
bryantaylor 0:eafc3fd41f75 621 void notify_completion(bool success);
bryantaylor 0:eafc3fd41f75 622 bool notify_completion_str(bool success, char* buffer);
bryantaylor 0:eafc3fd41f75 623 void notify_performance_coefficient(const char* measurement_name, const int value);
bryantaylor 0:eafc3fd41f75 624 void notify_performance_coefficient(const char* measurement_name, const unsigned int value);
bryantaylor 0:eafc3fd41f75 625 void notify_performance_coefficient(const char* measurement_name, const double value);
bryantaylor 0:eafc3fd41f75 626
bryantaylor 0:eafc3fd41f75 627 // Host test auto-detection API
bryantaylor 0:eafc3fd41f75 628 void notify_host_test_name(const char *host_test);
bryantaylor 0:eafc3fd41f75 629 void notify_timeout(int timeout);
bryantaylor 0:eafc3fd41f75 630 void notify_test_id(const char *test_id);
bryantaylor 0:eafc3fd41f75 631 void notify_test_description(const char *description);
bryantaylor 0:eafc3fd41f75 632
bryantaylor 0:eafc3fd41f75 633 // Host test auto-detection API
bryantaylor 0:eafc3fd41f75 634 #define MBED_HOSTTEST_START(TESTID) notify_test_id(TESTID); notify_start()
bryantaylor 0:eafc3fd41f75 635 #define MBED_HOSTTEST_SELECT(NAME) notify_host_test_name(#NAME)
bryantaylor 0:eafc3fd41f75 636 #define MBED_HOSTTEST_TIMEOUT(SECONDS) notify_timeout(SECONDS)
bryantaylor 0:eafc3fd41f75 637 #define MBED_HOSTTEST_DESCRIPTION(DESC) notify_test_description(#DESC)
bryantaylor 0:eafc3fd41f75 638 #define MBED_HOSTTEST_RESULT(RESULT) notify_completion(RESULT)
bryantaylor 0:eafc3fd41f75 639
bryantaylor 0:eafc3fd41f75 640 /**
bryantaylor 0:eafc3fd41f75 641 Test auto-detection preamble example:
bryantaylor 0:eafc3fd41f75 642 main() {
bryantaylor 0:eafc3fd41f75 643 MBED_HOSTTEST_TIMEOUT(10);
bryantaylor 0:eafc3fd41f75 644 MBED_HOSTTEST_SELECT( host_test );
bryantaylor 0:eafc3fd41f75 645 MBED_HOSTTEST_DESCRIPTION(Hello World);
bryantaylor 0:eafc3fd41f75 646 MBED_HOSTTEST_START("MBED_10");
bryantaylor 0:eafc3fd41f75 647 // Proper 'host_test.py' should take over supervising of this test
bryantaylor 0:eafc3fd41f75 648
bryantaylor 0:eafc3fd41f75 649 // Test code
bryantaylor 0:eafc3fd41f75 650 bool result = ...;
bryantaylor 0:eafc3fd41f75 651
bryantaylor 0:eafc3fd41f75 652 MBED_HOSTTEST_RESULT(result);
bryantaylor 0:eafc3fd41f75 653 }
bryantaylor 0:eafc3fd41f75 654 */
bryantaylor 0:eafc3fd41f75 655 .
bryantaylor 0:eafc3fd41f75 656 .
bryantaylor 0:eafc3fd41f75 657 .
bryantaylor 0:eafc3fd41f75 658 ```
bryantaylor 0:eafc3fd41f75 659
bryantaylor 0:eafc3fd41f75 660 Example of 'hello world' test:
bryantaylor 0:eafc3fd41f75 661 ```c++
bryantaylor 0:eafc3fd41f75 662 #include "mbed.h"
bryantaylor 0:eafc3fd41f75 663 #include "test_env.h"
bryantaylor 0:eafc3fd41f75 664
bryantaylor 0:eafc3fd41f75 665 #define CUSTOM_TIME 1256729737
bryantaylor 0:eafc3fd41f75 666
bryantaylor 0:eafc3fd41f75 667 int main() {
bryantaylor 0:eafc3fd41f75 668 MBED_HOSTTEST_TIMEOUT(20);
bryantaylor 0:eafc3fd41f75 669 MBED_HOSTTEST_SELECT(rtc_auto);
bryantaylor 0:eafc3fd41f75 670 MBED_HOSTTEST_DESCRIPTION(RTC);
bryantaylor 0:eafc3fd41f75 671 MBED_HOSTTEST_START("MBED_16");
bryantaylor 0:eafc3fd41f75 672
bryantaylor 0:eafc3fd41f75 673 char buffer[32] = {0};
bryantaylor 0:eafc3fd41f75 674 set_time(CUSTOM_TIME); // Set RTC time to Wed, 28 Oct 2009 11:35:37
bryantaylor 0:eafc3fd41f75 675 while(1) {
bryantaylor 0:eafc3fd41f75 676 time_t seconds = time(NULL);
bryantaylor 0:eafc3fd41f75 677 strftime(buffer, 32, "%Y-%m-%d %H:%M:%S %p", localtime(&seconds));
bryantaylor 0:eafc3fd41f75 678 printf("MBED: [%ld] [%s]\r\n", seconds, buffer);
bryantaylor 0:eafc3fd41f75 679 wait(1);
bryantaylor 0:eafc3fd41f75 680 }
bryantaylor 0:eafc3fd41f75 681 }
bryantaylor 0:eafc3fd41f75 682 ```
bryantaylor 0:eafc3fd41f75 683
bryantaylor 0:eafc3fd41f75 684 #### 'Unit test' test cases
bryantaylor 0:eafc3fd41f75 685 Second group of tests are unit tests. They are using CppUTest library and require you to write ```TEST_GROUP```s and ```TEST```s in your test files. Test suite will add test runner sources to your test automatically so you can concentrate on writing tests.
bryantaylor 0:eafc3fd41f75 686
bryantaylor 0:eafc3fd41f75 687 Example of unit test:
bryantaylor 0:eafc3fd41f75 688 ```c++
bryantaylor 0:eafc3fd41f75 689 #include "TestHarness.h"
bryantaylor 0:eafc3fd41f75 690 #include <utility>
bryantaylor 0:eafc3fd41f75 691 #include "mbed.h"
bryantaylor 0:eafc3fd41f75 692
bryantaylor 0:eafc3fd41f75 693 TEST_GROUP(BusOut_mask)
bryantaylor 0:eafc3fd41f75 694 {
bryantaylor 0:eafc3fd41f75 695 };
bryantaylor 0:eafc3fd41f75 696
bryantaylor 0:eafc3fd41f75 697 TEST(BusOut_mask, led_1_2_3)
bryantaylor 0:eafc3fd41f75 698 {
bryantaylor 0:eafc3fd41f75 699 BusOut bus_data(LED1, LED2, LED3);
bryantaylor 0:eafc3fd41f75 700 CHECK_EQUAL(0x07, bus_data.mask());
bryantaylor 0:eafc3fd41f75 701 }
bryantaylor 0:eafc3fd41f75 702
bryantaylor 0:eafc3fd41f75 703 TEST(BusOut_mask, led_nc_nc_nc_nc)
bryantaylor 0:eafc3fd41f75 704 {
bryantaylor 0:eafc3fd41f75 705 BusOut bus_data(NC, NC, NC, NC);
bryantaylor 0:eafc3fd41f75 706 CHECK_EQUAL(0x00, bus_data.mask());
bryantaylor 0:eafc3fd41f75 707 }
bryantaylor 0:eafc3fd41f75 708
bryantaylor 0:eafc3fd41f75 709 TEST(BusOut_mask, led_1_2_3_nc_nc)
bryantaylor 0:eafc3fd41f75 710 {
bryantaylor 0:eafc3fd41f75 711 BusOut bus_data(LED1, LED2, LED3, NC, NC);
bryantaylor 0:eafc3fd41f75 712 CHECK_EQUAL(0x07, bus_data.mask());
bryantaylor 0:eafc3fd41f75 713 }
bryantaylor 0:eafc3fd41f75 714
bryantaylor 0:eafc3fd41f75 715 TEST(BusOut_mask, led_1_nc_2_nc_nc_3)
bryantaylor 0:eafc3fd41f75 716 {
bryantaylor 0:eafc3fd41f75 717 BusOut bus_data(LED1, NC, LED2, NC, NC, LED3);
bryantaylor 0:eafc3fd41f75 718 CHECK_EQUAL(0x25, bus_data.mask());
bryantaylor 0:eafc3fd41f75 719 }
bryantaylor 0:eafc3fd41f75 720 ```
bryantaylor 0:eafc3fd41f75 721
bryantaylor 0:eafc3fd41f75 722 ## Example
bryantaylor 0:eafc3fd41f75 723 In below example we will run two example unit tests that are now available. tests ```UT_1``` and ```UT_2``` are unit tests used for now only to check if mbed SDK works with CppUTest library and if tests are being executed. In future number of unit tests will increase, nothing is also stopping you from writing and executing your own unit tests!
bryantaylor 0:eafc3fd41f75 724
bryantaylor 0:eafc3fd41f75 725 ### Example configuration
bryantaylor 0:eafc3fd41f75 726 By default unit tests ```UT_1``` and ```UT_2``` are not automated - simply test suite will ignore them. Also we do not want to create dependency to CppUTest library each time someone executes automation.
bryantaylor 0:eafc3fd41f75 727
bryantaylor 0:eafc3fd41f75 728 Note: To compile ```UT_1``` and ```UT_2``` tests CppUTest library described above, installation is needed and not all users wish to add UT libs to their project. Also new to mbed users may find it difficult. This is why unit testing is an extra feature active only after you deliberately install and enable needed components.
bryantaylor 0:eafc3fd41f75 729
bryantaylor 0:eafc3fd41f75 730 Bellow snippet shows how to modify 'automated' flag so test suite will consider unit tests ```UT_1``` and ```UT_2``` as part of "automated test portfolio". Just change flag 'automated' from ```False``` to ```True```.
bryantaylor 0:eafc3fd41f75 731
bryantaylor 0:eafc3fd41f75 732 ```tests.py``` listing related to ```UT_1``` and ```UT_2```:
bryantaylor 0:eafc3fd41f75 733 ```python
bryantaylor 0:eafc3fd41f75 734 .
bryantaylor 0:eafc3fd41f75 735 .
bryantaylor 0:eafc3fd41f75 736 .
bryantaylor 0:eafc3fd41f75 737 # CPPUTEST Library provides Unit testing Framework
bryantaylor 0:eafc3fd41f75 738 #
bryantaylor 0:eafc3fd41f75 739 # To write TESTs and TEST_GROUPs please add CPPUTEST_LIBRARY to 'dependencies'
bryantaylor 0:eafc3fd41f75 740 #
bryantaylor 0:eafc3fd41f75 741 # This will also include:
bryantaylor 0:eafc3fd41f75 742 # 1. test runner - main function with call to CommandLineTestRunner::RunAllTests(ac, av)
bryantaylor 0:eafc3fd41f75 743 # 2. Serial console object to print test result on serial port console
bryantaylor 0:eafc3fd41f75 744 #
bryantaylor 0:eafc3fd41f75 745
bryantaylor 0:eafc3fd41f75 746 # Unit testing with cpputest library
bryantaylor 0:eafc3fd41f75 747 {
bryantaylor 0:eafc3fd41f75 748 "id": "UT_1", "description": "Basic",
bryantaylor 0:eafc3fd41f75 749 "source_dir": join(TEST_DIR, "utest", "basic"),
bryantaylor 0:eafc3fd41f75 750 "dependencies": [MBED_LIBRARIES, TEST_MBED_LIB, CPPUTEST_LIBRARY],
bryantaylor 0:eafc3fd41f75 751 "automated": True,
bryantaylor 0:eafc3fd41f75 752 },
bryantaylor 0:eafc3fd41f75 753 {
bryantaylor 0:eafc3fd41f75 754 "id": "UT_2", "description": "Semihost file system",
bryantaylor 0:eafc3fd41f75 755 "source_dir": join(TEST_DIR, "utest", "semihost_fs"),
bryantaylor 0:eafc3fd41f75 756 "dependencies": [MBED_LIBRARIES, TEST_MBED_LIB, CPPUTEST_LIBRARY],
bryantaylor 0:eafc3fd41f75 757 "automated": True,
bryantaylor 0:eafc3fd41f75 758 "mcu": ["LPC1768", "LPC2368", "LPC11U24"]
bryantaylor 0:eafc3fd41f75 759 },
bryantaylor 0:eafc3fd41f75 760 .
bryantaylor 0:eafc3fd41f75 761 .
bryantaylor 0:eafc3fd41f75 762 .
bryantaylor 0:eafc3fd41f75 763 ```
bryantaylor 0:eafc3fd41f75 764
bryantaylor 0:eafc3fd41f75 765 ### Execute tests
bryantaylor 0:eafc3fd41f75 766 In my test I will use common [LPC1768](http://developer.mbed.org/platforms/mbed-LPC1768/) mbed-enabled board because unit test ```UT_2``` is checking semi-host functionality which is available on this board and handful of others.
bryantaylor 0:eafc3fd41f75 767
bryantaylor 0:eafc3fd41f75 768 Configure your ```test_spec.json``` and ```muts_all.json``` files (refer to test suite build script and automation description) and set mbed disk and serial port.
bryantaylor 0:eafc3fd41f75 769
bryantaylor 0:eafc3fd41f75 770 ```
bryantaylor 0:eafc3fd41f75 771 $ singletest.py -i test_spec.json -M muts_all.json -n UT_1,UT_2 -V
bryantaylor 0:eafc3fd41f75 772 Building library CMSIS (LPC1768, ARM)
bryantaylor 0:eafc3fd41f75 773 Building library MBED (LPC1768, ARM)
bryantaylor 0:eafc3fd41f75 774 Building library CPPUTEST (LPC1768, ARM)
bryantaylor 0:eafc3fd41f75 775 Building project BASIC (LPC1768, ARM)
bryantaylor 0:eafc3fd41f75 776 Executing 'python host_test.py -p COM77 -d E:\ -t 10'
bryantaylor 0:eafc3fd41f75 777 Test::Output::Start
bryantaylor 0:eafc3fd41f75 778 Host test instrumentation on port: "COM77" and disk: "E:\"
bryantaylor 0:eafc3fd41f75 779 TEST(FirstTestGroup, FirstTest) - 0 ms
bryantaylor 0:eafc3fd41f75 780
bryantaylor 0:eafc3fd41f75 781 OK (1 tests, 1 ran, 3 checks, 0 ignored, 0 filtered out, 3 ms)
bryantaylor 0:eafc3fd41f75 782
bryantaylor 0:eafc3fd41f75 783 {{success}}
bryantaylor 0:eafc3fd41f75 784 {{end}}
bryantaylor 0:eafc3fd41f75 785 Test::Output::Finish
bryantaylor 0:eafc3fd41f75 786 TargetTest::LPC1768::ARM::UT_1::Basic [OK] in 2.43 of 10 sec
bryantaylor 0:eafc3fd41f75 787 Building library CPPUTEST (LPC1768, ARM)
bryantaylor 0:eafc3fd41f75 788 Building project SEMIHOST_FS (LPC1768, ARM)
bryantaylor 0:eafc3fd41f75 789 Executing 'python host_test.py -p COM77 -d E:\ -t 10'
bryantaylor 0:eafc3fd41f75 790 Test::Output::Start
bryantaylor 0:eafc3fd41f75 791 Host test instrumentation on port: "COM77" and disk: "E:\"
bryantaylor 0:eafc3fd41f75 792 TEST(FirstTestGroup, FirstTest) - 9 ms
bryantaylor 0:eafc3fd41f75 793
bryantaylor 0:eafc3fd41f75 794 OK (1 tests, 1 ran, 10 checks, 0 ignored, 0 filtered out, 10 ms)
bryantaylor 0:eafc3fd41f75 795
bryantaylor 0:eafc3fd41f75 796 {{success}}
bryantaylor 0:eafc3fd41f75 797 {{end}}
bryantaylor 0:eafc3fd41f75 798 Test::Output::Finish
bryantaylor 0:eafc3fd41f75 799 TargetTest::LPC1768::ARM::UT_2::Semihost file system [OK] in 2.43 of 10 sec
bryantaylor 0:eafc3fd41f75 800 Test summary:
bryantaylor 0:eafc3fd41f75 801 +--------+---------+-----------+---------+----------------------+--------------------+---------------+-------+
bryantaylor 0:eafc3fd41f75 802 | Result | Target | Toolchain | Test ID | Test Description | Elapsed Time (sec) | Timeout (sec) | Loops |
bryantaylor 0:eafc3fd41f75 803 +--------+---------+-----------+---------+----------------------+--------------------+---------------+-------+
bryantaylor 0:eafc3fd41f75 804 | OK | LPC1768 | ARM | UT_1 | Basic | 2.43 | 10 | 1/1 |
bryantaylor 0:eafc3fd41f75 805 | OK | LPC1768 | ARM | UT_2 | Semihost file system | 2.43 | 10 | 1/1 |
bryantaylor 0:eafc3fd41f75 806 +--------+---------+-----------+---------+----------------------+--------------------+---------------+-------+
bryantaylor 0:eafc3fd41f75 807 Result: 2 OK
bryantaylor 0:eafc3fd41f75 808
bryantaylor 0:eafc3fd41f75 809 Completed in 12.02 sec
bryantaylor 0:eafc3fd41f75 810 ```
bryantaylor 0:eafc3fd41f75 811
bryantaylor 0:eafc3fd41f75 812 You can compile unit tests using various number of supported compilers, below just few examples with working solutions:
bryantaylor 0:eafc3fd41f75 813 ```
bryantaylor 0:eafc3fd41f75 814 Test summary:
bryantaylor 0:eafc3fd41f75 815 +--------+---------+-----------+---------+----------------------+--------------------+---------------+-------+
bryantaylor 0:eafc3fd41f75 816 | Result | Target | Toolchain | Test ID | Test Description | Elapsed Time (sec) | Timeout (sec) | Loops |
bryantaylor 0:eafc3fd41f75 817 +--------+---------+-----------+---------+----------------------+--------------------+---------------+-------+
bryantaylor 0:eafc3fd41f75 818 | OK | LPC1768 | ARM | UT_1 | Basic | 2.43 | 10 | 1/1 |
bryantaylor 0:eafc3fd41f75 819 | OK | LPC1768 | ARM | UT_2 | Semihost file system | 2.43 | 10 | 1/1 |
bryantaylor 0:eafc3fd41f75 820 | OK | LPC1768 | uARM | UT_1 | Basic | 2.43 | 10 | 1/1 |
bryantaylor 0:eafc3fd41f75 821 | OK | LPC1768 | uARM | UT_2 | Semihost file system | 2.43 | 10 | 1/1 |
bryantaylor 0:eafc3fd41f75 822 | OK | LPC1768 | GCC_ARM | UT_1 | Basic | 2.43 | 10 | 1/1 |
bryantaylor 0:eafc3fd41f75 823 | OK | LPC1768 | GCC_ARM | UT_2 | Semihost file system | 2.43 | 10 | 1/1 |
bryantaylor 0:eafc3fd41f75 824 | OK | LPC1768 | GCC_CR | UT_1 | Basic | 3.44 | 10 | 1/1 |
bryantaylor 0:eafc3fd41f75 825 | OK | LPC1768 | GCC_CR | UT_2 | Semihost file system | 3.43 | 10 | 1/1 |
bryantaylor 0:eafc3fd41f75 826 +--------+---------+-----------+---------+----------------------+--------------------+---------------+-------+
bryantaylor 0:eafc3fd41f75 827 Result: 8 OK
bryantaylor 0:eafc3fd41f75 828
bryantaylor 0:eafc3fd41f75 829 Completed in 55.85 sec
bryantaylor 0:eafc3fd41f75 830 ```