ON Semiconductor / mbed-os

5.2.1 - Updated I2C files

Dependents:   mbed-TFT-example-NCS36510 mbed-Accelerometer-example-NCS36510 mbed-Accelerometer-example-NCS36510

docs/testing_mbed_OS_5.md@0:098463de4c5d, 2017-01-25 (annotated)

Committer:
group-onsemi
Date:
Wed Jan 25 20:34:15 2017 +0000
Revision:
0:098463de4c5d
Initial commit

Who changed what in which revision?

UserRevisionLine numberNew contents of line
group-onsemi 0:098463de4c5d 1 # Testing in mbed OS 5
group-onsemi 0:098463de4c5d 2
group-onsemi 0:098463de4c5d 3 The way tests are run and compiled in mbed OS 5 is substantially different from previous versions of mbed. Previously, tests were located in one known location and a python file (`tools/tests.py`) kept track of their dependencies, capabilities, and configurations. mbed OS 5 has adopted a more distributed approach to testing. Test code lives alongside the application code, and which is dynamically discovered by the test tools.
group-onsemi 0:098463de4c5d 4
group-onsemi 0:098463de4c5d 5 ## Table of Contents
group-onsemi 0:098463de4c5d 6
group-onsemi 0:098463de4c5d 7 - [Using tests](#using-tests)
group-onsemi 0:098463de4c5d 8 - [Test code structure](#test-code-structure)
group-onsemi 0:098463de4c5d 9 - [Test discovery](#test-discovery)
group-onsemi 0:098463de4c5d 10 - [Test names](#test-names)
group-onsemi 0:098463de4c5d 11 - [Building tests](#building-tests)
group-onsemi 0:098463de4c5d 12 - [Building process](#building-process)
group-onsemi 0:098463de4c5d 13 - [App config](#app-config)
group-onsemi 0:098463de4c5d 14 - [Running tests](#running-tests)
group-onsemi 0:098463de4c5d 15 - [Writing tests](#writing-tests)
group-onsemi 0:098463de4c5d 16 - [Debugging tests](#debugging-tests)
group-onsemi 0:098463de4c5d 17 - [Exporting tests](#exporting-tests)
group-onsemi 0:098463de4c5d 18 - [Running a test while debugging](#running-a-test-while-debugging)
group-onsemi 0:098463de4c5d 19 - [Known issues](#known-issues)
group-onsemi 0:098463de4c5d 20
group-onsemi 0:098463de4c5d 21 ## Using tests
group-onsemi 0:098463de4c5d 22
group-onsemi 0:098463de4c5d 23 ### Test code structure
group-onsemi 0:098463de4c5d 24
group-onsemi 0:098463de4c5d 25 Tests can exist throughout mbed OS and your project's code. They are located under a special directory called `TESTS` (case is important!).
group-onsemi 0:098463de4c5d 26
group-onsemi 0:098463de4c5d 27 Placing code under this directory means it will be ignored when building applications and libraries. This code is only ever used when building tests. This is important since all tests require a `main()` function, and building it with your application would cause multiple `main()` functions to be defined.
group-onsemi 0:098463de4c5d 28
group-onsemi 0:098463de4c5d 29 In addition to being placed under a `TESTS` directory, test sources must exist under two other directories: a test group directory and a test case directory. The following are an examples of this structure:
group-onsemi 0:098463de4c5d 30 ```
group-onsemi 0:098463de4c5d 31 myproject/TESTS/test_group/test_case_1
group-onsemi 0:098463de4c5d 32 ```
group-onsemi 0:098463de4c5d 33
group-onsemi 0:098463de4c5d 34 In this example, `myproject` is the project root and all the source files under the `test_case_1` directory will be included in the test. Any other source files from the OS, libraries, and your project that apply to your target's configuration will also be included in the build of your test.
group-onsemi 0:098463de4c5d 35
group-onsemi 0:098463de4c5d 36 **Note:** Both the test group and test case directory can be named anything you like. However, the `TESTS` directory **must** be named `TESTS` for the tools to detect the test cases correctly.
group-onsemi 0:098463de4c5d 37
group-onsemi 0:098463de4c5d 38 #### Test discovery
group-onsemi 0:098463de4c5d 39
group-onsemi 0:098463de4c5d 40 Since test cases can exist throughout a project, the tools must find them in your project's file structure before building them. This is done by searching for paths that match the pattern detailed above in the [Test code structure](#test-code-structure) section.
group-onsemi 0:098463de4c5d 41
group-onsemi 0:098463de4c5d 42 Test discovery also obeys the same rules that are used when building your project. This means that tests that are placed under a directory with a prefix like `TARGET_`, `TOOLCHAIN_`, or `FEATURE_` will only be discovered, built, and run if your current configuration matches this prefix.
group-onsemi 0:098463de4c5d 43
group-onsemi 0:098463de4c5d 44 For example, if you place a test under the directory `FEATURE_BLE` with the following path:
group-onsemi 0:098463de4c5d 45
group-onsemi 0:098463de4c5d 46 ```
group-onsemi 0:098463de4c5d 47 myproject/mbed-os/features/FEATURE_BLE/TESTS/ble_tests/unit_test
group-onsemi 0:098463de4c5d 48 ```
group-onsemi 0:098463de4c5d 49
group-onsemi 0:098463de4c5d 50 This test case will only be discovered if the target being testing supports the BLE feature. Otherwise, the test will be ignored.
group-onsemi 0:098463de4c5d 51
group-onsemi 0:098463de4c5d 52 Generally, a test should not be placed under a `TARGET_` or `TOOLCHAIN_` directory, since most tests should be designed to work for all target and toolchain configurations.
group-onsemi 0:098463de4c5d 53
group-onsemi 0:098463de4c5d 54 Tests can also be completely ignored by using the `.mbedignore` file described [here](../ignoring_files_from_build.md)
group-onsemi 0:098463de4c5d 55
group-onsemi 0:098463de4c5d 56 #### Test names
group-onsemi 0:098463de4c5d 57
group-onsemi 0:098463de4c5d 58 A test case is named from its position in your project's file structure. For instance, in the above example, a test case with the path `myproject/TESTS/test_group/test_case_1` would be named `tests-test_group-test_case_1`. You will notice that the name is created by joining the directories that make up the path to the test case with a "dash" (`-`) character. This will be a unique name to identify the test case. You will see this name used throughout the build and testing process.
group-onsemi 0:098463de4c5d 59
group-onsemi 0:098463de4c5d 60 ### Building tests
group-onsemi 0:098463de4c5d 61
group-onsemi 0:098463de4c5d 62 Tests can be built easily through mbed CLI. For information on using mbed CLI, please see its [documentation](https://github.com/ARMmbed/mbed-cli).
group-onsemi 0:098463de4c5d 63
group-onsemi 0:098463de4c5d 64 When tests are built for a target and a given toolchain, the available tests are first discovered, then built in series. You can also create a "test specification" file, which can be used by our testing tools to run automated hardware tests. For more information on the test specification file, please see the documentation [here](https://github.com/ARMmbed/greentea#test-specification-json-formatted-input).
group-onsemi 0:098463de4c5d 65
group-onsemi 0:098463de4c5d 66 #### Building process
group-onsemi 0:098463de4c5d 67
group-onsemi 0:098463de4c5d 68 The process for building tests is handled by the `test.py` script (not to be confused with `tests.py`) located under the `tools` directory. This handles the discovery and building of all test cases for a given target and toolchain.
group-onsemi 0:098463de4c5d 69
group-onsemi 0:098463de4c5d 70 The full build process is:
group-onsemi 0:098463de4c5d 71
group-onsemi 0:098463de4c5d 72 1. Build the non-test code (all code not under a `TESTS` folder), but do not link it. The resulting object files are placed in the build directory.
group-onsemi 0:098463de4c5d 73 1. Find all tests that match the given target and toolchain.
group-onsemi 0:098463de4c5d 74 1. For each discovered test, build all of its source files and link it with the non-test code that was built in step 1.
group-onsemi 0:098463de4c5d 75 1. If specified, create a test specification file and place it in the given directory for use by testing tools. This is placed in the build directory by default when using mbed CLI.
group-onsemi 0:098463de4c5d 76
group-onsemi 0:098463de4c5d 77 #### App config
group-onsemi 0:098463de4c5d 78
group-onsemi 0:098463de4c5d 79 When building an mbed application, the presence of a `mbed_app.json` file allows you to set or override different config settings from libraries and targets. However, because the tests share a common build, this can cause issues when tests have different configurations that affect the OS.
group-onsemi 0:098463de4c5d 80
group-onsemi 0:098463de4c5d 81 The build system will look for an `mbed_app.json` file in your shared project files (any directory not inside of a `TESTS` folder). If this is found, this configuration file will be used for both the non-test code as well as each test case inside your project's source tree. If there is more than one `mbed_app.json` files in the source tree, the config system will error.
group-onsemi 0:098463de4c5d 82
group-onsemi 0:098463de4c5d 83 If you need to test with multiple configurations, then you can use the `--app-config` option. This will override the search for an `mbed_app.json` file and use the config file you specify for the build.
group-onsemi 0:098463de4c5d 84
group-onsemi 0:098463de4c5d 85 ### Running tests
group-onsemi 0:098463de4c5d 86
group-onsemi 0:098463de4c5d 87 Automated tests can be run easily through mbed CLI. For information on using mbed CLI, please see its documentation.
group-onsemi 0:098463de4c5d 88
group-onsemi 0:098463de4c5d 89 The testing process requires that the tests are built and that a test specification JSON file exists that describes these available tests. The test specification format is detailed [here](https://github.com/ARMmbed/greentea#test-specification-json-formatted-input).
group-onsemi 0:098463de4c5d 90
group-onsemi 0:098463de4c5d 91 The actual testing process is handled by the Greentea tool. To read more about this tool, please visit its [GitHub repository](https://github.com/ARMmbed/greentea).
group-onsemi 0:098463de4c5d 92
group-onsemi 0:098463de4c5d 93 ### Writing tests
group-onsemi 0:098463de4c5d 94
group-onsemi 0:098463de4c5d 95 You can write tests for your own project, or add more tests to mbed OS. Tests are written using the [Greentea client](https://github.com/ARMmbed/mbed-os/tree/master/features/frameworks/greentea-client), [UNITY](https://github.com/ARMmbed/mbed-os/tree/master/features/frameworks/unity), and [utest](https://github.com/ARMmbed/mbed-os/tree/master/features/frameworks/utest) frameworks, located in `/features/frameworks`. Below is an example test that uses all of these frameworks:
group-onsemi 0:098463de4c5d 96
group-onsemi 0:098463de4c5d 97 ```c++
group-onsemi 0:098463de4c5d 98 #include "mbed.h"
group-onsemi 0:098463de4c5d 99 #include "greentea-client/test_env.h"
group-onsemi 0:098463de4c5d 100 #include "unity.h"
group-onsemi 0:098463de4c5d 101 #include "utest.h"
group-onsemi 0:098463de4c5d 102 #include "rtos.h"
group-onsemi 0:098463de4c5d 103
group-onsemi 0:098463de4c5d 104 using namespace utest::v1;
group-onsemi 0:098463de4c5d 105
group-onsemi 0:098463de4c5d 106 // A test that returns successfully is considered successful
group-onsemi 0:098463de4c5d 107 void test_success() {
group-onsemi 0:098463de4c5d 108 TEST_ASSERT(true);
group-onsemi 0:098463de4c5d 109 }
group-onsemi 0:098463de4c5d 110
group-onsemi 0:098463de4c5d 111 // Tests that assert are considered failing
group-onsemi 0:098463de4c5d 112 void test_failure() {
group-onsemi 0:098463de4c5d 113 TEST_ASSERT(false);
group-onsemi 0:098463de4c5d 114 }
group-onsemi 0:098463de4c5d 115
group-onsemi 0:098463de4c5d 116 utest::v1::status_t test_setup(const size_t number_of_cases) {
group-onsemi 0:098463de4c5d 117 // Setup Greentea using a reasonable timeout in seconds
group-onsemi 0:098463de4c5d 118 GREENTEA_SETUP(40, "default_auto");
group-onsemi 0:098463de4c5d 119 return verbose_test_setup_handler(number_of_cases);
group-onsemi 0:098463de4c5d 120 }
group-onsemi 0:098463de4c5d 121
group-onsemi 0:098463de4c5d 122 // Test cases
group-onsemi 0:098463de4c5d 123 Case cases[] = {
group-onsemi 0:098463de4c5d 124 Case("Testing success test", test_success),
group-onsemi 0:098463de4c5d 125 Case("Testing failure test", test_failure),
group-onsemi 0:098463de4c5d 126 };
group-onsemi 0:098463de4c5d 127
group-onsemi 0:098463de4c5d 128 Specification specification(test_setup, cases);
group-onsemi 0:098463de4c5d 129
group-onsemi 0:098463de4c5d 130 // Entry point into the tests
group-onsemi 0:098463de4c5d 131 int main() {
group-onsemi 0:098463de4c5d 132 return !Harness::run(specification);
group-onsemi 0:098463de4c5d 133 }
group-onsemi 0:098463de4c5d 134 ```
group-onsemi 0:098463de4c5d 135
group-onsemi 0:098463de4c5d 136 This test will first run a case that succeeds, then a case that fails. This is a good template to use when creating tests. For more complex testing examples, please see the documentation for [utest](https://github.com/ARMmbed/mbed-os/tree/master/features/frameworks/utest).
group-onsemi 0:098463de4c5d 137
group-onsemi 0:098463de4c5d 138 ## Debugging tests
group-onsemi 0:098463de4c5d 139
group-onsemi 0:098463de4c5d 140 Debugging tests is a crucial part of the development and porting process. This section will cover exporting the test, then driving the test with the test tools while the target is attached to a debugger.
group-onsemi 0:098463de4c5d 141
group-onsemi 0:098463de4c5d 142 ### Exporting tests
group-onsemi 0:098463de4c5d 143
group-onsemi 0:098463de4c5d 144 Currently, the easiest way to export a test is to copy the test's source code from its test directory to your project's root. This way it will be treated like a normal application by the tools.
group-onsemi 0:098463de4c5d 145
group-onsemi 0:098463de4c5d 146 You can find the path to the test you wish to export by running the following command:
group-onsemi 0:098463de4c5d 147
group-onsemi 0:098463de4c5d 148 ```
group-onsemi 0:098463de4c5d 149 mbed test --compile-list -n <test name>
group-onsemi 0:098463de4c5d 150 ```
group-onsemi 0:098463de4c5d 151
group-onsemi 0:098463de4c5d 152 Once you've copied all of the test's source files to your project root, go ahead and export your project:
group-onsemi 0:098463de4c5d 153
group-onsemi 0:098463de4c5d 154 ```
group-onsemi 0:098463de4c5d 155 mbed export -i <IDE name>
group-onsemi 0:098463de4c5d 156 ```
group-onsemi 0:098463de4c5d 157
group-onsemi 0:098463de4c5d 158 Your exported project should now be in `projectfiles/<IDE>_<target>`. Go ahead and open this project in your IDE.
group-onsemi 0:098463de4c5d 159
group-onsemi 0:098463de4c5d 160 ### Running a test while debugging
group-onsemi 0:098463de4c5d 161
group-onsemi 0:098463de4c5d 162 Assuming your test was exported correctly to your IDE, go ahead and build the project and load it onto your target via your debugger.
group-onsemi 0:098463de4c5d 163
group-onsemi 0:098463de4c5d 164 Bring the target out of reset and run the program. Your target will now be waiting for a synchronizing character string to be sent from the test tools over the serial port. Do not run the `mbed test` commands, because that will attempt to flash the device, which you've already done with your IDE.
group-onsemi 0:098463de4c5d 165
group-onsemi 0:098463de4c5d 166 Instead, the underlying test tools can be used to drive the test. [htrun](https://github.com/ARMmbed/htrun) is the tool that needs to be used in this case. This is installed when you install the requirements for mbed OS. However, if you do not have it installed you can do this by running `pip install mbed-host-tests`.
group-onsemi 0:098463de4c5d 167
group-onsemi 0:098463de4c5d 168 First, find your target's serial port by running the following command:
group-onsemi 0:098463de4c5d 169
group-onsemi 0:098463de4c5d 170 ```
group-onsemi 0:098463de4c5d 171 $ mbed detect
group-onsemi 0:098463de4c5d 172
group-onsemi 0:098463de4c5d 173 [mbed] Detected KL46Z, port COM270, mounted D:
group-onsemi 0:098463de4c5d 174
group-onsemi 0:098463de4c5d 175 ...
group-onsemi 0:098463de4c5d 176 ```
group-onsemi 0:098463de4c5d 177
group-onsemi 0:098463de4c5d 178 From the output, take note of your target's serial port (in this case, it's `COM270`).
group-onsemi 0:098463de4c5d 179
group-onsemi 0:098463de4c5d 180 Run the following command when your device is running the test in your debugger:
group-onsemi 0:098463de4c5d 181
group-onsemi 0:098463de4c5d 182 ```
group-onsemi 0:098463de4c5d 183 mbedhtrun --skip-flashing --skip-reset -p <serial port>:9600
group-onsemi 0:098463de4c5d 184 ```
group-onsemi 0:098463de4c5d 185
group-onsemi 0:098463de4c5d 186 Replace `<serial port>` with the serial port you found by running `mbed detect` above.
group-onsemi 0:098463de4c5d 187
group-onsemi 0:098463de4c5d 188 So for the example above, the command would be:
group-onsemi 0:098463de4c5d 189
group-onsemi 0:098463de4c5d 190 ```
group-onsemi 0:098463de4c5d 191 mbedhtrun --skip-flashing --skip-reset -p COM270:9600
group-onsemi 0:098463de4c5d 192 ```
group-onsemi 0:098463de4c5d 193
group-onsemi 0:098463de4c5d 194 This detects your attached target and drives the test. At this point the test will proceed and allow you to debug it. If you need to rerun the test, simply reset the device with your debugger, run the program, and run the same command.
group-onsemi 0:098463de4c5d 195
group-onsemi 0:098463de4c5d 196 For an explanation of the arguments used in this command, please run `mbedhtrun --help`.
group-onsemi 0:098463de4c5d 197
group-onsemi 0:098463de4c5d 198 ## Known issues
group-onsemi 0:098463de4c5d 199
group-onsemi 0:098463de4c5d 200 - There cannot be a `main()` function outside of a `TESTS` directory when building and running tests. This is because this function will be included in the non-test code build as described in the [Building process](#building-process) section. When the test code is compiled and linked with the non-test code build, a linker error will occur due to their being multiple `main()` functions defined. For this reason, please either rename your main application file if you need to build and run tests or use a different project.
group-onsemi 0:098463de4c5d 201 - **NOTE:** This does not affect building projects or applications, just building and running tests.