erkin yucel / mbed-os

mbed os with nrf51 internal bandgap enabled to read battery level

Dependents:   BLE_file_test BLE_Blink ExternalEncoder

docs/testing_mbed_OS_5.md@0:f269e3021894, 2016-10-23 (annotated)

Committer:
elessair
Date:
Sun Oct 23 15:10:02 2016 +0000
Revision:
0:f269e3021894
Initial commit

Who changed what in which revision?

UserRevisionLine numberNew contents of line
elessair 0:f269e3021894 1 # Testing in mbed OS 5
elessair 0:f269e3021894 2
elessair 0:f269e3021894 3 The way tests are run and compiled in mbed OS 5 is substantially different from previous versions of mbed. Previously, tests were located in one known location and a python file (`tools/tests.py`) kept track of their dependencies, capabilities, and configurations. mbed OS 5 has adopted a more distributed approach to testing. Test code lives alongside the application code, and which is dynamically discovered by the test tools.
elessair 0:f269e3021894 4
elessair 0:f269e3021894 5 ## Table of Contents
elessair 0:f269e3021894 6
elessair 0:f269e3021894 7 - [Using tests](#using-tests)
elessair 0:f269e3021894 8 - [Test code structure](#test-code-structure)
elessair 0:f269e3021894 9 - [Test discovery](#test-discovery)
elessair 0:f269e3021894 10 - [Test names](#test-names)
elessair 0:f269e3021894 11 - [Building tests](#building-tests)
elessair 0:f269e3021894 12 - [Building process](#building-process)
elessair 0:f269e3021894 13 - [App config](#app-config)
elessair 0:f269e3021894 14 - [Running tests](#running-tests)
elessair 0:f269e3021894 15 - [Writing tests](#writing-tests)
elessair 0:f269e3021894 16 - [Debugging tests](#debugging-tests)
elessair 0:f269e3021894 17 - [Exporting tests](#exporting-tests)
elessair 0:f269e3021894 18 - [Running a test while debugging](#running-a-test-while-debugging)
elessair 0:f269e3021894 19 - [Known issues](#known-issues)
elessair 0:f269e3021894 20
elessair 0:f269e3021894 21 ## Using tests
elessair 0:f269e3021894 22
elessair 0:f269e3021894 23 ### Test code structure
elessair 0:f269e3021894 24
elessair 0:f269e3021894 25 Tests can exist throughout mbed OS and your project's code. They are located under a special directory called `TESTS` (case is important!).
elessair 0:f269e3021894 26
elessair 0:f269e3021894 27 Placing code under this directory means it will be ignored when building applications and libraries. This code is only ever used when building tests. This is important since all tests require a `main()` function, and building it with your application would cause multiple `main()` functions to be defined.
elessair 0:f269e3021894 28
elessair 0:f269e3021894 29 In addition to being placed under a `TESTS` directory, test sources must exist under two other directories: a test group directory and a test case directory. The following are an examples of this structure:
elessair 0:f269e3021894 30 ```
elessair 0:f269e3021894 31 myproject/TESTS/test_group/test_case_1
elessair 0:f269e3021894 32 ```
elessair 0:f269e3021894 33
elessair 0:f269e3021894 34 In this example, `myproject` is the project root and all the source files under the `test_case_1` directory will be included in the test. Any other source files from the OS, libraries, and your project that apply to your target's configuration will also be included in the build of your test.
elessair 0:f269e3021894 35
elessair 0:f269e3021894 36 **Note:** Both the test group and test case directory can be named anything you like. However, the `TESTS` directory **must** be named `TESTS` for the tools to detect the test cases correctly.
elessair 0:f269e3021894 37
elessair 0:f269e3021894 38 #### Test discovery
elessair 0:f269e3021894 39
elessair 0:f269e3021894 40 Since test cases can exist throughout a project, the tools must find them in your project's file structure before building them. This is done by searching for paths that match the pattern detailed above in the [Test code structure](#test-code-structure) section.
elessair 0:f269e3021894 41
elessair 0:f269e3021894 42 Test discovery also obeys the same rules that are used when building your project. This means that tests that are placed under a directory with a prefix like `TARGET_`, `TOOLCHAIN_`, or `FEATURE_` will only be discovered, built, and run if your current configuration matches this prefix.
elessair 0:f269e3021894 43
elessair 0:f269e3021894 44 For example, if you place a test under the directory `FEATURE_BLE` with the following path:
elessair 0:f269e3021894 45
elessair 0:f269e3021894 46 ```
elessair 0:f269e3021894 47 myproject/mbed-os/features/FEATURE_BLE/TESTS/ble_tests/unit_test
elessair 0:f269e3021894 48 ```
elessair 0:f269e3021894 49
elessair 0:f269e3021894 50 This test case will only be discovered if the target being testing supports the BLE feature. Otherwise, the test will be ignored.
elessair 0:f269e3021894 51
elessair 0:f269e3021894 52 Generally, a test should not be placed under a `TARGET_` or `TOOLCHAIN_` directory, since most tests should be designed to work for all target and toolchain configurations.
elessair 0:f269e3021894 53
elessair 0:f269e3021894 54 Tests can also be completely ignored by using the `.mbedignore` file described [here](../ignoring_files_from_build.md)
elessair 0:f269e3021894 55
elessair 0:f269e3021894 56 #### Test names
elessair 0:f269e3021894 57
elessair 0:f269e3021894 58 A test case is named from its position in your project's file structure. For instance, in the above example, a test case with the path `myproject/TESTS/test_group/test_case_1` would be named `tests-test_group-test_case_1`. You will notice that the name is created by joining the directories that make up the path to the test case with a "dash" (`-`) character. This will be a unique name to identify the test case. You will see this name used throughout the build and testing process.
elessair 0:f269e3021894 59
elessair 0:f269e3021894 60 ### Building tests
elessair 0:f269e3021894 61
elessair 0:f269e3021894 62 Tests can be built easily through mbed CLI. For information on using mbed CLI, please see its [documentation](https://github.com/ARMmbed/mbed-cli).
elessair 0:f269e3021894 63
elessair 0:f269e3021894 64 When tests are built for a target and a given toolchain, the available tests are first discovered, then built in series. You can also create a "test specification" file, which can be used by our testing tools to run automated hardware tests. For more information on the test specification file, please see the documentation [here](https://github.com/ARMmbed/greentea#test-specification-json-formatted-input).
elessair 0:f269e3021894 65
elessair 0:f269e3021894 66 #### Building process
elessair 0:f269e3021894 67
elessair 0:f269e3021894 68 The process for building tests is handled by the `test.py` script (not to be confused with `tests.py`) located under the `tools` directory. This handles the discovery and building of all test cases for a given target and toolchain.
elessair 0:f269e3021894 69
elessair 0:f269e3021894 70 The full build process is:
elessair 0:f269e3021894 71
elessair 0:f269e3021894 72 1. Build the non-test code (all code not under a `TESTS` folder), but do not link it. The resulting object files are placed in the build directory.
elessair 0:f269e3021894 73 1. Find all tests that match the given target and toolchain.
elessair 0:f269e3021894 74 1. For each discovered test, build all of its source files and link it with the non-test code that was built in step 1.
elessair 0:f269e3021894 75 1. If specified, create a test specification file and place it in the given directory for use by testing tools. This is placed in the build directory by default when using mbed CLI.
elessair 0:f269e3021894 76
elessair 0:f269e3021894 77 #### App config
elessair 0:f269e3021894 78
elessair 0:f269e3021894 79 When building an mbed application, the presence of a `mbed_app.json` file allows you to set or override different config settings from libraries and targets. However, because the tests share a common build, this can cause issues when tests have different configurations that affect the OS.
elessair 0:f269e3021894 80
elessair 0:f269e3021894 81 If you need to use app config, this must be set via the `--app-config` option when calling `mbed test`. **If this option is not specified, the build system will ignore all `mbed_app.json` files and use the default config values.**
elessair 0:f269e3021894 82
elessair 0:f269e3021894 83 ### Running tests
elessair 0:f269e3021894 84
elessair 0:f269e3021894 85 Automated tests can be run easily through mbed CLI. For information on using mbed CLI, please see its documentation.
elessair 0:f269e3021894 86
elessair 0:f269e3021894 87 The testing process requires that the tests are built and that a test specification JSON file exists that describes these available tests. The test specification format is detailed [here](https://github.com/ARMmbed/greentea#test-specification-json-formatted-input).
elessair 0:f269e3021894 88
elessair 0:f269e3021894 89 The actual testing process is handled by the Greentea tool. To read more about this tool, please visit its [GitHub repository](https://github.com/ARMmbed/greentea).
elessair 0:f269e3021894 90
elessair 0:f269e3021894 91 ### Writing tests
elessair 0:f269e3021894 92
elessair 0:f269e3021894 93 You can write tests for your own project, or add more tests to mbed OS. Tests are written using the [Greentea client](https://github.com/ARMmbed/mbed-os/tree/master/features/frameworks/greentea-client), [UNITY](https://github.com/ARMmbed/mbed-os/tree/master/features/frameworks/unity), and [utest](https://github.com/ARMmbed/mbed-os/tree/master/features/frameworks/utest) frameworks, located in `/features/frameworks`. Below is an example test that uses all of these frameworks:
elessair 0:f269e3021894 94
elessair 0:f269e3021894 95 ```c++
elessair 0:f269e3021894 96 #include "mbed.h"
elessair 0:f269e3021894 97 #include "greentea-client/test_env.h"
elessair 0:f269e3021894 98 #include "unity.h"
elessair 0:f269e3021894 99 #include "utest.h"
elessair 0:f269e3021894 100 #include "rtos.h"
elessair 0:f269e3021894 101
elessair 0:f269e3021894 102 using namespace utest::v1;
elessair 0:f269e3021894 103
elessair 0:f269e3021894 104 // A test that returns successfully is considered successful
elessair 0:f269e3021894 105 void test_success() {
elessair 0:f269e3021894 106 TEST_ASSERT(true);
elessair 0:f269e3021894 107 }
elessair 0:f269e3021894 108
elessair 0:f269e3021894 109 // Tests that assert are considered failing
elessair 0:f269e3021894 110 void test_failure() {
elessair 0:f269e3021894 111 TEST_ASSERT(false);
elessair 0:f269e3021894 112 }
elessair 0:f269e3021894 113
elessair 0:f269e3021894 114 utest::v1::status_t test_setup(const size_t number_of_cases) {
elessair 0:f269e3021894 115 // Setup Greentea using a reasonable timeout in seconds
elessair 0:f269e3021894 116 GREENTEA_SETUP(40, "default_auto");
elessair 0:f269e3021894 117 return verbose_test_setup_handler(number_of_cases);
elessair 0:f269e3021894 118 }
elessair 0:f269e3021894 119
elessair 0:f269e3021894 120 // Test cases
elessair 0:f269e3021894 121 Case cases[] = {
elessair 0:f269e3021894 122 Case("Testing success test", test_success),
elessair 0:f269e3021894 123 Case("Testing failure test", test_failure),
elessair 0:f269e3021894 124 };
elessair 0:f269e3021894 125
elessair 0:f269e3021894 126 Specification specification(test_setup, cases);
elessair 0:f269e3021894 127
elessair 0:f269e3021894 128 // Entry point into the tests
elessair 0:f269e3021894 129 int main() {
elessair 0:f269e3021894 130 return !Harness::run(specification);
elessair 0:f269e3021894 131 }
elessair 0:f269e3021894 132 ```
elessair 0:f269e3021894 133
elessair 0:f269e3021894 134 This test will first run a case that succeeds, then a case that fails. This is a good template to use when creating tests. For more complex testing examples, please see the documentation for [utest](https://github.com/ARMmbed/mbed-os/tree/master/features/frameworks/utest).
elessair 0:f269e3021894 135
elessair 0:f269e3021894 136 ## Debugging tests
elessair 0:f269e3021894 137
elessair 0:f269e3021894 138 Debugging tests is a crucial part of the development and porting process. This section will cover exporting the test, then driving the test with the test tools while the target is attached to a debugger.
elessair 0:f269e3021894 139
elessair 0:f269e3021894 140 ### Exporting tests
elessair 0:f269e3021894 141
elessair 0:f269e3021894 142 Currently, the easiest way to export a test is to copy the test's source code from its test directory to your project's root. This way it will be treated like a normal application by the tools.
elessair 0:f269e3021894 143
elessair 0:f269e3021894 144 You can find the path to the test you wish to export by running the following command:
elessair 0:f269e3021894 145
elessair 0:f269e3021894 146 ```
elessair 0:f269e3021894 147 mbed test --compile-list -n <test name>
elessair 0:f269e3021894 148 ```
elessair 0:f269e3021894 149
elessair 0:f269e3021894 150 Once you've copied all of the test's source files to your project root, go ahead and export your project:
elessair 0:f269e3021894 151
elessair 0:f269e3021894 152 ```
elessair 0:f269e3021894 153 mbed export -i <IDE name>
elessair 0:f269e3021894 154 ```
elessair 0:f269e3021894 155
elessair 0:f269e3021894 156 Your exported project should now be in `projectfiles/<IDE>_<target>`. Go ahead and open this project in your IDE.
elessair 0:f269e3021894 157
elessair 0:f269e3021894 158 ### Running a test while debugging
elessair 0:f269e3021894 159
elessair 0:f269e3021894 160 Assuming your test was exported correctly to your IDE, go ahead and build the project and load it onto your target via your debugger.
elessair 0:f269e3021894 161
elessair 0:f269e3021894 162 Bring the target out of reset and run the program. Your target will now be waiting for a synchronizing character string to be sent from the test tools over the serial port. Do not run the `mbed test` commands, because that will attempt to flash the device, which you've already done with your IDE.
elessair 0:f269e3021894 163
elessair 0:f269e3021894 164 Instead, the underlying test tools can be used to drive the test. [htrun](https://github.com/ARMmbed/htrun) is the tool that needs to be used in this case. This is installed when you install the requirements for mbed OS. However, if you do not have it installed you can do this by running `pip install mbed-host-tests`.
elessair 0:f269e3021894 165
elessair 0:f269e3021894 166 First, find your target's serial port by running the following command:
elessair 0:f269e3021894 167
elessair 0:f269e3021894 168 ```
elessair 0:f269e3021894 169 $ mbed detect
elessair 0:f269e3021894 170
elessair 0:f269e3021894 171 [mbed] Detected KL46Z, port COM270, mounted D:
elessair 0:f269e3021894 172
elessair 0:f269e3021894 173 ...
elessair 0:f269e3021894 174 ```
elessair 0:f269e3021894 175
elessair 0:f269e3021894 176 From the output, take note of your target's serial port (in this case, it's `COM270`).
elessair 0:f269e3021894 177
elessair 0:f269e3021894 178 Run the following command when your device is running the test in your debugger:
elessair 0:f269e3021894 179
elessair 0:f269e3021894 180 ```
elessair 0:f269e3021894 181 mbedhtrun --skip-flashing --skip-reset -p <serial port>:9600
elessair 0:f269e3021894 182 ```
elessair 0:f269e3021894 183
elessair 0:f269e3021894 184 Replace `<serial port>` with the serial port you found by running `mbed detect` above.
elessair 0:f269e3021894 185
elessair 0:f269e3021894 186 So for the example above, the command would be:
elessair 0:f269e3021894 187
elessair 0:f269e3021894 188 ```
elessair 0:f269e3021894 189 mbedhtrun --skip-flashing --skip-reset -p COM270:9600
elessair 0:f269e3021894 190 ```
elessair 0:f269e3021894 191
elessair 0:f269e3021894 192 This detects your attached target and drives the test. At this point the test will proceed and allow you to debug it. If you need to rerun the test, simply reset the device with your debugger, run the program, and run the same command.
elessair 0:f269e3021894 193
elessair 0:f269e3021894 194 For an explanation of the arguments used in this command, please run `mbedhtrun --help`.
elessair 0:f269e3021894 195
elessair 0:f269e3021894 196 ## Known issues
elessair 0:f269e3021894 197
elessair 0:f269e3021894 198 - There cannot be a `main()` function outside of a `TESTS` directory when building and running tests. This is because this function will be included in the non-test code build as described in the [Building process](#building-process) section. When the test code is compiled and linked with the non-test code build, a linker error will occur due to their being multiple `main()` functions defined. For this reason, please either rename your main application file if you need to build and run tests or use a different project.
elessair 0:f269e3021894 199 - **NOTE:** This does not affect building projects or applications, just building and running tests.