Greg Steiert / pegasus_dev

Development mbed library for MAX32630FTHR

Dependents:   blinky_max32630fthr

docs/testing_mbed_OS_5.md@0:5c4d7b2438d3, 2016-11-11 (annotated)

Committer:
switches
Date:
Fri Nov 11 20:59:50 2016 +0000
Revision:
0:5c4d7b2438d3
Initial commit

Who changed what in which revision?

UserRevisionLine numberNew contents of line
switches 0:5c4d7b2438d3 1 # Testing in mbed OS 5
switches 0:5c4d7b2438d3 2
switches 0:5c4d7b2438d3 3 The way tests are run and compiled in mbed OS 5 is substantially different from previous versions of mbed. Previously, tests were located in one known location and a python file (`tools/tests.py`) kept track of their dependencies, capabilities, and configurations. mbed OS 5 has adopted a more distributed approach to testing. Test code lives alongside the application code, and which is dynamically discovered by the test tools.
switches 0:5c4d7b2438d3 4
switches 0:5c4d7b2438d3 5 ## Table of Contents
switches 0:5c4d7b2438d3 6
switches 0:5c4d7b2438d3 7 - [Using tests](#using-tests)
switches 0:5c4d7b2438d3 8 - [Test code structure](#test-code-structure)
switches 0:5c4d7b2438d3 9 - [Test discovery](#test-discovery)
switches 0:5c4d7b2438d3 10 - [Test names](#test-names)
switches 0:5c4d7b2438d3 11 - [Building tests](#building-tests)
switches 0:5c4d7b2438d3 12 - [Building process](#building-process)
switches 0:5c4d7b2438d3 13 - [App config](#app-config)
switches 0:5c4d7b2438d3 14 - [Running tests](#running-tests)
switches 0:5c4d7b2438d3 15 - [Writing tests](#writing-tests)
switches 0:5c4d7b2438d3 16 - [Debugging tests](#debugging-tests)
switches 0:5c4d7b2438d3 17 - [Exporting tests](#exporting-tests)
switches 0:5c4d7b2438d3 18 - [Running a test while debugging](#running-a-test-while-debugging)
switches 0:5c4d7b2438d3 19 - [Known issues](#known-issues)
switches 0:5c4d7b2438d3 20
switches 0:5c4d7b2438d3 21 ## Using tests
switches 0:5c4d7b2438d3 22
switches 0:5c4d7b2438d3 23 ### Test code structure
switches 0:5c4d7b2438d3 24
switches 0:5c4d7b2438d3 25 Tests can exist throughout mbed OS and your project's code. They are located under a special directory called `TESTS` (case is important!).
switches 0:5c4d7b2438d3 26
switches 0:5c4d7b2438d3 27 Placing code under this directory means it will be ignored when building applications and libraries. This code is only ever used when building tests. This is important since all tests require a `main()` function, and building it with your application would cause multiple `main()` functions to be defined.
switches 0:5c4d7b2438d3 28
switches 0:5c4d7b2438d3 29 In addition to being placed under a `TESTS` directory, test sources must exist under two other directories: a test group directory and a test case directory. The following are an examples of this structure:
switches 0:5c4d7b2438d3 30 ```
switches 0:5c4d7b2438d3 31 myproject/TESTS/test_group/test_case_1
switches 0:5c4d7b2438d3 32 ```
switches 0:5c4d7b2438d3 33
switches 0:5c4d7b2438d3 34 In this example, `myproject` is the project root and all the source files under the `test_case_1` directory will be included in the test. Any other source files from the OS, libraries, and your project that apply to your target's configuration will also be included in the build of your test.
switches 0:5c4d7b2438d3 35
switches 0:5c4d7b2438d3 36 **Note:** Both the test group and test case directory can be named anything you like. However, the `TESTS` directory **must** be named `TESTS` for the tools to detect the test cases correctly.
switches 0:5c4d7b2438d3 37
switches 0:5c4d7b2438d3 38 #### Test discovery
switches 0:5c4d7b2438d3 39
switches 0:5c4d7b2438d3 40 Since test cases can exist throughout a project, the tools must find them in your project's file structure before building them. This is done by searching for paths that match the pattern detailed above in the [Test code structure](#test-code-structure) section.
switches 0:5c4d7b2438d3 41
switches 0:5c4d7b2438d3 42 Test discovery also obeys the same rules that are used when building your project. This means that tests that are placed under a directory with a prefix like `TARGET_`, `TOOLCHAIN_`, or `FEATURE_` will only be discovered, built, and run if your current configuration matches this prefix.
switches 0:5c4d7b2438d3 43
switches 0:5c4d7b2438d3 44 For example, if you place a test under the directory `FEATURE_BLE` with the following path:
switches 0:5c4d7b2438d3 45
switches 0:5c4d7b2438d3 46 ```
switches 0:5c4d7b2438d3 47 myproject/mbed-os/features/FEATURE_BLE/TESTS/ble_tests/unit_test
switches 0:5c4d7b2438d3 48 ```
switches 0:5c4d7b2438d3 49
switches 0:5c4d7b2438d3 50 This test case will only be discovered if the target being testing supports the BLE feature. Otherwise, the test will be ignored.
switches 0:5c4d7b2438d3 51
switches 0:5c4d7b2438d3 52 Generally, a test should not be placed under a `TARGET_` or `TOOLCHAIN_` directory, since most tests should be designed to work for all target and toolchain configurations.
switches 0:5c4d7b2438d3 53
switches 0:5c4d7b2438d3 54 Tests can also be completely ignored by using the `.mbedignore` file described [here](../ignoring_files_from_build.md)
switches 0:5c4d7b2438d3 55
switches 0:5c4d7b2438d3 56 #### Test names
switches 0:5c4d7b2438d3 57
switches 0:5c4d7b2438d3 58 A test case is named from its position in your project's file structure. For instance, in the above example, a test case with the path `myproject/TESTS/test_group/test_case_1` would be named `tests-test_group-test_case_1`. You will notice that the name is created by joining the directories that make up the path to the test case with a "dash" (`-`) character. This will be a unique name to identify the test case. You will see this name used throughout the build and testing process.
switches 0:5c4d7b2438d3 59
switches 0:5c4d7b2438d3 60 ### Building tests
switches 0:5c4d7b2438d3 61
switches 0:5c4d7b2438d3 62 Tests can be built easily through mbed CLI. For information on using mbed CLI, please see its [documentation](https://github.com/ARMmbed/mbed-cli).
switches 0:5c4d7b2438d3 63
switches 0:5c4d7b2438d3 64 When tests are built for a target and a given toolchain, the available tests are first discovered, then built in series. You can also create a "test specification" file, which can be used by our testing tools to run automated hardware tests. For more information on the test specification file, please see the documentation [here](https://github.com/ARMmbed/greentea#test-specification-json-formatted-input).
switches 0:5c4d7b2438d3 65
switches 0:5c4d7b2438d3 66 #### Building process
switches 0:5c4d7b2438d3 67
switches 0:5c4d7b2438d3 68 The process for building tests is handled by the `test.py` script (not to be confused with `tests.py`) located under the `tools` directory. This handles the discovery and building of all test cases for a given target and toolchain.
switches 0:5c4d7b2438d3 69
switches 0:5c4d7b2438d3 70 The full build process is:
switches 0:5c4d7b2438d3 71
switches 0:5c4d7b2438d3 72 1. Build the non-test code (all code not under a `TESTS` folder), but do not link it. The resulting object files are placed in the build directory.
switches 0:5c4d7b2438d3 73 1. Find all tests that match the given target and toolchain.
switches 0:5c4d7b2438d3 74 1. For each discovered test, build all of its source files and link it with the non-test code that was built in step 1.
switches 0:5c4d7b2438d3 75 1. If specified, create a test specification file and place it in the given directory for use by testing tools. This is placed in the build directory by default when using mbed CLI.
switches 0:5c4d7b2438d3 76
switches 0:5c4d7b2438d3 77 #### App config
switches 0:5c4d7b2438d3 78
switches 0:5c4d7b2438d3 79 When building an mbed application, the presence of a `mbed_app.json` file allows you to set or override different config settings from libraries and targets. However, because the tests share a common build, this can cause issues when tests have different configurations that affect the OS.
switches 0:5c4d7b2438d3 80
switches 0:5c4d7b2438d3 81 The build system will look for an `mbed_app.json` file in your shared project files (any directory not inside of a `TESTS` folder). If this is found, this configuration file will be used for both the non-test code as well as each test case inside your project's source tree. If there is more than one `mbed_app.json` files in the source tree, the config system will error.
switches 0:5c4d7b2438d3 82
switches 0:5c4d7b2438d3 83 If you need to test with multiple configurations, then you can use the `--app-config` option. This will override the search for an `mbed_app.json` file and use the config file you specify for the build.
switches 0:5c4d7b2438d3 84
switches 0:5c4d7b2438d3 85 ### Running tests
switches 0:5c4d7b2438d3 86
switches 0:5c4d7b2438d3 87 Automated tests can be run easily through mbed CLI. For information on using mbed CLI, please see its documentation.
switches 0:5c4d7b2438d3 88
switches 0:5c4d7b2438d3 89 The testing process requires that the tests are built and that a test specification JSON file exists that describes these available tests. The test specification format is detailed [here](https://github.com/ARMmbed/greentea#test-specification-json-formatted-input).
switches 0:5c4d7b2438d3 90
switches 0:5c4d7b2438d3 91 The actual testing process is handled by the Greentea tool. To read more about this tool, please visit its [GitHub repository](https://github.com/ARMmbed/greentea).
switches 0:5c4d7b2438d3 92
switches 0:5c4d7b2438d3 93 ### Writing tests
switches 0:5c4d7b2438d3 94
switches 0:5c4d7b2438d3 95 You can write tests for your own project, or add more tests to mbed OS. Tests are written using the [Greentea client](https://github.com/ARMmbed/mbed-os/tree/master/features/frameworks/greentea-client), [UNITY](https://github.com/ARMmbed/mbed-os/tree/master/features/frameworks/unity), and [utest](https://github.com/ARMmbed/mbed-os/tree/master/features/frameworks/utest) frameworks, located in `/features/frameworks`. Below is an example test that uses all of these frameworks:
switches 0:5c4d7b2438d3 96
switches 0:5c4d7b2438d3 97 ```c++
switches 0:5c4d7b2438d3 98 #include "mbed.h"
switches 0:5c4d7b2438d3 99 #include "greentea-client/test_env.h"
switches 0:5c4d7b2438d3 100 #include "unity.h"
switches 0:5c4d7b2438d3 101 #include "utest.h"
switches 0:5c4d7b2438d3 102 #include "rtos.h"
switches 0:5c4d7b2438d3 103
switches 0:5c4d7b2438d3 104 using namespace utest::v1;
switches 0:5c4d7b2438d3 105
switches 0:5c4d7b2438d3 106 // A test that returns successfully is considered successful
switches 0:5c4d7b2438d3 107 void test_success() {
switches 0:5c4d7b2438d3 108 TEST_ASSERT(true);
switches 0:5c4d7b2438d3 109 }
switches 0:5c4d7b2438d3 110
switches 0:5c4d7b2438d3 111 // Tests that assert are considered failing
switches 0:5c4d7b2438d3 112 void test_failure() {
switches 0:5c4d7b2438d3 113 TEST_ASSERT(false);
switches 0:5c4d7b2438d3 114 }
switches 0:5c4d7b2438d3 115
switches 0:5c4d7b2438d3 116 utest::v1::status_t test_setup(const size_t number_of_cases) {
switches 0:5c4d7b2438d3 117 // Setup Greentea using a reasonable timeout in seconds
switches 0:5c4d7b2438d3 118 GREENTEA_SETUP(40, "default_auto");
switches 0:5c4d7b2438d3 119 return verbose_test_setup_handler(number_of_cases);
switches 0:5c4d7b2438d3 120 }
switches 0:5c4d7b2438d3 121
switches 0:5c4d7b2438d3 122 // Test cases
switches 0:5c4d7b2438d3 123 Case cases[] = {
switches 0:5c4d7b2438d3 124 Case("Testing success test", test_success),
switches 0:5c4d7b2438d3 125 Case("Testing failure test", test_failure),
switches 0:5c4d7b2438d3 126 };
switches 0:5c4d7b2438d3 127
switches 0:5c4d7b2438d3 128 Specification specification(test_setup, cases);
switches 0:5c4d7b2438d3 129
switches 0:5c4d7b2438d3 130 // Entry point into the tests
switches 0:5c4d7b2438d3 131 int main() {
switches 0:5c4d7b2438d3 132 return !Harness::run(specification);
switches 0:5c4d7b2438d3 133 }
switches 0:5c4d7b2438d3 134 ```
switches 0:5c4d7b2438d3 135
switches 0:5c4d7b2438d3 136 This test will first run a case that succeeds, then a case that fails. This is a good template to use when creating tests. For more complex testing examples, please see the documentation for [utest](https://github.com/ARMmbed/mbed-os/tree/master/features/frameworks/utest).
switches 0:5c4d7b2438d3 137
switches 0:5c4d7b2438d3 138 ## Debugging tests
switches 0:5c4d7b2438d3 139
switches 0:5c4d7b2438d3 140 Debugging tests is a crucial part of the development and porting process. This section will cover exporting the test, then driving the test with the test tools while the target is attached to a debugger.
switches 0:5c4d7b2438d3 141
switches 0:5c4d7b2438d3 142 ### Exporting tests
switches 0:5c4d7b2438d3 143
switches 0:5c4d7b2438d3 144 Currently, the easiest way to export a test is to copy the test's source code from its test directory to your project's root. This way it will be treated like a normal application by the tools.
switches 0:5c4d7b2438d3 145
switches 0:5c4d7b2438d3 146 You can find the path to the test you wish to export by running the following command:
switches 0:5c4d7b2438d3 147
switches 0:5c4d7b2438d3 148 ```
switches 0:5c4d7b2438d3 149 mbed test --compile-list -n <test name>
switches 0:5c4d7b2438d3 150 ```
switches 0:5c4d7b2438d3 151
switches 0:5c4d7b2438d3 152 Once you've copied all of the test's source files to your project root, go ahead and export your project:
switches 0:5c4d7b2438d3 153
switches 0:5c4d7b2438d3 154 ```
switches 0:5c4d7b2438d3 155 mbed export -i <IDE name>
switches 0:5c4d7b2438d3 156 ```
switches 0:5c4d7b2438d3 157
switches 0:5c4d7b2438d3 158 Your exported project should now be in `projectfiles/<IDE>_<target>`. Go ahead and open this project in your IDE.
switches 0:5c4d7b2438d3 159
switches 0:5c4d7b2438d3 160 ### Running a test while debugging
switches 0:5c4d7b2438d3 161
switches 0:5c4d7b2438d3 162 Assuming your test was exported correctly to your IDE, go ahead and build the project and load it onto your target via your debugger.
switches 0:5c4d7b2438d3 163
switches 0:5c4d7b2438d3 164 Bring the target out of reset and run the program. Your target will now be waiting for a synchronizing character string to be sent from the test tools over the serial port. Do not run the `mbed test` commands, because that will attempt to flash the device, which you've already done with your IDE.
switches 0:5c4d7b2438d3 165
switches 0:5c4d7b2438d3 166 Instead, the underlying test tools can be used to drive the test. [htrun](https://github.com/ARMmbed/htrun) is the tool that needs to be used in this case. This is installed when you install the requirements for mbed OS. However, if you do not have it installed you can do this by running `pip install mbed-host-tests`.
switches 0:5c4d7b2438d3 167
switches 0:5c4d7b2438d3 168 First, find your target's serial port by running the following command:
switches 0:5c4d7b2438d3 169
switches 0:5c4d7b2438d3 170 ```
switches 0:5c4d7b2438d3 171 $ mbed detect
switches 0:5c4d7b2438d3 172
switches 0:5c4d7b2438d3 173 [mbed] Detected KL46Z, port COM270, mounted D:
switches 0:5c4d7b2438d3 174
switches 0:5c4d7b2438d3 175 ...
switches 0:5c4d7b2438d3 176 ```
switches 0:5c4d7b2438d3 177
switches 0:5c4d7b2438d3 178 From the output, take note of your target's serial port (in this case, it's `COM270`).
switches 0:5c4d7b2438d3 179
switches 0:5c4d7b2438d3 180 Run the following command when your device is running the test in your debugger:
switches 0:5c4d7b2438d3 181
switches 0:5c4d7b2438d3 182 ```
switches 0:5c4d7b2438d3 183 mbedhtrun --skip-flashing --skip-reset -p <serial port>:9600
switches 0:5c4d7b2438d3 184 ```
switches 0:5c4d7b2438d3 185
switches 0:5c4d7b2438d3 186 Replace `<serial port>` with the serial port you found by running `mbed detect` above.
switches 0:5c4d7b2438d3 187
switches 0:5c4d7b2438d3 188 So for the example above, the command would be:
switches 0:5c4d7b2438d3 189
switches 0:5c4d7b2438d3 190 ```
switches 0:5c4d7b2438d3 191 mbedhtrun --skip-flashing --skip-reset -p COM270:9600
switches 0:5c4d7b2438d3 192 ```
switches 0:5c4d7b2438d3 193
switches 0:5c4d7b2438d3 194 This detects your attached target and drives the test. At this point the test will proceed and allow you to debug it. If you need to rerun the test, simply reset the device with your debugger, run the program, and run the same command.
switches 0:5c4d7b2438d3 195
switches 0:5c4d7b2438d3 196 For an explanation of the arguments used in this command, please run `mbedhtrun --help`.
switches 0:5c4d7b2438d3 197
switches 0:5c4d7b2438d3 198 ## Known issues
switches 0:5c4d7b2438d3 199
switches 0:5c4d7b2438d3 200 - There cannot be a `main()` function outside of a `TESTS` directory when building and running tests. This is because this function will be included in the non-test code build as described in the [Building process](#building-process) section. When the test code is compiled and linked with the non-test code build, a linker error will occur due to their being multiple `main()` functions defined. For this reason, please either rename your main application file if you need to build and run tests or use a different project.
switches 0:5c4d7b2438d3 201 - **NOTE:** This does not affect building projects or applications, just building and running tests.