Maxim nexpaq / nexpaq_dev

Preliminary main mbed library for nexpaq development

docs/testing_mbed_OS_5.md@1:d96dbedaebdb, 2016-11-04 (annotated)

Committer:
nexpaq
Date:
Fri Nov 04 20:54:50 2016 +0000
Revision:
1:d96dbedaebdb
Parent:
0:6c56fb4bc5f0 
Removed extra directories for other platforms

Who changed what in which revision?

UserRevisionLine numberNew contents of line
nexpaq 0:6c56fb4bc5f0 1 # Testing in mbed OS 5
nexpaq 0:6c56fb4bc5f0 2
nexpaq 0:6c56fb4bc5f0 3 The way tests are run and compiled in mbed OS 5 is substantially different from previous versions of mbed. Previously, tests were located in one known location and a python file (`tools/tests.py`) kept track of their dependencies, capabilities, and configurations. mbed OS 5 has adopted a more distributed approach to testing. Test code lives alongside the application code, and which is dynamically discovered by the test tools.
nexpaq 0:6c56fb4bc5f0 4
nexpaq 0:6c56fb4bc5f0 5 ## Table of Contents
nexpaq 0:6c56fb4bc5f0 6
nexpaq 0:6c56fb4bc5f0 7 - [Using tests](#using-tests)
nexpaq 0:6c56fb4bc5f0 8 - [Test code structure](#test-code-structure)
nexpaq 0:6c56fb4bc5f0 9 - [Test discovery](#test-discovery)
nexpaq 0:6c56fb4bc5f0 10 - [Test names](#test-names)
nexpaq 0:6c56fb4bc5f0 11 - [Building tests](#building-tests)
nexpaq 0:6c56fb4bc5f0 12 - [Building process](#building-process)
nexpaq 0:6c56fb4bc5f0 13 - [Running tests](#running-tests)
nexpaq 0:6c56fb4bc5f0 14 - [Writing tests](#writing-tests)
nexpaq 0:6c56fb4bc5f0 15 - [Debugging tests](#debugging-tests)
nexpaq 0:6c56fb4bc5f0 16 - [Exporting tests](#exporting-tests)
nexpaq 0:6c56fb4bc5f0 17 - [Running a test while debugging](#running-a-test-while-debugging)
nexpaq 0:6c56fb4bc5f0 18 - [Known issues](#known-issues)
nexpaq 0:6c56fb4bc5f0 19
nexpaq 0:6c56fb4bc5f0 20 ## Using tests
nexpaq 0:6c56fb4bc5f0 21
nexpaq 0:6c56fb4bc5f0 22 ### Test code structure
nexpaq 0:6c56fb4bc5f0 23
nexpaq 0:6c56fb4bc5f0 24 Tests can exist throughout mbed OS and your project's code. They are located under a special directory called `TESTS` (case is important!).
nexpaq 0:6c56fb4bc5f0 25
nexpaq 0:6c56fb4bc5f0 26 Placing code under this directory means it will be ignored when building applications and libraries. This code is only ever used when building tests. This is important since all tests require a `main()` function, and building it with your application would cause multiple `main()` functions to be defined.
nexpaq 0:6c56fb4bc5f0 27
nexpaq 0:6c56fb4bc5f0 28 In addition to being placed under a `TESTS` directory, test sources must exist under two other directories: a test group directory and a test case directory. The following are an examples of this structure:
nexpaq 0:6c56fb4bc5f0 29 ```
nexpaq 0:6c56fb4bc5f0 30 myproject/TESTS/test_group/test_case_1
nexpaq 0:6c56fb4bc5f0 31 ```
nexpaq 0:6c56fb4bc5f0 32
nexpaq 0:6c56fb4bc5f0 33 In this example, `myproject` is the project root and all the source files under the `test_case_1` directory will be included in the test. Any other source files from the OS, libraries, and your project that apply to your target's configuration will also be included in the build of your test.
nexpaq 0:6c56fb4bc5f0 34
nexpaq 0:6c56fb4bc5f0 35 **Note:** Both the test group and test case directory can be named anything you like. However, the `TESTS` directory **must** be named `TESTS` for the tools to detect the test cases correctly.
nexpaq 0:6c56fb4bc5f0 36
nexpaq 0:6c56fb4bc5f0 37 #### Test discovery
nexpaq 0:6c56fb4bc5f0 38
nexpaq 0:6c56fb4bc5f0 39 Since test cases can exist throughout a project, the tools must find them in your project's file structure before building them. This is done by searching for paths that match the pattern detailed above in the [Test code structure](#test-code-structure) section.
nexpaq 0:6c56fb4bc5f0 40
nexpaq 0:6c56fb4bc5f0 41 Test discovery also obeys the same rules that are used when building your project. This means that tests that are placed under a directory with a prefix like `TARGET_`, `TOOLCHAIN_`, or `FEATURE_` will only be discovered, built, and run if your current configuration matches this prefix.
nexpaq 0:6c56fb4bc5f0 42
nexpaq 0:6c56fb4bc5f0 43 For example, if you place a test under the directory `FEATURE_BLE` with the following path:
nexpaq 0:6c56fb4bc5f0 44
nexpaq 0:6c56fb4bc5f0 45 ```
nexpaq 0:6c56fb4bc5f0 46 myproject/mbed-os/features/FEATURE_BLE/TESTS/ble_tests/unit_test
nexpaq 0:6c56fb4bc5f0 47 ```
nexpaq 0:6c56fb4bc5f0 48
nexpaq 0:6c56fb4bc5f0 49 This test case will only be discovered if the target being testing supports the BLE feature. Otherwise, the test will be ignored.
nexpaq 0:6c56fb4bc5f0 50
nexpaq 0:6c56fb4bc5f0 51 Generally, a test should not be placed under a `TARGET_` or `TOOLCHAIN_` directory, since most tests should be designed to work for all target and toolchain configurations.
nexpaq 0:6c56fb4bc5f0 52
nexpaq 0:6c56fb4bc5f0 53 Tests can also be completely ignored by using the `.mbedignore` file described [here](../ignoring_files_from_build.md)
nexpaq 0:6c56fb4bc5f0 54
nexpaq 0:6c56fb4bc5f0 55 #### Test names
nexpaq 0:6c56fb4bc5f0 56
nexpaq 0:6c56fb4bc5f0 57 A test case is named from its position in your project's file structure. For instance, in the above example, a test case with the path `myproject/TESTS/test_group/test_case_1` would be named `tests-test_group-test_case_1`. You will notice that the name is created by joining the directories that make up the path to the test case with a "dash" (`-`) character. This will be a unique name to identify the test case. You will see this name used throughout the build and testing process.
nexpaq 0:6c56fb4bc5f0 58
nexpaq 0:6c56fb4bc5f0 59 ### Building tests
nexpaq 0:6c56fb4bc5f0 60
nexpaq 0:6c56fb4bc5f0 61 Tests can be built easily through mbed CLI. For information on using mbed CLI, please see its [documentation](https://github.com/ARMmbed/mbed-cli).
nexpaq 0:6c56fb4bc5f0 62
nexpaq 0:6c56fb4bc5f0 63 When tests are built for a target and a given toolchain, the available tests are first discovered, then built in series. You can also create a "test specification" file, which can be used by our testing tools to run automated hardware tests. For more information on the test specification file, please see the documentation [here](https://github.com/ARMmbed/greentea#test-specification-json-formatted-input).
nexpaq 0:6c56fb4bc5f0 64
nexpaq 0:6c56fb4bc5f0 65 #### Building process
nexpaq 0:6c56fb4bc5f0 66
nexpaq 0:6c56fb4bc5f0 67 The process for building tests is handled by the `test.py` script (not to be confused with `tests.py`) located under the `tools` directory. This handles the discovery and building of all test cases for a given target and toolchain.
nexpaq 0:6c56fb4bc5f0 68
nexpaq 0:6c56fb4bc5f0 69 The full build process is:
nexpaq 0:6c56fb4bc5f0 70
nexpaq 0:6c56fb4bc5f0 71 1. Build the non-test code (all code not under a `TESTS` folder), but do not link it. The resulting object files are placed in the build directory.
nexpaq 0:6c56fb4bc5f0 72 1. Find all tests that match the given target and toolchain.
nexpaq 0:6c56fb4bc5f0 73 1. For each discovered test, build all of its source files and link it with the non-test code that was built in step 1.
nexpaq 0:6c56fb4bc5f0 74 1. If specified, create a test specification file and place it in the given directory for use by testing tools. This is placed in the build directory by default when using mbed CLI.
nexpaq 0:6c56fb4bc5f0 75
nexpaq 0:6c56fb4bc5f0 76 ### Running tests
nexpaq 0:6c56fb4bc5f0 77
nexpaq 0:6c56fb4bc5f0 78 Automated tests can be run easily through mbed CLI. For information on using mbed CLI, please see its documentation.
nexpaq 0:6c56fb4bc5f0 79
nexpaq 0:6c56fb4bc5f0 80 The testing process requires that the tests are built and that a test specification JSON file exists that describes these available tests. The test specification format is detailed [here](https://github.com/ARMmbed/greentea#test-specification-json-formatted-input).
nexpaq 0:6c56fb4bc5f0 81
nexpaq 0:6c56fb4bc5f0 82 The actual testing process is handled by the Greentea tool. To read more about this tool, please visit its [GitHub repository](https://github.com/ARMmbed/greentea).
nexpaq 0:6c56fb4bc5f0 83
nexpaq 0:6c56fb4bc5f0 84 ### Writing tests
nexpaq 0:6c56fb4bc5f0 85
nexpaq 0:6c56fb4bc5f0 86 You can write tests for your own project, or add more tests to mbed OS. Tests are written using the [Greentea client](https://github.com/ARMmbed/mbed-os/tree/master/features/frameworks/greentea-client), [UNITY](https://github.com/ARMmbed/mbed-os/tree/master/features/frameworks/unity), and [utest](https://github.com/ARMmbed/mbed-os/tree/master/features/frameworks/utest) frameworks, located in `/features/frameworks`. Below is an example test that uses all of these frameworks:
nexpaq 0:6c56fb4bc5f0 87
nexpaq 0:6c56fb4bc5f0 88 ```c++
nexpaq 0:6c56fb4bc5f0 89 #include "mbed.h"
nexpaq 0:6c56fb4bc5f0 90 #include "greentea-client/test_env.h"
nexpaq 0:6c56fb4bc5f0 91 #include "unity.h"
nexpaq 0:6c56fb4bc5f0 92 #include "utest.h"
nexpaq 0:6c56fb4bc5f0 93 #include "rtos.h"
nexpaq 0:6c56fb4bc5f0 94
nexpaq 0:6c56fb4bc5f0 95 using namespace utest::v1;
nexpaq 0:6c56fb4bc5f0 96
nexpaq 0:6c56fb4bc5f0 97 // A test that returns successfully is considered successful
nexpaq 0:6c56fb4bc5f0 98 void test_success() {
nexpaq 0:6c56fb4bc5f0 99 TEST_ASSERT(true);
nexpaq 0:6c56fb4bc5f0 100 }
nexpaq 0:6c56fb4bc5f0 101
nexpaq 0:6c56fb4bc5f0 102 // Tests that assert are considered failing
nexpaq 0:6c56fb4bc5f0 103 void test_failure() {
nexpaq 0:6c56fb4bc5f0 104 TEST_ASSERT(false);
nexpaq 0:6c56fb4bc5f0 105 }
nexpaq 0:6c56fb4bc5f0 106
nexpaq 0:6c56fb4bc5f0 107 utest::v1::status_t test_setup(const size_t number_of_cases) {
nexpaq 0:6c56fb4bc5f0 108 // Setup Greentea using a reasonable timeout in seconds
nexpaq 0:6c56fb4bc5f0 109 GREENTEA_SETUP(40, "default_auto");
nexpaq 0:6c56fb4bc5f0 110 return verbose_test_setup_handler(number_of_cases);
nexpaq 0:6c56fb4bc5f0 111 }
nexpaq 0:6c56fb4bc5f0 112
nexpaq 0:6c56fb4bc5f0 113 // Test cases
nexpaq 0:6c56fb4bc5f0 114 Case cases[] = {
nexpaq 0:6c56fb4bc5f0 115 Case("Testing success test", test_success),
nexpaq 0:6c56fb4bc5f0 116 Case("Testing failure test", test_failure),
nexpaq 0:6c56fb4bc5f0 117 };
nexpaq 0:6c56fb4bc5f0 118
nexpaq 0:6c56fb4bc5f0 119 Specification specification(test_setup, cases);
nexpaq 0:6c56fb4bc5f0 120
nexpaq 0:6c56fb4bc5f0 121 // Entry point into the tests
nexpaq 0:6c56fb4bc5f0 122 int main() {
nexpaq 0:6c56fb4bc5f0 123 return !Harness::run(specification);
nexpaq 0:6c56fb4bc5f0 124 }
nexpaq 0:6c56fb4bc5f0 125 ```
nexpaq 0:6c56fb4bc5f0 126
nexpaq 0:6c56fb4bc5f0 127 This test will first run a case that succeeds, then a case that fails. This is a good template to use when creating tests. For more complex testing examples, please see the documentation for [utest](https://github.com/ARMmbed/mbed-os/tree/master/features/frameworks/utest).
nexpaq 0:6c56fb4bc5f0 128
nexpaq 0:6c56fb4bc5f0 129 ## Debugging tests
nexpaq 0:6c56fb4bc5f0 130
nexpaq 0:6c56fb4bc5f0 131 Debugging tests is a crucial part of the development and porting process. This section will cover exporting the test, then driving the test with the test tools while the target is attached to a debugger.
nexpaq 0:6c56fb4bc5f0 132
nexpaq 0:6c56fb4bc5f0 133 ### Exporting tests
nexpaq 0:6c56fb4bc5f0 134
nexpaq 0:6c56fb4bc5f0 135 Currently, the easiest way to export a test is to copy the test's source code from its test directory to your project's root. This way it will be treated like a normal application by the tools.
nexpaq 0:6c56fb4bc5f0 136
nexpaq 0:6c56fb4bc5f0 137 You can find the path to the test you wish to export by running the following command:
nexpaq 0:6c56fb4bc5f0 138
nexpaq 0:6c56fb4bc5f0 139 ```
nexpaq 0:6c56fb4bc5f0 140 mbed test --compile-list -n <test name>
nexpaq 0:6c56fb4bc5f0 141 ```
nexpaq 0:6c56fb4bc5f0 142
nexpaq 0:6c56fb4bc5f0 143 Once you've copied all of the test's source files to your project root, go ahead and export your project:
nexpaq 0:6c56fb4bc5f0 144
nexpaq 0:6c56fb4bc5f0 145 ```
nexpaq 0:6c56fb4bc5f0 146 mbed export -i <IDE name>
nexpaq 0:6c56fb4bc5f0 147 ```
nexpaq 0:6c56fb4bc5f0 148
nexpaq 0:6c56fb4bc5f0 149 Your exported project should now be in `projectfiles/<IDE>_<target>`. Go ahead and open this project in your IDE.
nexpaq 0:6c56fb4bc5f0 150
nexpaq 0:6c56fb4bc5f0 151 ### Running a test while debugging
nexpaq 0:6c56fb4bc5f0 152
nexpaq 0:6c56fb4bc5f0 153 Assuming your test was exported correctly to your IDE, go ahead and build the project and load it onto your target via your debugger.
nexpaq 0:6c56fb4bc5f0 154
nexpaq 0:6c56fb4bc5f0 155 Bring the target out of reset and run the program. Your target will now be waiting for a synchronizing character string to be sent from the test tools over the serial port. Do not run the `mbed test` commands, because that will attempt to flash the device, which you've already done with your IDE.
nexpaq 0:6c56fb4bc5f0 156
nexpaq 0:6c56fb4bc5f0 157 Instead, the underlying test tools can be used to drive the test. [htrun](https://github.com/ARMmbed/htrun) is the tool that needs to be used in this case. This is installed when you install the requirements for mbed OS. However, if you do not have it installed you can do this by running `pip install mbed-host-tests`.
nexpaq 0:6c56fb4bc5f0 158
nexpaq 0:6c56fb4bc5f0 159 First, find your target's serial port by running the following command:
nexpaq 0:6c56fb4bc5f0 160
nexpaq 0:6c56fb4bc5f0 161 ```
nexpaq 0:6c56fb4bc5f0 162 $ mbed detect
nexpaq 0:6c56fb4bc5f0 163
nexpaq 0:6c56fb4bc5f0 164 [mbed] Detected KL46Z, port COM270, mounted D:
nexpaq 0:6c56fb4bc5f0 165
nexpaq 0:6c56fb4bc5f0 166 ...
nexpaq 0:6c56fb4bc5f0 167 ```
nexpaq 0:6c56fb4bc5f0 168
nexpaq 0:6c56fb4bc5f0 169 From the output, take note of your target's serial port (in this case, it's `COM270`).
nexpaq 0:6c56fb4bc5f0 170
nexpaq 0:6c56fb4bc5f0 171 Run the following command when your device is running the test in your debugger:
nexpaq 0:6c56fb4bc5f0 172
nexpaq 0:6c56fb4bc5f0 173 ```
nexpaq 0:6c56fb4bc5f0 174 mbedhtrun --skip-flashing --skip-reset -p <serial port>:9600
nexpaq 0:6c56fb4bc5f0 175 ```
nexpaq 0:6c56fb4bc5f0 176
nexpaq 0:6c56fb4bc5f0 177 Replace `<serial port>` with the serial port you found by running `mbed detect` above.
nexpaq 0:6c56fb4bc5f0 178
nexpaq 0:6c56fb4bc5f0 179 So for the example above, the command would be:
nexpaq 0:6c56fb4bc5f0 180
nexpaq 0:6c56fb4bc5f0 181 ```
nexpaq 0:6c56fb4bc5f0 182 mbedhtrun --skip-flashing --skip-reset -p COM270:9600
nexpaq 0:6c56fb4bc5f0 183 ```
nexpaq 0:6c56fb4bc5f0 184
nexpaq 0:6c56fb4bc5f0 185 This detects your attached target and drives the test. At this point the test will proceed and allow you to debug it. If you need to rerun the test, simply reset the device with your debugger, run the program, and run the same command.
nexpaq 0:6c56fb4bc5f0 186
nexpaq 0:6c56fb4bc5f0 187 For an explanation of the arguments used in this command, please run `mbedhtrun --help`.
nexpaq 0:6c56fb4bc5f0 188
nexpaq 0:6c56fb4bc5f0 189 ## Known issues
nexpaq 0:6c56fb4bc5f0 190
nexpaq 0:6c56fb4bc5f0 191 - There cannot be a `main()` function outside of a `TESTS` directory when building and running tests. This is because this function will be included in the non-test code build as described in the [Building process](#building-process) section. When the test code is compiled and linked with the non-test code build, a linker error will occur due to their being multiple `main()` functions defined. For this reason, please either rename your main application file if you need to build and run tests or use a different project.
nexpaq 0:6c56fb4bc5f0 192 - **NOTE:** This does not affect building projects or applications, just building and running tests.