Magnificent7 / Hackathon

Includes library modifications to allow access to AIN_4 (AIN_0 / 5)

mbd_os/docs/testing_mbed_OS_5.md@0:eafc3fd41f75, 2016-09-20 (annotated)

Committer:
bryantaylor
Date:
Tue Sep 20 21:26:12 2016 +0000
Revision:
0:eafc3fd41f75
hackathon

Who changed what in which revision?

UserRevisionLine numberNew contents of line
bryantaylor 0:eafc3fd41f75 1 # Testing in mbed OS 5
bryantaylor 0:eafc3fd41f75 2
bryantaylor 0:eafc3fd41f75 3 The way tests are run and compiled in mbed OS 5 is substantially different from previous versions of mbed. Previously, tests were located in one known location and a python file (`tools/tests.py`) kept track of their dependencies, capabilities, and configurations. mbed OS 5 has adopted a more distributed approach to testing. Test code lives alongside the application code, and which is dynamically discovered by the test tools.
bryantaylor 0:eafc3fd41f75 4
bryantaylor 0:eafc3fd41f75 5 ## Table of Contents
bryantaylor 0:eafc3fd41f75 6
bryantaylor 0:eafc3fd41f75 7 - [Using tests](#using-tests)
bryantaylor 0:eafc3fd41f75 8 - [Test code structure](#test-code-structure)
bryantaylor 0:eafc3fd41f75 9 - [Test discovery](#test-discovery)
bryantaylor 0:eafc3fd41f75 10 - [Test names](#test-names)
bryantaylor 0:eafc3fd41f75 11 - [Building tests](#building-tests)
bryantaylor 0:eafc3fd41f75 12 - [Building process](#building-process)
bryantaylor 0:eafc3fd41f75 13 - [Running tests](#running-tests)
bryantaylor 0:eafc3fd41f75 14 - [Writing tests](#writing-tests)
bryantaylor 0:eafc3fd41f75 15 - [Debugging tests](#debugging-tests)
bryantaylor 0:eafc3fd41f75 16 - [Exporting tests](#exporting-tests)
bryantaylor 0:eafc3fd41f75 17 - [Running a test while debugging](#running-a-test-while-debugging)
bryantaylor 0:eafc3fd41f75 18 - [Known issues](#known-issues)
bryantaylor 0:eafc3fd41f75 19
bryantaylor 0:eafc3fd41f75 20 ## Using tests
bryantaylor 0:eafc3fd41f75 21
bryantaylor 0:eafc3fd41f75 22 ### Test code structure
bryantaylor 0:eafc3fd41f75 23
bryantaylor 0:eafc3fd41f75 24 Tests can exist throughout mbed OS and your project's code. They are located under a special directory called `TESTS` (case is important!).
bryantaylor 0:eafc3fd41f75 25
bryantaylor 0:eafc3fd41f75 26 Placing code under this directory means it will be ignored when building applications and libraries. This code is only ever used when building tests. This is important since all tests require a `main()` function, and building it with your application would cause multiple `main()` functions to be defined.
bryantaylor 0:eafc3fd41f75 27
bryantaylor 0:eafc3fd41f75 28 In addition to being placed under a `TESTS` directory, test sources must exist under two other directories: a test group directory and a test case directory. The following are an examples of this structure:
bryantaylor 0:eafc3fd41f75 29 ```
bryantaylor 0:eafc3fd41f75 30 myproject/TESTS/test_group/test_case_1
bryantaylor 0:eafc3fd41f75 31 ```
bryantaylor 0:eafc3fd41f75 32
bryantaylor 0:eafc3fd41f75 33 In this example, `myproject` is the project root and all the source files under the `test_case_1` directory will be included in the test. Any other source files from the OS, libraries, and your project that apply to your target's configuration will also be included in the build of your test.
bryantaylor 0:eafc3fd41f75 34
bryantaylor 0:eafc3fd41f75 35 **Note:** Both the test group and test case directory can be named anything you like. However, the `TESTS` directory **must** be named `TESTS` for the tools to detect the test cases correctly.
bryantaylor 0:eafc3fd41f75 36
bryantaylor 0:eafc3fd41f75 37 #### Test discovery
bryantaylor 0:eafc3fd41f75 38
bryantaylor 0:eafc3fd41f75 39 Since test cases can exist throughout a project, the tools must find them in your project's file structure before building them. This is done by searching for paths that match the pattern detailed above in the [Test code structure](#test-code-structure) section.
bryantaylor 0:eafc3fd41f75 40
bryantaylor 0:eafc3fd41f75 41 Test discovery also obeys the same rules that are used when building your project. This means that tests that are placed under a directory with a prefix like `TARGET_`, `TOOLCHAIN_`, or `FEATURE_` will only be discovered, built, and run if your current configuration matches this prefix.
bryantaylor 0:eafc3fd41f75 42
bryantaylor 0:eafc3fd41f75 43 For example, if you place a test under the directory `FEATURE_BLE` with the following path:
bryantaylor 0:eafc3fd41f75 44
bryantaylor 0:eafc3fd41f75 45 ```
bryantaylor 0:eafc3fd41f75 46 myproject/mbed-os/features/FEATURE_BLE/TESTS/ble_tests/unit_test
bryantaylor 0:eafc3fd41f75 47 ```
bryantaylor 0:eafc3fd41f75 48
bryantaylor 0:eafc3fd41f75 49 This test case will only be discovered if the target being testing supports the BLE feature. Otherwise, the test will be ignored.
bryantaylor 0:eafc3fd41f75 50
bryantaylor 0:eafc3fd41f75 51 Generally, a test should not be placed under a `TARGET_` or `TOOLCHAIN_` directory, since most tests should be designed to work for all target and toolchain configurations.
bryantaylor 0:eafc3fd41f75 52
bryantaylor 0:eafc3fd41f75 53 Tests can also be completely ignored by using the `.mbedignore` file described [here](../ignoring_files_from_build.md)
bryantaylor 0:eafc3fd41f75 54
bryantaylor 0:eafc3fd41f75 55 #### Test names
bryantaylor 0:eafc3fd41f75 56
bryantaylor 0:eafc3fd41f75 57 A test case is named from its position in your project's file structure. For instance, in the above example, a test case with the path `myproject/TESTS/test_group/test_case_1` would be named `tests-test_group-test_case_1`. You will notice that the name is created by joining the directories that make up the path to the test case with a "dash" (`-`) character. This will be a unique name to identify the test case. You will see this name used throughout the build and testing process.
bryantaylor 0:eafc3fd41f75 58
bryantaylor 0:eafc3fd41f75 59 ### Building tests
bryantaylor 0:eafc3fd41f75 60
bryantaylor 0:eafc3fd41f75 61 Tests can be built easily through mbed CLI. For information on using mbed CLI, please see its [documentation](https://github.com/ARMmbed/mbed-cli).
bryantaylor 0:eafc3fd41f75 62
bryantaylor 0:eafc3fd41f75 63 When tests are built for a target and a given toolchain, the available tests are first discovered, then built in series. You can also create a "test specification" file, which can be used by our testing tools to run automated hardware tests. For more information on the test specification file, please see the documentation [here](https://github.com/ARMmbed/greentea#test-specification-json-formatted-input).
bryantaylor 0:eafc3fd41f75 64
bryantaylor 0:eafc3fd41f75 65 #### Building process
bryantaylor 0:eafc3fd41f75 66
bryantaylor 0:eafc3fd41f75 67 The process for building tests is handled by the `test.py` script (not to be confused with `tests.py`) located under the `tools` directory. This handles the discovery and building of all test cases for a given target and toolchain.
bryantaylor 0:eafc3fd41f75 68
bryantaylor 0:eafc3fd41f75 69 The full build process is:
bryantaylor 0:eafc3fd41f75 70
bryantaylor 0:eafc3fd41f75 71 1. Build the non-test code (all code not under a `TESTS` folder), but do not link it. The resulting object files are placed in the build directory.
bryantaylor 0:eafc3fd41f75 72 1. Find all tests that match the given target and toolchain.
bryantaylor 0:eafc3fd41f75 73 1. For each discovered test, build all of its source files and link it with the non-test code that was built in step 1.
bryantaylor 0:eafc3fd41f75 74 1. If specified, create a test specification file and place it in the given directory for use by testing tools. This is placed in the build directory by default when using mbed CLI.
bryantaylor 0:eafc3fd41f75 75
bryantaylor 0:eafc3fd41f75 76 ### Running tests
bryantaylor 0:eafc3fd41f75 77
bryantaylor 0:eafc3fd41f75 78 Automated tests can be run easily through mbed CLI. For information on using mbed CLI, please see its documentation.
bryantaylor 0:eafc3fd41f75 79
bryantaylor 0:eafc3fd41f75 80 The testing process requires that the tests are built and that a test specification JSON file exists that describes these available tests. The test specification format is detailed [here](https://github.com/ARMmbed/greentea#test-specification-json-formatted-input).
bryantaylor 0:eafc3fd41f75 81
bryantaylor 0:eafc3fd41f75 82 The actual testing process is handled by the Greentea tool. To read more about this tool, please visit its [GitHub repository](https://github.com/ARMmbed/greentea).
bryantaylor 0:eafc3fd41f75 83
bryantaylor 0:eafc3fd41f75 84 ### Writing tests
bryantaylor 0:eafc3fd41f75 85
bryantaylor 0:eafc3fd41f75 86 You can write tests for your own project, or add more tests to mbed OS. Tests are written using the [Greentea client](https://github.com/ARMmbed/mbed-os/tree/master/features/frameworks/greentea-client), [UNITY](https://github.com/ARMmbed/mbed-os/tree/master/features/frameworks/unity), and [utest](https://github.com/ARMmbed/mbed-os/tree/master/features/frameworks/utest) frameworks, located in `/features/frameworks`. Below is an example test that uses all of these frameworks:
bryantaylor 0:eafc3fd41f75 87
bryantaylor 0:eafc3fd41f75 88 ```c++
bryantaylor 0:eafc3fd41f75 89 #include "mbed.h"
bryantaylor 0:eafc3fd41f75 90 #include "greentea-client/test_env.h"
bryantaylor 0:eafc3fd41f75 91 #include "unity.h"
bryantaylor 0:eafc3fd41f75 92 #include "utest.h"
bryantaylor 0:eafc3fd41f75 93 #include "rtos.h"
bryantaylor 0:eafc3fd41f75 94
bryantaylor 0:eafc3fd41f75 95 using namespace utest::v1;
bryantaylor 0:eafc3fd41f75 96
bryantaylor 0:eafc3fd41f75 97 // A test that returns successfully is considered successful
bryantaylor 0:eafc3fd41f75 98 void test_success() {
bryantaylor 0:eafc3fd41f75 99 TEST_ASSERT(true);
bryantaylor 0:eafc3fd41f75 100 }
bryantaylor 0:eafc3fd41f75 101
bryantaylor 0:eafc3fd41f75 102 // Tests that assert are considered failing
bryantaylor 0:eafc3fd41f75 103 void test_failure() {
bryantaylor 0:eafc3fd41f75 104 TEST_ASSERT(false);
bryantaylor 0:eafc3fd41f75 105 }
bryantaylor 0:eafc3fd41f75 106
bryantaylor 0:eafc3fd41f75 107 utest::v1::status_t test_setup(const size_t number_of_cases) {
bryantaylor 0:eafc3fd41f75 108 // Setup Greentea using a reasonable timeout in seconds
bryantaylor 0:eafc3fd41f75 109 GREENTEA_SETUP(40, "default_auto");
bryantaylor 0:eafc3fd41f75 110 return verbose_test_setup_handler(number_of_cases);
bryantaylor 0:eafc3fd41f75 111 }
bryantaylor 0:eafc3fd41f75 112
bryantaylor 0:eafc3fd41f75 113 // Test cases
bryantaylor 0:eafc3fd41f75 114 Case cases[] = {
bryantaylor 0:eafc3fd41f75 115 Case("Testing success test", test_success),
bryantaylor 0:eafc3fd41f75 116 Case("Testing failure test", test_failure),
bryantaylor 0:eafc3fd41f75 117 };
bryantaylor 0:eafc3fd41f75 118
bryantaylor 0:eafc3fd41f75 119 Specification specification(test_setup, cases);
bryantaylor 0:eafc3fd41f75 120
bryantaylor 0:eafc3fd41f75 121 // Entry point into the tests
bryantaylor 0:eafc3fd41f75 122 int main() {
bryantaylor 0:eafc3fd41f75 123 return !Harness::run(specification);
bryantaylor 0:eafc3fd41f75 124 }
bryantaylor 0:eafc3fd41f75 125 ```
bryantaylor 0:eafc3fd41f75 126
bryantaylor 0:eafc3fd41f75 127 This test will first run a case that succeeds, then a case that fails. This is a good template to use when creating tests. For more complex testing examples, please see the documentation for [utest](https://github.com/ARMmbed/mbed-os/tree/master/features/frameworks/utest).
bryantaylor 0:eafc3fd41f75 128
bryantaylor 0:eafc3fd41f75 129 ## Debugging tests
bryantaylor 0:eafc3fd41f75 130
bryantaylor 0:eafc3fd41f75 131 Debugging tests is a crucial part of the development and porting process. This section will cover exporting the test, then driving the test with the test tools while the target is attached to a debugger.
bryantaylor 0:eafc3fd41f75 132
bryantaylor 0:eafc3fd41f75 133 ### Exporting tests
bryantaylor 0:eafc3fd41f75 134
bryantaylor 0:eafc3fd41f75 135 Currently, the easiest way to export a test is to copy the test's source code from its test directory to your project's root. This way it will be treated like a normal application by the tools.
bryantaylor 0:eafc3fd41f75 136
bryantaylor 0:eafc3fd41f75 137 You can find the path to the test you wish to export by running the following command:
bryantaylor 0:eafc3fd41f75 138
bryantaylor 0:eafc3fd41f75 139 ```
bryantaylor 0:eafc3fd41f75 140 mbed test --compile-list -n <test name>
bryantaylor 0:eafc3fd41f75 141 ```
bryantaylor 0:eafc3fd41f75 142
bryantaylor 0:eafc3fd41f75 143 Once you've copied all of the test's source files to your project root, go ahead and export your project:
bryantaylor 0:eafc3fd41f75 144
bryantaylor 0:eafc3fd41f75 145 ```
bryantaylor 0:eafc3fd41f75 146 mbed export -i <IDE name>
bryantaylor 0:eafc3fd41f75 147 ```
bryantaylor 0:eafc3fd41f75 148
bryantaylor 0:eafc3fd41f75 149 Your exported project should now be in `projectfiles/<IDE>_<target>`. Go ahead and open this project in your IDE.
bryantaylor 0:eafc3fd41f75 150
bryantaylor 0:eafc3fd41f75 151 ### Running a test while debugging
bryantaylor 0:eafc3fd41f75 152
bryantaylor 0:eafc3fd41f75 153 Assuming your test was exported correctly to your IDE, go ahead and build the project and load it onto your target via your debugger.
bryantaylor 0:eafc3fd41f75 154
bryantaylor 0:eafc3fd41f75 155 Bring the target out of reset and run the program. Your target will now be waiting for a synchronizing character string to be sent from the test tools over the serial port. Do not run the `mbed test` commands, because that will attempt to flash the device, which you've already done with your IDE.
bryantaylor 0:eafc3fd41f75 156
bryantaylor 0:eafc3fd41f75 157 Instead, the underlying test tools can be used to drive the test. [htrun](https://github.com/ARMmbed/htrun) is the tool that needs to be used in this case. This is installed when you install the requirements for mbed OS. However, if you do not have it installed you can do this by running `pip install mbed-host-tests`.
bryantaylor 0:eafc3fd41f75 158
bryantaylor 0:eafc3fd41f75 159 First, find your target's serial port by running the following command:
bryantaylor 0:eafc3fd41f75 160
bryantaylor 0:eafc3fd41f75 161 ```
bryantaylor 0:eafc3fd41f75 162 $ mbed detect
bryantaylor 0:eafc3fd41f75 163
bryantaylor 0:eafc3fd41f75 164 [mbed] Detected KL46Z, port COM270, mounted D:
bryantaylor 0:eafc3fd41f75 165
bryantaylor 0:eafc3fd41f75 166 ...
bryantaylor 0:eafc3fd41f75 167 ```
bryantaylor 0:eafc3fd41f75 168
bryantaylor 0:eafc3fd41f75 169 From the output, take note of your target's serial port (in this case, it's `COM270`).
bryantaylor 0:eafc3fd41f75 170
bryantaylor 0:eafc3fd41f75 171 Run the following command when your device is running the test in your debugger:
bryantaylor 0:eafc3fd41f75 172
bryantaylor 0:eafc3fd41f75 173 ```
bryantaylor 0:eafc3fd41f75 174 mbedhtrun --skip-flashing --skip-reset -p <serial port>:9600
bryantaylor 0:eafc3fd41f75 175 ```
bryantaylor 0:eafc3fd41f75 176
bryantaylor 0:eafc3fd41f75 177 Replace `<serial port>` with the serial port you found by running `mbed detect` above.
bryantaylor 0:eafc3fd41f75 178
bryantaylor 0:eafc3fd41f75 179 So for the example above, the command would be:
bryantaylor 0:eafc3fd41f75 180
bryantaylor 0:eafc3fd41f75 181 ```
bryantaylor 0:eafc3fd41f75 182 mbedhtrun --skip-flashing --skip-reset -p COM270:9600
bryantaylor 0:eafc3fd41f75 183 ```
bryantaylor 0:eafc3fd41f75 184
bryantaylor 0:eafc3fd41f75 185 This detects your attached target and drives the test. At this point the test will proceed and allow you to debug it. If you need to rerun the test, simply reset the device with your debugger, run the program, and run the same command.
bryantaylor 0:eafc3fd41f75 186
bryantaylor 0:eafc3fd41f75 187 For an explanation of the arguments used in this command, please run `mbedhtrun --help`.
bryantaylor 0:eafc3fd41f75 188
bryantaylor 0:eafc3fd41f75 189 ## Known issues
bryantaylor 0:eafc3fd41f75 190
bryantaylor 0:eafc3fd41f75 191 - There cannot be a `main()` function outside of a `TESTS` directory when building and running tests. This is because this function will be included in the non-test code build as described in the [Building process](#building-process) section. When the test code is compiled and linked with the non-test code build, a linker error will occur due to their being multiple `main()` functions defined. For this reason, please either rename your main application file if you need to build and run tests or use a different project.
bryantaylor 0:eafc3fd41f75 192 - **NOTE:** This does not affect building projects or applications, just building and running tests.