joey shelton / LED_Demo

WORKS

Dependencies:   MAX44000 PWM_Tone_Library nexpaq_mdk

Fork of LED_Demo by Maxim nexpaq

mbd_os/docs/testing_mbed_OS_5.md@9:6bb35cef007d, 2016-10-22 (annotated)

Committer:
cyberjoey
Date:
Sat Oct 22 01:31:58 2016 +0000
Revision:
9:6bb35cef007d
Parent:
1:55a6170b404f 
WORKING

Who changed what in which revision?

UserRevisionLine numberNew contents of line
nexpaq 1:55a6170b404f 1 # Testing in mbed OS 5
nexpaq 1:55a6170b404f 2
nexpaq 1:55a6170b404f 3 The way tests are run and compiled in mbed OS 5 is substantially different from previous versions of mbed. Previously, tests were located in one known location and a python file (`tools/tests.py`) kept track of their dependencies, capabilities, and configurations. mbed OS 5 has adopted a more distributed approach to testing. Test code lives alongside the application code, and which is dynamically discovered by the test tools.
nexpaq 1:55a6170b404f 4
nexpaq 1:55a6170b404f 5 ## Table of Contents
nexpaq 1:55a6170b404f 6
nexpaq 1:55a6170b404f 7 - [Using tests](#using-tests)
nexpaq 1:55a6170b404f 8 - [Test code structure](#test-code-structure)
nexpaq 1:55a6170b404f 9 - [Test discovery](#test-discovery)
nexpaq 1:55a6170b404f 10 - [Test names](#test-names)
nexpaq 1:55a6170b404f 11 - [Building tests](#building-tests)
nexpaq 1:55a6170b404f 12 - [Building process](#building-process)
nexpaq 1:55a6170b404f 13 - [Running tests](#running-tests)
nexpaq 1:55a6170b404f 14 - [Writing tests](#writing-tests)
nexpaq 1:55a6170b404f 15 - [Debugging tests](#debugging-tests)
nexpaq 1:55a6170b404f 16 - [Exporting tests](#exporting-tests)
nexpaq 1:55a6170b404f 17 - [Running a test while debugging](#running-a-test-while-debugging)
nexpaq 1:55a6170b404f 18 - [Known issues](#known-issues)
nexpaq 1:55a6170b404f 19
nexpaq 1:55a6170b404f 20 ## Using tests
nexpaq 1:55a6170b404f 21
nexpaq 1:55a6170b404f 22 ### Test code structure
nexpaq 1:55a6170b404f 23
nexpaq 1:55a6170b404f 24 Tests can exist throughout mbed OS and your project's code. They are located under a special directory called `TESTS` (case is important!).
nexpaq 1:55a6170b404f 25
nexpaq 1:55a6170b404f 26 Placing code under this directory means it will be ignored when building applications and libraries. This code is only ever used when building tests. This is important since all tests require a `main()` function, and building it with your application would cause multiple `main()` functions to be defined.
nexpaq 1:55a6170b404f 27
nexpaq 1:55a6170b404f 28 In addition to being placed under a `TESTS` directory, test sources must exist under two other directories: a test group directory and a test case directory. The following are an examples of this structure:
nexpaq 1:55a6170b404f 29 ```
nexpaq 1:55a6170b404f 30 myproject/TESTS/test_group/test_case_1
nexpaq 1:55a6170b404f 31 ```
nexpaq 1:55a6170b404f 32
nexpaq 1:55a6170b404f 33 In this example, `myproject` is the project root and all the source files under the `test_case_1` directory will be included in the test. Any other source files from the OS, libraries, and your project that apply to your target's configuration will also be included in the build of your test.
nexpaq 1:55a6170b404f 34
nexpaq 1:55a6170b404f 35 **Note:** Both the test group and test case directory can be named anything you like. However, the `TESTS` directory **must** be named `TESTS` for the tools to detect the test cases correctly.
nexpaq 1:55a6170b404f 36
nexpaq 1:55a6170b404f 37 #### Test discovery
nexpaq 1:55a6170b404f 38
nexpaq 1:55a6170b404f 39 Since test cases can exist throughout a project, the tools must find them in your project's file structure before building them. This is done by searching for paths that match the pattern detailed above in the [Test code structure](#test-code-structure) section.
nexpaq 1:55a6170b404f 40
nexpaq 1:55a6170b404f 41 Test discovery also obeys the same rules that are used when building your project. This means that tests that are placed under a directory with a prefix like `TARGET_`, `TOOLCHAIN_`, or `FEATURE_` will only be discovered, built, and run if your current configuration matches this prefix.
nexpaq 1:55a6170b404f 42
nexpaq 1:55a6170b404f 43 For example, if you place a test under the directory `FEATURE_BLE` with the following path:
nexpaq 1:55a6170b404f 44
nexpaq 1:55a6170b404f 45 ```
nexpaq 1:55a6170b404f 46 myproject/mbed-os/features/FEATURE_BLE/TESTS/ble_tests/unit_test
nexpaq 1:55a6170b404f 47 ```
nexpaq 1:55a6170b404f 48
nexpaq 1:55a6170b404f 49 This test case will only be discovered if the target being testing supports the BLE feature. Otherwise, the test will be ignored.
nexpaq 1:55a6170b404f 50
nexpaq 1:55a6170b404f 51 Generally, a test should not be placed under a `TARGET_` or `TOOLCHAIN_` directory, since most tests should be designed to work for all target and toolchain configurations.
nexpaq 1:55a6170b404f 52
nexpaq 1:55a6170b404f 53 Tests can also be completely ignored by using the `.mbedignore` file described [here](../ignoring_files_from_build.md)
nexpaq 1:55a6170b404f 54
nexpaq 1:55a6170b404f 55 #### Test names
nexpaq 1:55a6170b404f 56
nexpaq 1:55a6170b404f 57 A test case is named from its position in your project's file structure. For instance, in the above example, a test case with the path `myproject/TESTS/test_group/test_case_1` would be named `tests-test_group-test_case_1`. You will notice that the name is created by joining the directories that make up the path to the test case with a "dash" (`-`) character. This will be a unique name to identify the test case. You will see this name used throughout the build and testing process.
nexpaq 1:55a6170b404f 58
nexpaq 1:55a6170b404f 59 ### Building tests
nexpaq 1:55a6170b404f 60
nexpaq 1:55a6170b404f 61 Tests can be built easily through mbed CLI. For information on using mbed CLI, please see its [documentation](https://github.com/ARMmbed/mbed-cli).
nexpaq 1:55a6170b404f 62
nexpaq 1:55a6170b404f 63 When tests are built for a target and a given toolchain, the available tests are first discovered, then built in series. You can also create a "test specification" file, which can be used by our testing tools to run automated hardware tests. For more information on the test specification file, please see the documentation [here](https://github.com/ARMmbed/greentea#test-specification-json-formatted-input).
nexpaq 1:55a6170b404f 64
nexpaq 1:55a6170b404f 65 #### Building process
nexpaq 1:55a6170b404f 66
nexpaq 1:55a6170b404f 67 The process for building tests is handled by the `test.py` script (not to be confused with `tests.py`) located under the `tools` directory. This handles the discovery and building of all test cases for a given target and toolchain.
nexpaq 1:55a6170b404f 68
nexpaq 1:55a6170b404f 69 The full build process is:
nexpaq 1:55a6170b404f 70
nexpaq 1:55a6170b404f 71 1. Build the non-test code (all code not under a `TESTS` folder), but do not link it. The resulting object files are placed in the build directory.
nexpaq 1:55a6170b404f 72 1. Find all tests that match the given target and toolchain.
nexpaq 1:55a6170b404f 73 1. For each discovered test, build all of its source files and link it with the non-test code that was built in step 1.
nexpaq 1:55a6170b404f 74 1. If specified, create a test specification file and place it in the given directory for use by testing tools. This is placed in the build directory by default when using mbed CLI.
nexpaq 1:55a6170b404f 75
nexpaq 1:55a6170b404f 76 ### Running tests
nexpaq 1:55a6170b404f 77
nexpaq 1:55a6170b404f 78 Automated tests can be run easily through mbed CLI. For information on using mbed CLI, please see its documentation.
nexpaq 1:55a6170b404f 79
nexpaq 1:55a6170b404f 80 The testing process requires that the tests are built and that a test specification JSON file exists that describes these available tests. The test specification format is detailed [here](https://github.com/ARMmbed/greentea#test-specification-json-formatted-input).
nexpaq 1:55a6170b404f 81
nexpaq 1:55a6170b404f 82 The actual testing process is handled by the Greentea tool. To read more about this tool, please visit its [GitHub repository](https://github.com/ARMmbed/greentea).
nexpaq 1:55a6170b404f 83
nexpaq 1:55a6170b404f 84 ### Writing tests
nexpaq 1:55a6170b404f 85
nexpaq 1:55a6170b404f 86 You can write tests for your own project, or add more tests to mbed OS. Tests are written using the [Greentea client](https://github.com/ARMmbed/mbed-os/tree/master/features/frameworks/greentea-client), [UNITY](https://github.com/ARMmbed/mbed-os/tree/master/features/frameworks/unity), and [utest](https://github.com/ARMmbed/mbed-os/tree/master/features/frameworks/utest) frameworks, located in `/features/frameworks`. Below is an example test that uses all of these frameworks:
nexpaq 1:55a6170b404f 87
nexpaq 1:55a6170b404f 88 ```c++
nexpaq 1:55a6170b404f 89 #include "mbed.h"
nexpaq 1:55a6170b404f 90 #include "greentea-client/test_env.h"
nexpaq 1:55a6170b404f 91 #include "unity.h"
nexpaq 1:55a6170b404f 92 #include "utest.h"
nexpaq 1:55a6170b404f 93 #include "rtos.h"
nexpaq 1:55a6170b404f 94
nexpaq 1:55a6170b404f 95 using namespace utest::v1;
nexpaq 1:55a6170b404f 96
nexpaq 1:55a6170b404f 97 // A test that returns successfully is considered successful
nexpaq 1:55a6170b404f 98 void test_success() {
nexpaq 1:55a6170b404f 99 TEST_ASSERT(true);
nexpaq 1:55a6170b404f 100 }
nexpaq 1:55a6170b404f 101
nexpaq 1:55a6170b404f 102 // Tests that assert are considered failing
nexpaq 1:55a6170b404f 103 void test_failure() {
nexpaq 1:55a6170b404f 104 TEST_ASSERT(false);
nexpaq 1:55a6170b404f 105 }
nexpaq 1:55a6170b404f 106
nexpaq 1:55a6170b404f 107 utest::v1::status_t test_setup(const size_t number_of_cases) {
nexpaq 1:55a6170b404f 108 // Setup Greentea using a reasonable timeout in seconds
nexpaq 1:55a6170b404f 109 GREENTEA_SETUP(40, "default_auto");
nexpaq 1:55a6170b404f 110 return verbose_test_setup_handler(number_of_cases);
nexpaq 1:55a6170b404f 111 }
nexpaq 1:55a6170b404f 112
nexpaq 1:55a6170b404f 113 // Test cases
nexpaq 1:55a6170b404f 114 Case cases[] = {
nexpaq 1:55a6170b404f 115 Case("Testing success test", test_success),
nexpaq 1:55a6170b404f 116 Case("Testing failure test", test_failure),
nexpaq 1:55a6170b404f 117 };
nexpaq 1:55a6170b404f 118
nexpaq 1:55a6170b404f 119 Specification specification(test_setup, cases);
nexpaq 1:55a6170b404f 120
nexpaq 1:55a6170b404f 121 // Entry point into the tests
nexpaq 1:55a6170b404f 122 int main() {
nexpaq 1:55a6170b404f 123 return !Harness::run(specification);
nexpaq 1:55a6170b404f 124 }
nexpaq 1:55a6170b404f 125 ```
nexpaq 1:55a6170b404f 126
nexpaq 1:55a6170b404f 127 This test will first run a case that succeeds, then a case that fails. This is a good template to use when creating tests. For more complex testing examples, please see the documentation for [utest](https://github.com/ARMmbed/mbed-os/tree/master/features/frameworks/utest).
nexpaq 1:55a6170b404f 128
nexpaq 1:55a6170b404f 129 ## Debugging tests
nexpaq 1:55a6170b404f 130
nexpaq 1:55a6170b404f 131 Debugging tests is a crucial part of the development and porting process. This section will cover exporting the test, then driving the test with the test tools while the target is attached to a debugger.
nexpaq 1:55a6170b404f 132
nexpaq 1:55a6170b404f 133 ### Exporting tests
nexpaq 1:55a6170b404f 134
nexpaq 1:55a6170b404f 135 Currently, the easiest way to export a test is to copy the test's source code from its test directory to your project's root. This way it will be treated like a normal application by the tools.
nexpaq 1:55a6170b404f 136
nexpaq 1:55a6170b404f 137 You can find the path to the test you wish to export by running the following command:
nexpaq 1:55a6170b404f 138
nexpaq 1:55a6170b404f 139 ```
nexpaq 1:55a6170b404f 140 mbed test --compile-list -n <test name>
nexpaq 1:55a6170b404f 141 ```
nexpaq 1:55a6170b404f 142
nexpaq 1:55a6170b404f 143 Once you've copied all of the test's source files to your project root, go ahead and export your project:
nexpaq 1:55a6170b404f 144
nexpaq 1:55a6170b404f 145 ```
nexpaq 1:55a6170b404f 146 mbed export -i <IDE name>
nexpaq 1:55a6170b404f 147 ```
nexpaq 1:55a6170b404f 148
nexpaq 1:55a6170b404f 149 Your exported project should now be in `projectfiles/<IDE>_<target>`. Go ahead and open this project in your IDE.
nexpaq 1:55a6170b404f 150
nexpaq 1:55a6170b404f 151 ### Running a test while debugging
nexpaq 1:55a6170b404f 152
nexpaq 1:55a6170b404f 153 Assuming your test was exported correctly to your IDE, go ahead and build the project and load it onto your target via your debugger.
nexpaq 1:55a6170b404f 154
nexpaq 1:55a6170b404f 155 Bring the target out of reset and run the program. Your target will now be waiting for a synchronizing character string to be sent from the test tools over the serial port. Do not run the `mbed test` commands, because that will attempt to flash the device, which you've already done with your IDE.
nexpaq 1:55a6170b404f 156
nexpaq 1:55a6170b404f 157 Instead, the underlying test tools can be used to drive the test. [htrun](https://github.com/ARMmbed/htrun) is the tool that needs to be used in this case. This is installed when you install the requirements for mbed OS. However, if you do not have it installed you can do this by running `pip install mbed-host-tests`.
nexpaq 1:55a6170b404f 158
nexpaq 1:55a6170b404f 159 First, find your target's serial port by running the following command:
nexpaq 1:55a6170b404f 160
nexpaq 1:55a6170b404f 161 ```
nexpaq 1:55a6170b404f 162 $ mbed detect
nexpaq 1:55a6170b404f 163
nexpaq 1:55a6170b404f 164 [mbed] Detected KL46Z, port COM270, mounted D:
nexpaq 1:55a6170b404f 165
nexpaq 1:55a6170b404f 166 ...
nexpaq 1:55a6170b404f 167 ```
nexpaq 1:55a6170b404f 168
nexpaq 1:55a6170b404f 169 From the output, take note of your target's serial port (in this case, it's `COM270`).
nexpaq 1:55a6170b404f 170
nexpaq 1:55a6170b404f 171 Run the following command when your device is running the test in your debugger:
nexpaq 1:55a6170b404f 172
nexpaq 1:55a6170b404f 173 ```
nexpaq 1:55a6170b404f 174 mbedhtrun --skip-flashing --skip-reset -p <serial port>:9600
nexpaq 1:55a6170b404f 175 ```
nexpaq 1:55a6170b404f 176
nexpaq 1:55a6170b404f 177 Replace `<serial port>` with the serial port you found by running `mbed detect` above.
nexpaq 1:55a6170b404f 178
nexpaq 1:55a6170b404f 179 So for the example above, the command would be:
nexpaq 1:55a6170b404f 180
nexpaq 1:55a6170b404f 181 ```
nexpaq 1:55a6170b404f 182 mbedhtrun --skip-flashing --skip-reset -p COM270:9600
nexpaq 1:55a6170b404f 183 ```
nexpaq 1:55a6170b404f 184
nexpaq 1:55a6170b404f 185 This detects your attached target and drives the test. At this point the test will proceed and allow you to debug it. If you need to rerun the test, simply reset the device with your debugger, run the program, and run the same command.
nexpaq 1:55a6170b404f 186
nexpaq 1:55a6170b404f 187 For an explanation of the arguments used in this command, please run `mbedhtrun --help`.
nexpaq 1:55a6170b404f 188
nexpaq 1:55a6170b404f 189 ## Known issues
nexpaq 1:55a6170b404f 190
nexpaq 1:55a6170b404f 191 - There cannot be a `main()` function outside of a `TESTS` directory when building and running tests. This is because this function will be included in the non-test code build as described in the [Building process](#building-process) section. When the test code is compiled and linked with the non-test code build, a linker error will occur due to their being multiple `main()` functions defined. For this reason, please either rename your main application file if you need to build and run tests or use a different project.
nexpaq 1:55a6170b404f 192 - **NOTE:** This does not affect building projects or applications, just building and running tests.