You are viewing an older revision! See the latest version
Mbed SDK automated test suite
Mbed SDK automated test suite¶
Introduction¶
Test suit allows users to run locally on their machines Mbed SDK’s tests included in Mbed SDK repository. It also allows users to create their own tests and for example add new tests to test set as they progress with their project. If test is generic enough it could be included into official Mbed SDK test pool just do it via normal pull request!
Each test is supervised by python script called “host test” which will at least Test suite is using build script API to compile and build test source together with required by test libraries like CMSIS, Mbed, Ethernet, USB etc.
Prerequisites¶
- Installed Python 2.7.6 programming language: https://www.python.org/download/releases/2.7.
- Installed pyserial module for Python 2.7: https://pypi.python.org/pypi/pyserial
- Installed prettytable module for Python 2.7: https://code.google.com/p/prettytable/wiki/Installation
- You can check if you have correctly installed the above modules (or you already have them) by starting Python and importing both modules:
- Installed Git open source distributed version control system.
- Installed at least one of the supported by Mbed SDK workspace tools compilers:
Compiler | Mbed SDK Abbreviation | Example Version |
---|---|---|
Keil ARM Compiler | ARM, uARM | ARM C/C++ Compiler, 5.03 [Build 117] |
GCC ARM | GCC_ARM | gcc version 4.8.3 20131129 (release) [ARM/embedded-4_8-branch revision 205641] (GNU Tools for ARM Embedded Processors) |
GCC CodeSourcery | GCC_CS | gcc version 4.8.1 (Sourcery CodeBench Lite 2013.11-24) |
GCC CodeRed | GCC_CR | gcc version 4.6.2 20121016 (release) [ARM/embedded-4_6-branch revision 192487] (GNU Tools for ARM Embedded Processors) |
IAR Embedded Workbench | IAR | IAR ANSI C/C++ Compiler V6.70.1.5641/W32 for ARM |
- Mbed board. You can find list of supported platforms here: https://mbed.org/platforms/
Test suite core: singletest.py script¶
singletest.py is a test suite script which allows users to compile, build tests and test runners (also supports CppUTest UT library). Script also is responsible for test execution on devices selected by configuration files.
Parameters of singletest.py¶
Test execution script singletest.py is a powerful tool to run tests for Mbed SDK platform. It is flexible and allows users to configure test execution process and define which mbed platforms will be tested.
By specifying external configuration files (in JSON format) you can gain flexibility and prepare many different test scenarios. Just pass configuration file names to your script and run it.
MUTs Specification¶
You can easily configure your MUTs (Mbed Under Test) by creating configuration file with MUTs' description.
Note: This configuration file must be in JSON format (http://www.w3schools.com/json/).
Note: Unfortunately JSON format is not allowing you to have comments inside JSON code.
Example 1 - three MUTs configuration (no peripherals)¶
Lets see some example and let's try to configure small "test farm" with three devices connected to your host computer. In this example no peripherals (like SD card or EEPROM) are connected to our Mbed boards. We will use three platforms in this example:
- NXP LPC1768 board https://mbed.org/platforms/mbed-LPC1768.
- Freescale KL25Z board https://mbed.org/platforms/KL25Z and
- STMicroelectronics NUCLEO_F103RB board https://mbed.org/platforms/ST-Nucleo-F103RB.
After connecting boards to our host machine (PC) we can check which serial ports and disks they occupy. For our example let's assume that:
- LPC1768 is on COM4 and disk drive is J:.
- KL25Z is on COM39 and disk drive is E:.
- NUCLEO_F103RB is on COM11 and disk drive is I:.
If you are working under Linux your port and disk could look like /dev/ttyACM5 and /media/usb5.
This information is needed to create muts_all.json configuration file. Its name will be passed to singletest.py script after -M (MUTs specification) switch. let's see how this file's content would look like in our example below:
muts_all.json
{ "1" : {"mcu": "LPC1768", "port":"COM4", "disk":"J:\\", "peripherals": [] }, "2" : {"mcu": "KL25Z", "port":"COM39", "disk":"E:\\", "peripherals": [] }, "3" : {"mcu": "NUCLEO_F103RB", "port":"COM11", "disk":"I:\\", "peripherals": [] } }
Note: We will leave field "peripherals" empty for the sake of this example. We will explain it later. All you need to do now is to properly fill fields "mcu", "port" and "disk".
Note: Please make sure files muts_all.json and test_spec.json are in workspace_tools/ directory. We will assume in this example they are.
Where to find "mcu" names? You can use option -S of build script build.py to check all supported off-line MCUs' names. Calling build.py script with -S option in workspace_tools/ directory should give you below list:
build.py -S
~\workspace_tools> build.py -S +---------------+-----------+-----------+-----------+-----------+-----------+-----------+------------+---------------+ | Platform | ARM | uARM | GCC_ARM | GCC_CS | GCC_CR | IAR | GCC_CW_EWL | GCC_CW_NEWLIB | +---------------+-----------+-----------+-----------+-----------+-----------+-----------+------------+---------------+ | ARCH_BLE | Default | - | Supported | - | - | - | - | - | | ARCH_PRO | Default | Supported | Supported | Supported | Supported | Supported | - | - | | ARM_MPS2 | Default | - | Supported | - | - | - | - | - | | DISCO_F051R8 | - | Default | Supported | - | - | - | - | - | | DISCO_F100RB | - | Default | Supported | - | - | - | - | - | | DISCO_F303VC | - | Default | Supported | - | - | - | - | - | | DISCO_F407VG | - | Default | Supported | - | - | - | - | - | | HRM1017 | Default | - | Supported | - | - | - | - | - | | K20D5M | Default | - | Supported | - | - | - | - | - | | K64F | Default | - | Supported | - | - | - | - | - | | KL05Z | Supported | Default | Supported | - | - | - | - | - | | KL25Z | Default | - | Supported | - | - | - | Supported | Supported | | KL46Z | Default | - | Supported | - | - | - | - | - | | LPC1114 | Supported | Default | Supported | - | Supported | - | - | - | | LPC11C24 | Default | Supported | Supported | - | - | - | - | - | | LPC11U24 | Supported | Default | Supported | - | - | - | - | - | | LPC11U24_301 | Default | Supported | Supported | - | - | - | - | - | | LPC11U35_401 | Supported | Default | Supported | - | Supported | - | - | - | | LPC11U35_501 | Supported | Default | Supported | - | Supported | - | - | - | | LPC11U68 | - | Default | Supported | - | Supported | - | - | - | | LPC1347 | Default | - | Supported | - | - | - | - | - | | LPC1549 | - | Default | - | - | Supported | - | - | - | | LPC1768 | Default | Supported | Supported | Supported | Supported | Supported | - | - | | LPC2368 | Default | - | Supported | - | Supported | - | - | - | | LPC4088 | Default | - | Supported | - | Supported | - | - | - | | LPC4330_M4 | Default | - | Supported | - | Supported | Supported | - | - | | LPC810 | - | Default | - | - | - | - | - | - | | LPC812 | - | Default | - | - | - | - | - | - | | LPCCAPPUCCINO | - | Default | Supported | - | Supported | - | - | - | | NRF51822 | Default | - | Supported | - | - | - | - | - | | NUCLEO_F030R8 | Supported | Default | - | - | - | - | - | - | | NUCLEO_F072RB | Supported | Default | - | - | - | - | - | - | | NUCLEO_F103RB | Supported | Default | Supported | - | - | - | - | - | | NUCLEO_F302R8 | Supported | Default | - | - | - | - | - | - | | NUCLEO_F334R8 | Supported | Default | - | - | - | - | - | - | | NUCLEO_F401RE | Supported | Default | Supported | - | - | - | - | - | | NUCLEO_F411RE | Supported | Default | - | - | - | - | - | - | | NUCLEO_L053R8 | Supported | Default | - | - | - | - | - | - | | NUCLEO_L152RE | Supported | Default | - | - | - | - | - | - | | STM32F3XX | Supported | Default | Supported | - | - | - | - | - | | STM32F407 | Default | - | Supported | - | - | - | - | - | | UBLOX_C027 | Default | Supported | Supported | Supported | Supported | Supported | - | - | | XADOW_M0 | Supported | Default | Supported | - | Supported | - | - | - | +---------------+-----------+-----------+-----------+-----------+-----------+-----------+------------+---------------+ *Default - default on-line compiler *Supported - supported off-line compiler Total permutations: 119
Note: If you update devices' firmware or even disconnect / reconnect Mbed device you may find that serial port / disk configuration changed. You need to update configuration file accordingly or you will face connection problems and obviously tests will not run.
Now we've already defined how our devices are connected to our host PC. We can continue and define which of this MUTs will be tested and which compilers we will use to compile and build Mbed SDK and tests. To do so we need to create test specification file (let's call it test_spec.json) and put inside our configuration file information about which MUTs we actually want to test. We will pass this file's name to singletest.py script using -i switch.
Below we can see how sample test_spec.json file content could look like. (I've included all possible toolchains, we will change it in a moment).
test_spec.json
{ "targets": { "LPC1768" : ["ARM", "uARM", "GCC_ARM", "GCC_CS", "GCC_CR", "IAR"], "KL25Z" : ["ARM", "GCC_ARM"], "NUCLEO_F103RB" : ["ARM", "uARM"] } }
Above example configuration will force tests for LPC1768, KL25Z, NUCLEO_F103RB platforms and:
- Compilers: ARM, uARM, GCC_ARM, GCC_CS, GCC_CR and IAR will be used to compile tests for LPC1768.
- Compilers: ARM and GCC_ARM will be used for KL25Z platform.
- Compilers: ARM and uARM will be used for NUCLEO_F103RB platform.
For our example purposes let's assume we only have Keil ARM compiler, so let's change configuration in test_spec.json file and reduce number of compiler to those we actually have:
test_spec.json
{ "targets": { "LPC1768" : ["ARM", "uARM"], "KL25Z" : ["ARM"], "NUCLEO_F103RB" : ["ARM", "uARM"] } }
Example 1 - Configuration summary¶
We've successfully connected all parts of our small "test farm" in two files test_spec.json and muts_all.json. Let's see how configuration logic looks like in Example 1:
Note: uARM is flavor of ARM compiler. In case of uARM we will link with microlib so we can save some space on your Mbed flash. There are other limitations concerning using of microlib (e.g. it does not support C++) but we will not discuss them here further.
Just a reminder. If you are unsure which Mbed platform supports which toolchains please go to workspace_tools/ directory in Mbed SDK repository and call build.py script with option -S:
c:\temp\repos\mbed\workspace_tools> python build.py -S +---------------+-----------+-----------+-----------+-----------+-----------+-----------+------------+---------------+ | Platform | ARM | uARM | GCC_ARM | GCC_CS | GCC_CR | IAR | GCC_CW_EWL | GCC_CW_NEWLIB | +---------------+-----------+-----------+-----------+-----------+-----------+-----------+------------+---------------+ ... | KL25Z | Supported | - | Supported | - | - | - | Supported | Supported | ... | LPC1768 | Supported | Supported | Supported | Supported | Supported | Supported | - | - | ... | NUCLEO_F103RB | Supported | Supported | Supported | - | - | - | - | - | ... +---------------+-----------+-----------+-----------+-----------+-----------+-----------+------------+---------------+ Total permutations: 101
Example 1 - Run your tests¶
After you configure all your MUTs and compilers you are ready to run tests. Make sure your devices are connected and your configuration files reflect your current configuration (serial ports, devices). Go to workspace_tools directory in your mbed location.
cd workspace_tools/
and execute test script.
singletest.py -i test_spec.json -M muts_all.json
Example 1 - summary¶
Having multiple configuration files allows you to manage your test scenarios in more flexible manner. You can:
- set up all platforms and toolchains used during testing,
- define (using script's -n switch) which tests you want to run during testing or
- just run regression (all tests). Regression is default setting for test script.
You can also force singletest.py script to:
- run only peripherals' tests (switch -P) or
- just skip peripherals' tests (switch -C).
Below you can see full singletes.py help command (-h switch) listing:
c:\temp\repos\mbed\workspace_tools> singletest.py -h Usage: singletest.py [options] This script allows you to run mbed defined test cases for particular MCU(s) and corresponding toolchain(s). Options: -h, --help show this help message and exit -i FILE, --tests=FILE Points to file with test specification -M FILE, --MUTS=FILE Points to file with MUTs specification (overwrites settings.py and private_settings.py) -g, --goanna-for-tests Run Goanna static analyse tool for tests -G, --goanna-for-sdk Run Goanna static analyse tool for mbed SDK -s, --suppress-summary Suppresses display of wellformatted table with test results -t, --test-summary Dsiplays wellformatted tablew ith test x toolchain test result per target -r, --test-automation-report Prints information about all tests and exits -P, --only-peripherals Test only peripheral declared for MUT and skip common tests -C, --only-commons Test only board internals. Skip perpherials tests and perform common tests. -n TEST_BY_NAMES, --test-by-names=TEST_BY_NAMES Runs only test enumerated it this switch -S, --supported-toolchains Displays supported matrix of MCUs and toolchains -v, --verbose Verbose mode (pronts some extra information) Example: singletest.py -i test_spec.json -M muts_all.json
Example 2¶
This example will show you how to configure single device, run general tests or only peripheral tests. We will also show some real test result examples.
- We will test only one board ST Nucleo board F334R8 connected to our PC (port COM 46 and disk is E:).
- We will also connect EEPROM 24LC256 to SDA, SCL pins of our Nucleo board and define 24LC256 peripheral to make sure our test suite will run all available tests for 24LC256.
- Let's configure our one MUT and set uARM as the only compiler we will use to compiler Mbed SDK and tests.
- We also need to create two configuration files muts_all.json and test_spec.json to pass our small testbed configuration to test script.
muts_all.json
{ "1" : { "mcu": "NUCLEO_F334R8", "port":"COM46", "disk":"E:\\", "peripherals": ["24LC256"] } }
Note: By defining "peripherals": ["24LC256"] we are passing to test suite information that this particular board has EEPROM 24LC256 connected to our board.
test_spec.json
{ "targets": { "NUCLEO_F334R8" : ["uARM"] } }
Above configuration now looks like this:
- Please make sure device is connected before we will start running tests.
- Please make sure files muts_all.json and test_spec.json are in workspace_tools/ directory. We will assume in this example they are.
To build Mbed SDK, compile, build and run tests please execute below command in workspace_tools/ directory.
singletest.py -i test_spec.json -M muts_all.json
~\workspace_tools> singletest.py -i test_spec.json -M muts_all.json >>> BUILD LIBRARY CMSIS (NUCLEO_F334R8, uARM) >>> BUILD LIBRARY MBED (NUCLEO_F334R8, uARM) >>> BUILD PROJECT: I2C_EEPROM_LINE (NUCLEO_F334R8, uARM) TargetTest::NUCLEO_F334R8::uARM::MBED_A25::I2C EEPROM line read/write test [OK] in 12.41 of 15 sec >>> BUILD PROJECT: BASIC (NUCLEO_F334R8, uARM) TargetTest::NUCLEO_F334R8::uARM::MBED_A1::Basic [OK] in 3.42 of 10 sec >>> BUILD PROJECT: DEV_NULL (NUCLEO_F334R8, uARM) TargetTest::NUCLEO_F334R8::uARM::EXAMPLE_1::/dev/null [OK] in 3.42 of 10 sec >>> BUILD PROJECT: TIMEOUT (NUCLEO_F334R8, uARM) TargetTest::NUCLEO_F334R8::uARM::MBED_24::Timeout Int us [OK] in 11.47 of 15 sec >>> BUILD PROJECT: TIME_US (NUCLEO_F334R8, uARM) TargetTest::NUCLEO_F334R8::uARM::MBED_25::Time us [OK] in 11.43 of 15 sec >>> BUILD PROJECT: DIV (NUCLEO_F334R8, uARM) TargetTest::NUCLEO_F334R8::uARM::MBED_26::Integer constant division [OK] in 3.37 of 10 sec >>> BUILD PROJECT: TICKER_2 (NUCLEO_F334R8, uARM) TargetTest::NUCLEO_F334R8::uARM::MBED_23::Ticker Int us [OK] in 12.43 of 15 sec >>> BUILD PROJECT: I2C_EEPROM (NUCLEO_F334R8, uARM) TargetTest::NUCLEO_F334R8::uARM::MBED_A19::I2C EEPROM read/write test [OK] in 11.42 of 15 sec >>> BUILD PROJECT: TICKER (NUCLEO_F334R8, uARM) TargetTest::NUCLEO_F334R8::uARM::MBED_11::Ticker Int [OK] in 12.43 of 20 sec >>> BUILD PROJECT: HELLO (NUCLEO_F334R8, uARM) TargetTest::NUCLEO_F334R8::uARM::MBED_10::Hello World [OK] in 2.42 of 10 sec >>> BUILD PROJECT: CPP (NUCLEO_F334R8, uARM) TargetTest::NUCLEO_F334R8::uARM::MBED_12::C++ [OK] in 3.42 of 10 sec >>> BUILD PROJECT: RTC (NUCLEO_F334R8, uARM) TargetTest::NUCLEO_F334R8::uARM::MBED_16::RTC [OK] in 4.76 of 15 sec >>> BUILD PROJECT: STDIO (NUCLEO_F334R8, uARM) TargetTest::NUCLEO_F334R8::uARM::MBED_2::stdio [UNDEF] in 20.42 of 20 sec >>> BUILD PROJECT: ECHO (NUCLEO_F334R8, uARM) TargetTest::NUCLEO_F334R8::uARM::MBED_A9::Serial Echo at 115200 [UNDEF] in 10.37 of 10 sec Test summary: +--------+---------------+-----------+-----------+---------------------------------+--------------------+---------------+ | Result | Target | Toolchain | Test ID | Test Description | Elapsed Time (sec) | Timeout (sec) | +--------+---------------+-----------+-----------+---------------------------------+--------------------+---------------+ | OK | NUCLEO_F334R8 | uARM | MBED_A25 | I2C EEPROM line read/write test | 12.41 | 15 | | OK | NUCLEO_F334R8 | uARM | MBED_A1 | Basic | 3.42 | 10 | | OK | NUCLEO_F334R8 | uARM | EXAMPLE_1 | /dev/null | 3.42 | 10 | | OK | NUCLEO_F334R8 | uARM | MBED_24 | Timeout Int us | 11.47 | 15 | | OK | NUCLEO_F334R8 | uARM | MBED_25 | Time us | 11.43 | 15 | | OK | NUCLEO_F334R8 | uARM | MBED_26 | Integer constant division | 3.37 | 10 | | OK | NUCLEO_F334R8 | uARM | MBED_23 | Ticker Int us | 12.43 | 15 | | OK | NUCLEO_F334R8 | uARM | MBED_A19 | I2C EEPROM read/write test | 11.42 | 15 | | OK | NUCLEO_F334R8 | uARM | MBED_11 | Ticker Int | 12.43 | 20 | | OK | NUCLEO_F334R8 | uARM | MBED_10 | Hello World | 2.42 | 10 | | OK | NUCLEO_F334R8 | uARM | MBED_12 | C++ | 3.42 | 10 | | OK | NUCLEO_F334R8 | uARM | MBED_16 | RTC | 4.76 | 15 | | UNDEF | NUCLEO_F334R8 | uARM | MBED_2 | stdio | 20.42 | 20 | | UNDEF | NUCLEO_F334R8 | uARM | MBED_A9 | Serial Echo at 115200 | 10.37 | 10 | +--------+---------------+-----------+-----------+---------------------------------+--------------------+---------------+ Result: 2 UNDEF / 12 OK Completed in 160 sec
If we want to get additional test summary with results in separate columns please use option -t.
singletest.py -i test_spec.json -M muts_all.json -t
Test summary: +---------------+-----------+---------------------------------+-------+ | Target | Test ID | Test Description | uARM | +---------------+-----------+---------------------------------+-------+ | NUCLEO_F334R8 | EXAMPLE_1 | /dev/null | OK | | NUCLEO_F334R8 | MBED_10 | Hello World | OK | | NUCLEO_F334R8 | MBED_11 | Ticker Int | OK | | NUCLEO_F334R8 | MBED_12 | C++ | OK | | NUCLEO_F334R8 | MBED_16 | RTC | OK | | NUCLEO_F334R8 | MBED_2 | stdio | UNDEF | | NUCLEO_F334R8 | MBED_23 | Ticker Int us | OK | | NUCLEO_F334R8 | MBED_24 | Timeout Int us | OK | | NUCLEO_F334R8 | MBED_25 | Time us | OK | | NUCLEO_F334R8 | MBED_26 | Integer constant division | OK | | NUCLEO_F334R8 | MBED_A1 | Basic | OK | | NUCLEO_F334R8 | MBED_A19 | I2C EEPROM read/write test | OK | | NUCLEO_F334R8 | MBED_A25 | I2C EEPROM line read/write test | OK | | NUCLEO_F334R8 | MBED_A9 | Serial Echo at 115200 | UNDEF | +---------------+-----------+---------------------------------+-------+
To only compile all required libraries and tests please use option -O (capital 'o').
singletest.py -i test_spec.json -M muts_all.json -O
workspace_tools> singletest.py -i test_spec.json -M muts_all.json -O >>> BUILD LIBRARY CMSIS (NUCLEO_F334R8, uARM) >>> BUILD LIBRARY MBED (NUCLEO_F334R8, uARM) >>> BUILD PROJECT: I2C_EEPROM_LINE (NUCLEO_F334R8, uARM) >>> BUILD PROJECT: BASIC (NUCLEO_F334R8, uARM) >>> BUILD PROJECT: DEV_NULL (NUCLEO_F334R8, uARM) >>> BUILD PROJECT: TIMEOUT (NUCLEO_F334R8, uARM) >>> BUILD PROJECT: TIME_US (NUCLEO_F334R8, uARM) >>> BUILD PROJECT: DIV (NUCLEO_F334R8, uARM) >>> BUILD PROJECT: TICKER_2 (NUCLEO_F334R8, uARM) >>> BUILD PROJECT: I2C_EEPROM (NUCLEO_F334R8, uARM) >>> BUILD PROJECT: TICKER (NUCLEO_F334R8, uARM) >>> BUILD PROJECT: HELLO (NUCLEO_F334R8, uARM) >>> BUILD PROJECT: CPP (NUCLEO_F334R8, uARM) >>> BUILD PROJECT: RTC (NUCLEO_F334R8, uARM) >>> BUILD PROJECT: STDIO (NUCLEO_F334R8, uARM) >>> BUILD PROJECT: ECHO (NUCLEO_F334R8, uARM) Completed in 1 sec
To add extra verbose prints please use option -v.
singletest.py -i test_spec.json -M muts_all.json -v
~\workspace_tools> singletest.py -i test_spec.json -M muts_all.json -v { u'targets': { u'NUCLEO_F334R8': [u'uARM']}} { u'1': { u'disk': u'E:\\', u'mcu': u'NUCLEO_F334R8', u'peripherals': [u'24LC256'], u'port': u'COM46'}} >>> BUILD LIBRARY CMSIS (NUCLEO_F334R8, uARM) >>> BUILD LIBRARY MBED (NUCLEO_F334R8, uARM) TargetTest::NUCLEO_F334R8::TestSkipped(ethernet) TargetTest::NUCLEO_F334R8::TestSkipped(ethernet) TargetTest::NUCLEO_F334R8::TestSkipped(ethernet) TargetTest::NUCLEO_F334R8::TestSkipped(ethernet) TargetTest::NUCLEO_F334R8::TestSkipped(ethernet) TargetTest::NUCLEO_F334R8::TestSkipped(ethernet) TargetTest::NUCLEO_F334R8::TestSkipped(ethernet) TargetTest::NUCLEO_F334R8::TestSkipped(ethernet) TargetTest::NUCLEO_F334R8::TestSkipped(ethernet) >>> BUILD PROJECT: I2C_EEPROM_LINE (NUCLEO_F334R8, uARM) Test::Output::Start Mbed: "COM46" "E:\" I2C: I2C Frequency: 400000 Hz I2C: Lines pattern write test ... [OK] I2C: Write errors: 0 ... [OK] I2C: Lines pattern read test ... [OK] I2C: Read errors: 0/1000 ... [OK] EEPROM: Pattern match errors: 0/1000 ... [OK] {{success}} {{end}} Test::Output::Finish TargetTest::NUCLEO_F334R8::uARM::MBED_A25::I2C EEPROM line read/write test [OK] in 12.40 of 15 sec >>> BUILD PROJECT: BASIC (NUCLEO_F334R8, uARM) Test::Output::Start Mbed: "COM46" "E:\" {{success}} {{end}} Test::Output::Finish TargetTest::NUCLEO_F334R8::uARM::MBED_A1::Basic [OK] in 3.42 of 10 sec >>> BUILD PROJECT: DEV_NULL (NUCLEO_F334R8, uARM) Test::Output::Start Mbed: "COM46" "E:\" Received 33 bytes {success} {end} Test::Output::Finish TargetTest::NUCLEO_F334R8::uARM::EXAMPLE_1::/dev/null [OK] in 3.42 of 10 sec TargetTest::NUCLEO_F334R8::TestSkipped(digital_loop) TargetTest::NUCLEO_F334R8::TestSkipped(TMP102) TargetTest::NUCLEO_F334R8::TestSkipped(digital_loop) >>> BUILD PROJECT: TIMEOUT (NUCLEO_F334R8, uARM) Test::Output::Start Mbed: "COM46" "E:\" Test started . skipped . skipped . skipped . in 1.01 sec (0.01) [OK] . in 1.00 sec (0.00) [OK] . in 1.01 sec (0.01) [OK] . in 1.01 sec (0.01) [OK] . in 1.01 sec (0.01) [OK] . in 1.01 sec (0.01) [OK] . in 1.01 sec (0.01) [OK] Completed in 9.08 sec {success} {end} Test::Output::Finish TargetTest::NUCLEO_F334R8::uARM::MBED_24::Timeout Int us [OK] in 11.54 of 15 sec >>> BUILD PROJECT: TIME_US (NUCLEO_F334R8, uARM) Test::Output::Start Mbed: "COM46" "E:\" Test started . skipped . skipped . skipped . in 1.00 sec (0.00) [OK] . in 1.00 sec (0.00) [OK] . in 1.00 sec (0.00) [OK] . in 1.00 sec (0.00) [OK] . in 1.00 sec (0.00) [OK] . in 1.00 sec (0.00) [OK] . in 1.00 sec (0.00) [OK] Completed in 9.01 sec {success} {end} Test::Output::Finish TargetTest::NUCLEO_F334R8::uARM::MBED_25::Time us [OK] in 11.48 of 15 sec >>> BUILD PROJECT: DIV (NUCLEO_F334R8, uARM) Test::Output::Start Mbed: "COM46" "E:\" 64bit: 0x7FFFFFFF8: expected 0x55555555 got 0x55555555 ... [OK] 64bit: 0x17FFFFFFE8: expected 0xFFFFFFFF got 0xFFFFFFFF ... [OK] {{success}} {{end}} Test::Output::Finish TargetTest::NUCLEO_F334R8::uARM::MBED_26::Integer constant division [OK] in 3.43 of 10 sec >>> BUILD PROJECT: TICKER_2 (NUCLEO_F334R8, uARM) Test::Output::Start Mbed: "COM46" "E:\" Test started . skipped . skipped . skipped . in 1.00 sec (0.00) [OK] . in 1.00 sec (0.00) [OK] . in 1.00 sec (0.00) [OK] . in 1.00 sec (0.00) [OK] . in 1.00 sec (0.00) [OK] . in 1.00 sec (0.00) [OK] . in 1.00 sec (0.00) [OK] Completed in 9.00 sec {success} {end} Test::Output::Finish TargetTest::NUCLEO_F334R8::uARM::MBED_23::Ticker Int us [OK] in 11.45 of 15 sec >>> BUILD PROJECT: I2C_EEPROM (NUCLEO_F334R8, uARM) Test::Output::Start Mbed: "COM46" "E:\" I2C: I2C Frequency: 400000 Hz I2C: Write 0x66 at address 0x0000 test ... Unable to write data to EEPROM (i2c_stat = 0x01), aborting {{failure}} {{end}} Test::Output::Finish TargetTest::NUCLEO_F334R8::uARM::MBED_A19::I2C EEPROM read/write test [FAIL] in 3.43 of 15 sec TargetTest::NUCLEO_F334R8::TestSkipped(port_loop) TargetTest::NUCLEO_F334R8::TestSkipped(port_loop) TargetTest::NUCLEO_F334R8::TestSkipped(SD) >>> BUILD PROJECT: TICKER (NUCLEO_F334R8, uARM) Test::Output::Start Mbed: "COM46" "E:\" Test started . skipped . skipped . skipped . in 1.00 sec (0.00) [OK] . in 1.00 sec (0.00) [OK] . in 1.00 sec (0.00) [OK] . in 1.00 sec (0.00) [OK] . in 1.00 sec (0.00) [OK] . in 1.00 sec (0.00) [OK] . in 1.00 sec (0.00) [OK] Completed in 9.00 sec {success} {end} Test::Output::Finish TargetTest::NUCLEO_F334R8::uARM::MBED_11::Ticker Int [OK] in 11.43 of 20 sec >>> BUILD PROJECT: HELLO (NUCLEO_F334R8, uARM) Test::Output::Start Mbed: "COM46" "E:\" Hello World {success} {end} Test::Output::Finish TargetTest::NUCLEO_F334R8::uARM::MBED_10::Hello World [OK] in 2.43 of 10 sec >>> BUILD PROJECT: CPP (NUCLEO_F334R8, uARM) Test::Output::Start Mbed: "COM46" "E:\" Static::init Static::stack_test Stack::init Stack::hello Stack::destroy Static::check_init: OK Heap::init Heap::hello Heap::check_init: OK Heap::destroy {{success}} {{end}} Test::Output::Finish TargetTest::NUCLEO_F334R8::uARM::MBED_12::C++ [OK] in 3.43 of 10 sec >>> BUILD PROJECT: RTC (NUCLEO_F334R8, uARM) Test::Output::Start Mbed: "COM46" "E:\" Got RTC time: [1256729737] [2009-10-28 11:35:37 AM] ... OK Got RTC time: [1256729738] [2009-10-28 11:35:38 AM] ... OK Got RTC time: [1256729739] [2009-10-28 11:35:39 AM] ... OK Got RTC time: [1256729740] [2009-10-28 11:35:40 AM] ... OK Got RTC time: [1256729741] [2009-10-28 11:35:41 AM] ... OK {success} {end} Test::Output::Finish TargetTest::NUCLEO_F334R8::uARM::MBED_16::RTC [OK] in 4.72 of 15 sec >>> BUILD PROJECT: STDIO (NUCLEO_F334R8, uARM) Test::Output::Start Mbed: "COM46" "E:\" Generated number: 3112 Test::Output::Finish TargetTest::NUCLEO_F334R8::uARM::MBED_2::stdio [UNDEF] in 20.38 of 20 sec TargetTest::NUCLEO_F334R8::TestSkipped(digital_loop) >>> BUILD PROJECT: ECHO (NUCLEO_F334R8, uARM) Test::Output::Start Mbed: "COM46" "E:\" Starting the ECHO test Test::Output::Finish TargetTest::NUCLEO_F334R8::uARM::MBED_A9::Serial Echo at 115200 [UNDEF] in 10.43 of 10 sec Test summary: +--------+---------------+-----------+-----------+---------------------------------+--------------------+---------------+ | Result | Target | Toolchain | Test ID | Test Description | Elapsed Time (sec) | Timeout (sec) | +--------+---------------+-----------+-----------+---------------------------------+--------------------+---------------+ | OK | NUCLEO_F334R8 | uARM | MBED_A25 | I2C EEPROM line read/write test | 12.4 | 15 | | OK | NUCLEO_F334R8 | uARM | MBED_A1 | Basic | 3.42 | 10 | | OK | NUCLEO_F334R8 | uARM | EXAMPLE_1 | /dev/null | 3.42 | 10 | | OK | NUCLEO_F334R8 | uARM | MBED_24 | Timeout Int us | 11.54 | 15 | | OK | NUCLEO_F334R8 | uARM | MBED_25 | Time us | 11.48 | 15 | | OK | NUCLEO_F334R8 | uARM | MBED_26 | Integer constant division | 3.43 | 10 | | OK | NUCLEO_F334R8 | uARM | MBED_23 | Ticker Int us | 11.45 | 15 | | FAIL | NUCLEO_F334R8 | uARM | MBED_A19 | I2C EEPROM read/write test | 3.43 | 15 | | OK | NUCLEO_F334R8 | uARM | MBED_11 | Ticker Int | 11.43 | 20 | | OK | NUCLEO_F334R8 | uARM | MBED_10 | Hello World | 2.43 | 10 | | OK | NUCLEO_F334R8 | uARM | MBED_12 | C++ | 3.43 | 10 | | OK | NUCLEO_F334R8 | uARM | MBED_16 | RTC | 4.72 | 15 | | UNDEF | NUCLEO_F334R8 | uARM | MBED_2 | stdio | 20.38 | 20 | | UNDEF | NUCLEO_F334R8 | uARM | MBED_A9 | Serial Echo at 115200 | 10.43 | 10 | +--------+---------------+-----------+-----------+---------------------------------+--------------------+---------------+ Result: 1 FAIL / 2 UNDEF / 11 OK Completed in 150 sec
Shuffle your tests¶
What is host test?¶
Test suite support test supervisor concept. This concept is realized by separate Python script called "host test". Host tests can be found in workspace_tools/host_tests/ directory. Host test script is executed in parallel with test runner to monitor test execution. Basic host test just monitors device's default serial port for test results returned by test runner. Simple tests will print test result on serial port. In other cases host tests can for example judge by test results returned by test runner is test passed or failed. It all depends on test itself. In some cases host test can be TCP server echoing packets from test runner and judging packet loss. In other cases it can just check if values returned from accelerometer are actually valid (sane).
Peripherals testing¶
- only peripherals -P switch
MUTs specification¶
muts_all.json
Test specification¶
test_spec.JSON xxx
CI integration / verbose modes¶
xxx
Vocabulary¶
Test runner¶
A test runner is an executable program that runs tests implemented using test framework and reports the test results.
Appendix A¶
Example muts_all.json file¶
muts_all.json
{ "1" : {"mcu" : "LPC1768", "port" : "COM4", "disk" : "J:\\", "peripherals" : ["TMP102", "digital_loop", "port_loop", "analog_loop", "SD"]}, "2" : {"mcu" : "KL25Z", "port" : "COM39", "disk" : "E:\\", "peripherals" : ["SD", "24LC256", "KL25Z"]}, "3" : {"mcu" : "LPC11U24", "port" : "COM5", "disk" : "E:\\", "peripherals" : ["TMP102", "digital_loop", "port_loop", "analog_loop"]}, "4" : {"mcu" : "NUCLEO_F103RB", "port" : "COM11", "disk" : "I:\\", "peripherals" : ["24LC256"]}, "5" : {"mcu" : "NRF51822", "port" : "COM21", "disk" : "I:\\", "peripherals" : []}, "6" : {"mcu" : "UBLOX_C027", "port" : "COM14", "disk" : "F:\\", "peripherals" : []}, "7" : {"mcu" : "LPC2368", "port" : "COM15", "disk" : "G:\\", "peripherals" : ["TMP102", "digital_loop", "port_loop", "analog_loop", "SD"]}, "8" : {"mcu" : "LPC812", "port" : "COM6", "disk" : "E:\\", "peripherals" : ["24LC256"]}, "9" : {"mcu" : "LPC1549", "port" : "COM32", "disk" : "I:\\", "peripherals" : ["24LC256"]}, "10" : {"mcu" : "LPC4088", "port" : "COM17", "disk" : "E:\\", "peripherals" : []}, "11" : {"mcu" : "KL46Z", "port" : "COM40", "disk" : "F:\\", "peripherals" : ["SD", "24LC256", "KL46Z"]}, "12" : {"mcu" : "K64F", "port" : "COM18", "disk" : "E:\\", "peripherals" : ["ethernet"]}, "13" : {"mcu" : "NUCLEO_L152RE", "port" : "COM9", "disk" : "E:\\", "peripherals" : []}, "14" : {"mcu" : "NUCLEO_F302R8", "port" : "COM34", "disk" : "E:\\", "peripherals" : ["24LC256"]}, "15" : {"mcu" : "NUCLEO_F401RE", "port" : "COM47", "disk" : "E:\\", "peripherals" : ["SD"]}, "16" : {"mcu" : "NUCLEO_F030R8", "port" : "COM12", "disk" : "H:\\", "peripherals" : ["24LC256"]}, "17" : {"mcu" : "NUCLEO_L053R8", "port" : "COM35", "disk" : "E:\\", "peripherals" : ["24LC256"]}, "18" : {"mcu" : "LPC11U68", "port" : "COM36", "disk" : "E:\\", "peripherals" : ["24LC256"]}, "19" : {"mcu" : "NUCLEO_F411RE", "port" : "COM44", "disk" : "F:\\", "peripherals" : ["24LC256"]}, "20" : {"mcu" : "NUCLEO_F072RB", "port" : "COM45", "disk" : "F:\\", "peripherals" : ["24LC256"]}, "21" : {"mcu" : "NUCLEO_F334R8", "port" : "COM46", "disk" : "E:\\", "peripherals" : ["24LC256"]}, "22" : {"mcu" : "XADOW_M0", "port" : "COM51", "disk" : "E:\\", "peripherals" : []} }
Example test_spec.json file¶
test_spec.json
{ "targets": { "KL25Z": ["ARM", "GCC_ARM"], "LPC1768": ["ARM", "GCC_ARM", "GCC_CR", "GCC_CS", "IAR"], "LPC11U24": ["uARM"], "NRF51822": ["ARM"], "LPC2368": ["ARM"], "LPC4088": ["ARM"], "NUCLEO_F334R8" : ["uARM"] } }