Users » PrzemekWirkus » Notebook » Mbed SDK - automated test suite

Mbed SDK - automated test suite

Page last updated 08 Jul 2014, by Przemek Wirkus. 0 replies Automation, CI, continuous, integration, test, tools, workspace, workspace_tools

Mbed SDK automated test suite

Introduction

Test suit allows users to run locally on their machines Mbed SDK’s tests included in Mbed SDK repository. It also allows users to create their own tests and for example add new tests to test set as they progress with their project. If test is generic enough it could be included into official Mbed SDK test pool just do it via normal pull request!

Each test is supervised by python script called “host test” which will at least Test suite is using build script API to compile and build test source together with required by test libraries like CMSIS, Mbed, Ethernet, USB etc.

Prerequisites

Installed Python 2.7.6 programming language: https://www.python.org/download/releases/2.7.
- Installed pyserial module for Python 2.7: https://pypi.python.org/pypi/pyserial
- Installed prettytable module for Python 2.7: https://code.google.com/p/prettytable/wiki/Installation
- You can check if you have correctly installed the above modules (or you already have them) by starting Python and importing both modules:
Installed Git open source distributed version control system.
Installed at least one of the supported by Mbed SDK workspace tools compilers:

Compiler	Mbed SDK Abbreviation	Example Version
Keil ARM Compiler	ARM, uARM	ARM C/C++ Compiler, 5.03 [Build 117]
GCC ARM	GCC_ARM	gcc version 4.8.3 20131129 (release) [ARM/embedded-4_8-branch revision 205641] (GNU Tools for ARM Embedded Processors)
GCC CodeSourcery	GCC_CS	gcc version 4.8.1 (Sourcery CodeBench Lite 2013.11-24)
GCC CodeRed	GCC_CR	gcc version 4.6.2 20121016 (release) [ARM/embedded-4_6-branch revision 192487] (GNU Tools for ARM Embedded Processors)
IAR Embedded Workbench	IAR	IAR ANSI C/C++ Compiler V6.70.1.5641/W32 for ARM

Mbed board. You can find list of supported platforms here: https://mbed.org/platforms/

Test suite core: singletest.py script

singletest.py is a test suite script which allows users to compile, build tests and test runners (also supports CppUTest UT library). Script also is responsible for test execution on devices selected by configuration files.

Parameters of singletest.py

Test execution script singletest.py is a powerful tool to run tests for Mbed SDK platform. It is flexible and allows users to configure test execution process and define which mbed platforms will be tested.

By specifying external configuration files (in JSON format) you can gain flexibility and prepare many different test scenarios. Just pass configuration file names to your script and run it.

MUTs Specification

You can easily configure your MUTs (Mbed Under Test) by creating configuration file with MUTs' description.

Note: This configuration file must be in JSON format (http://www.w3schools.com/json/).

Note: Unfortunately JSON format is not allowing you to have comments inside JSON code.

Example 1 - three MUTs configuration (no peripherals)

Lets see some example and let's try to configure small "test farm" with three devices connected to your host computer. In this example no peripherals (like SD card or EEPROM) are connected to our Mbed boards. We will use three platforms in this example:

NXP LPC1768 board https://mbed.org/platforms/mbed-LPC1768.
Freescale KL25Z board https://mbed.org/platforms/KL25Z and
STMicroelectronics NUCLEO_F103RB board https://mbed.org/platforms/ST-Nucleo-F103RB.

After connecting boards to our host machine (PC) we can check which serial ports and disks they occupy. For our example let's assume that:

LPC1768 is on COM4 and disk drive is J:.
KL25Z is on COM39 and disk drive is E:.
NUCLEO_F103RB is on COM11 and disk drive is I:.

If you are working under Linux your port and disk could look like /dev/ttyACM5 and /media/usb5.

This information is needed to create muts_all.json configuration file. Its name will be passed to singletest.py script after -M (MUTs specification) switch. let's see how this file's content would look like in our example below:

muts_all.json

{
    "1" : {"mcu": "LPC1768",
        "port":"COM4",
        "disk":"J:\\",
        "peripherals": []
    },

    "2" : {"mcu": "KL25Z",
        "port":"COM39",
        "disk":"E:\\",
        "peripherals": []
    },

    "3" : {"mcu": "NUCLEO_F103RB",
        "port":"COM11",
        "disk":"I:\\",
        "peripherals": []
    }
}

Note: We will leave field "peripherals" empty for the sake of this example. We will explain it later. All you need to do now is to properly fill fields "mcu", "port" and "disk".

Note: Please make sure files muts_all.json and test_spec.json are in workspace_tools/ directory. We will assume in this example they are.

Where to find "mcu" names? You can use option -S of build script build.py to check all supported off-line MCUs' names. Calling build.py script with -S option in workspace_tools/ directory should give you below list:

build.py -S

~\workspace_tools> build.py -S
+---------------+-----------+-----------+-----------+-----------+-----------+-----------+------------+---------------+
| Platform      |    ARM    |    uARM   |  GCC_ARM  |   GCC_CS  |   GCC_CR  |    IAR    | GCC_CW_EWL | GCC_CW_NEWLIB |
+---------------+-----------+-----------+-----------+-----------+-----------+-----------+------------+---------------+
| ARCH_BLE      |  Default  |     -     | Supported |     -     |     -     |     -     |     -      |       -       |
| ARCH_PRO      |  Default  | Supported | Supported | Supported | Supported | Supported |     -      |       -       |
| ARM_MPS2      |  Default  |     -     | Supported |     -     |     -     |     -     |     -      |       -       |
| DISCO_F051R8  |     -     |  Default  | Supported |     -     |     -     |     -     |     -      |       -       |
| DISCO_F100RB  |     -     |  Default  | Supported |     -     |     -     |     -     |     -      |       -       |
| DISCO_F303VC  |     -     |  Default  | Supported |     -     |     -     |     -     |     -      |       -       |
| DISCO_F407VG  |     -     |  Default  | Supported |     -     |     -     |     -     |     -      |       -       |
| HRM1017       |  Default  |     -     | Supported |     -     |     -     |     -     |     -      |       -       |
| K20D5M        |  Default  |     -     | Supported |     -     |     -     |     -     |     -      |       -       |
| K64F          |  Default  |     -     | Supported |     -     |     -     |     -     |     -      |       -       |
| KL05Z         | Supported |  Default  | Supported |     -     |     -     |     -     |     -      |       -       |
| KL25Z         |  Default  |     -     | Supported |     -     |     -     |     -     | Supported  |   Supported   |
| KL46Z         |  Default  |     -     | Supported |     -     |     -     |     -     |     -      |       -       |
| LPC1114       | Supported |  Default  | Supported |     -     | Supported |     -     |     -      |       -       |
| LPC11C24      |  Default  | Supported | Supported |     -     |     -     |     -     |     -      |       -       |
| LPC11U24      | Supported |  Default  | Supported |     -     |     -     |     -     |     -      |       -       |
| LPC11U24_301  |  Default  | Supported | Supported |     -     |     -     |     -     |     -      |       -       |
| LPC11U35_401  | Supported |  Default  | Supported |     -     | Supported |     -     |     -      |       -       |
| LPC11U35_501  | Supported |  Default  | Supported |     -     | Supported |     -     |     -      |       -       |
| LPC11U68      |     -     |  Default  | Supported |     -     | Supported |     -     |     -      |       -       |
| LPC1347       |  Default  |     -     | Supported |     -     |     -     |     -     |     -      |       -       |
| LPC1549       |     -     |  Default  |     -     |     -     | Supported |     -     |     -      |       -       |
| LPC1768       |  Default  | Supported | Supported | Supported | Supported | Supported |     -      |       -       |
| LPC2368       |  Default  |     -     | Supported |     -     | Supported |     -     |     -      |       -       |
| LPC4088       |  Default  |     -     | Supported |     -     | Supported |     -     |     -      |       -       |
| LPC4330_M4    |  Default  |     -     | Supported |     -     | Supported | Supported |     -      |       -       |
| LPC810        |     -     |  Default  |     -     |     -     |     -     |     -     |     -      |       -       |
| LPC812        |     -     |  Default  |     -     |     -     |     -     |     -     |     -      |       -       |
| LPCCAPPUCCINO |     -     |  Default  | Supported |     -     | Supported |     -     |     -      |       -       |
| NRF51822      |  Default  |     -     | Supported |     -     |     -     |     -     |     -      |       -       |
| NUCLEO_F030R8 | Supported |  Default  |     -     |     -     |     -     |     -     |     -      |       -       |
| NUCLEO_F072RB | Supported |  Default  |     -     |     -     |     -     |     -     |     -      |       -       |
| NUCLEO_F103RB | Supported |  Default  | Supported |     -     |     -     |     -     |     -      |       -       |
| NUCLEO_F302R8 | Supported |  Default  |     -     |     -     |     -     |     -     |     -      |       -       |
| NUCLEO_F334R8 | Supported |  Default  |     -     |     -     |     -     |     -     |     -      |       -       |
| NUCLEO_F401RE | Supported |  Default  | Supported |     -     |     -     |     -     |     -      |       -       |
| NUCLEO_F411RE | Supported |  Default  |     -     |     -     |     -     |     -     |     -      |       -       |
| NUCLEO_L053R8 | Supported |  Default  |     -     |     -     |     -     |     -     |     -      |       -       |
| NUCLEO_L152RE | Supported |  Default  |     -     |     -     |     -     |     -     |     -      |       -       |
| STM32F3XX     | Supported |  Default  | Supported |     -     |     -     |     -     |     -      |       -       |
| STM32F407     |  Default  |     -     | Supported |     -     |     -     |     -     |     -      |       -       |
| UBLOX_C027    |  Default  | Supported | Supported | Supported | Supported | Supported |     -      |       -       |
| XADOW_M0      | Supported |  Default  | Supported |     -     | Supported |     -     |     -      |       -       |
+---------------+-----------+-----------+-----------+-----------+-----------+-----------+------------+---------------+
*Default - default on-line compiler
*Supported - supported off-line compiler

Total permutations: 119

Note: If you update devices' firmware or even disconnect / reconnect Mbed device you may find that serial port / disk configuration changed. You need to update configuration file accordingly or you will face connection problems and obviously tests will not run.

Now we've already defined how our devices are connected to our host PC. We can continue and define which of this MUTs will be tested and which compilers we will use to compile and build Mbed SDK and tests. To do so we need to create test specification file (let's call it test_spec.json) and put inside our configuration file information about which MUTs we actually want to test. We will pass this file's name to singletest.py script using -i switch.

Below we can see how sample test_spec.json file content could look like. (I've included all possible toolchains, we will change it in a moment).

test_spec.json

{
    "targets": {
        "LPC1768" : ["ARM", "uARM", "GCC_ARM", "GCC_CS", "GCC_CR", "IAR"],
        "KL25Z" : ["ARM", "GCC_ARM"],
        "NUCLEO_F103RB" : ["ARM", "uARM"]
     }
}

Above example configuration will force tests for LPC1768, KL25Z, NUCLEO_F103RB platforms and:

Compilers: ARM, uARM, GCC_ARM, GCC_CS, GCC_CR and IAR will be used to compile tests for LPC1768.
Compilers: ARM and GCC_ARM will be used for KL25Z platform.
Compilers: ARM and uARM will be used for NUCLEO_F103RB platform.

For our example purposes let's assume we only have Keil ARM compiler, so let's change configuration in test_spec.json file and reduce number of compiler to those we actually have:

test_spec.json

{
    "targets": {
        "LPC1768" : ["ARM", "uARM"],
        "KL25Z" : ["ARM"],
        "NUCLEO_F103RB" : ["ARM", "uARM"]
     }
}

Example 1 - Configuration summary

We've successfully connected all parts of our small "test farm" in two files test_spec.json and muts_all.json. Let's see how configuration logic looks like in Example 1:

/media/uploads/PrzemekWirkus/example_singletest_config.png

Note: uARM is flavor of ARM compiler. In case of uARM we will link with microlib so we can save some space on your Mbed flash. There are other limitations concerning using of microlib (e.g. it does not support C++) but we will not discuss them here further.

Just a reminder. If you are unsure which Mbed platform supports which toolchains please go to workspace_tools/ directory in Mbed SDK repository and call build.py script with option -S:

c:\temp\repos\mbed\workspace_tools> python build.py -S
+---------------+-----------+-----------+-----------+-----------+-----------+-----------+------------+---------------+
| Platform      |    ARM    |    uARM   |  GCC_ARM  |   GCC_CS  |   GCC_CR  |    IAR    | GCC_CW_EWL | GCC_CW_NEWLIB |
+---------------+-----------+-----------+-----------+-----------+-----------+-----------+------------+---------------+
...
| KL25Z         | Supported |     -     | Supported |     -     |     -     |     -     | Supported  |   Supported   |
...
| LPC1768       | Supported | Supported | Supported | Supported | Supported | Supported |     -      |       -       |
...
| NUCLEO_F103RB | Supported | Supported | Supported |     -     |     -     |     -     |     -      |       -       |
...
+---------------+-----------+-----------+-----------+-----------+-----------+-----------+------------+---------------+
Total permutations: 101

Example 1 - Run your tests

After you configure all your MUTs and compilers you are ready to run tests. Make sure your devices are connected and your configuration files reflect your current configuration (serial ports, devices). Go to workspace_tools directory in your mbed location.

cd workspace_tools/

and execute test script.

singletest.py -i test_spec.json -M muts_all.json

Example 1 - summary

Having multiple configuration files allows you to manage your test scenarios in more flexible manner. You can:

set up all platforms and toolchains used during testing,
define (using script's -n switch) which tests you want to run during testing or
just run regression (all tests). Regression is default setting for test script.

You can also force singletest.py script to:

run only peripherals' tests (switch -P) or
just skip peripherals' tests (switch -C).

Below you can see full singletes.py help command (-h switch) listing:

c:\temp\repos\mbed\workspace_tools> singletest.py -h
Usage: singletest.py [options]

This script allows you to run mbed defined test cases for particular MCU(s)
and corresponding toolchain(s).

Options:
  -h, --help            show this help message and exit
  -i FILE, --tests=FILE
                        Points to file with test specification
  -M FILE, --MUTS=FILE  Points to file with MUTs specification (overwrites
                        settings.py and private_settings.py)
  -g, --goanna-for-tests
                        Run Goanna static analyse tool for tests
  -G, --goanna-for-sdk  Run Goanna static analyse tool for mbed SDK
  -s, --suppress-summary
                        Suppresses display of wellformatted table with test
                        results
  -t, --test-summary    Dsiplays wellformatted tablew ith test x toolchain
                        test result per target
  -r, --test-automation-report
                        Prints information about all tests and exits
  -P, --only-peripherals
                        Test only peripheral declared for MUT and skip common
                        tests
  -C, --only-commons    Test only board internals. Skip perpherials tests and
                        perform common tests.
  -n TEST_BY_NAMES, --test-by-names=TEST_BY_NAMES
                        Runs only test enumerated it this switch
  -S, --supported-toolchains
                        Displays supported matrix of MCUs and toolchains
  -v, --verbose         Verbose mode (pronts some extra information)

Example: singletest.py -i test_spec.json -M muts_all.json

Example 2

This example will show you how to configure single device, run general tests or only peripheral tests. We will also show some real test result examples.

We will test only one board ST Nucleo board F334R8 connected to our PC (port COM 46 and disk is E:).
We will also connect EEPROM 24LC256 to SDA, SCL pins of our Nucleo board and define 24LC256 peripheral to make sure our test suite will run all available tests for 24LC256.

Let's configure our one MUT and set uARM as the only compiler we will use to compiler Mbed SDK and tests.
We also need to create two configuration files muts_all.json and test_spec.json to pass our small testbed configuration to test script.

muts_all.json

{
    "1" : { 
            "mcu": "NUCLEO_F334R8",
            "port":"COM46",
            "disk":"E:\\",
            "peripherals": ["24LC256"]
    }
}

Note: By defining "peripherals": ["24LC256"] we are passing to test suite information that this particular board has EEPROM 24LC256 connected to our board.

test_spec.json

{
    "targets": {
        "NUCLEO_F334R8" : ["uARM"]
     }
}

Above configuration now looks like this:

/media/uploads/PrzemekWirkus/example_2_config.png

Please make sure device is connected before we will start running tests.
Please make sure files muts_all.json and test_spec.json are in workspace_tools/ directory. We will assume in this example they are.

To build Mbed SDK, compile, build and run tests please execute below command in workspace_tools/ directory.

singletest.py -i test_spec.json -M muts_all.json

~\workspace_tools> singletest.py -i test_spec.json -M muts_all.json

>>> BUILD LIBRARY CMSIS (NUCLEO_F334R8, uARM)

>>> BUILD LIBRARY MBED (NUCLEO_F334R8, uARM)

>>> BUILD PROJECT: I2C_EEPROM_LINE (NUCLEO_F334R8, uARM)
TargetTest::NUCLEO_F334R8::uARM::MBED_A25::I2C EEPROM line read/write test [OK] in 12.41 of 15 sec

>>> BUILD PROJECT: BASIC (NUCLEO_F334R8, uARM)
TargetTest::NUCLEO_F334R8::uARM::MBED_A1::Basic [OK] in 3.42 of 10 sec

>>> BUILD PROJECT: DEV_NULL (NUCLEO_F334R8, uARM)
TargetTest::NUCLEO_F334R8::uARM::EXAMPLE_1::/dev/null [OK] in 3.42 of 10 sec

>>> BUILD PROJECT: TIMEOUT (NUCLEO_F334R8, uARM)
TargetTest::NUCLEO_F334R8::uARM::MBED_24::Timeout Int us [OK] in 11.47 of 15 sec

>>> BUILD PROJECT: TIME_US (NUCLEO_F334R8, uARM)
TargetTest::NUCLEO_F334R8::uARM::MBED_25::Time us [OK] in 11.43 of 15 sec

>>> BUILD PROJECT: DIV (NUCLEO_F334R8, uARM)
TargetTest::NUCLEO_F334R8::uARM::MBED_26::Integer constant division [OK] in 3.37 of 10 sec

>>> BUILD PROJECT: TICKER_2 (NUCLEO_F334R8, uARM)
TargetTest::NUCLEO_F334R8::uARM::MBED_23::Ticker Int us [OK] in 12.43 of 15 sec

>>> BUILD PROJECT: I2C_EEPROM (NUCLEO_F334R8, uARM)
TargetTest::NUCLEO_F334R8::uARM::MBED_A19::I2C EEPROM read/write test [OK] in 11.42 of 15 sec

>>> BUILD PROJECT: TICKER (NUCLEO_F334R8, uARM)
TargetTest::NUCLEO_F334R8::uARM::MBED_11::Ticker Int [OK] in 12.43 of 20 sec

>>> BUILD PROJECT: HELLO (NUCLEO_F334R8, uARM)
TargetTest::NUCLEO_F334R8::uARM::MBED_10::Hello World [OK] in 2.42 of 10 sec

>>> BUILD PROJECT: CPP (NUCLEO_F334R8, uARM)
TargetTest::NUCLEO_F334R8::uARM::MBED_12::C++ [OK] in 3.42 of 10 sec

>>> BUILD PROJECT: RTC (NUCLEO_F334R8, uARM)
TargetTest::NUCLEO_F334R8::uARM::MBED_16::RTC [OK] in 4.76 of 15 sec

>>> BUILD PROJECT: STDIO (NUCLEO_F334R8, uARM)
TargetTest::NUCLEO_F334R8::uARM::MBED_2::stdio [UNDEF] in 20.42 of 20 sec

>>> BUILD PROJECT: ECHO (NUCLEO_F334R8, uARM)
TargetTest::NUCLEO_F334R8::uARM::MBED_A9::Serial Echo at 115200 [UNDEF] in 10.37 of 10 sec
Test summary:
+--------+---------------+-----------+-----------+---------------------------------+--------------------+---------------+
| Result | Target        | Toolchain | Test ID   | Test Description                | Elapsed Time (sec) | Timeout (sec) |
+--------+---------------+-----------+-----------+---------------------------------+--------------------+---------------+
| OK     | NUCLEO_F334R8 | uARM      | MBED_A25  | I2C EEPROM line read/write test |       12.41        |       15      |
| OK     | NUCLEO_F334R8 | uARM      | MBED_A1   | Basic                           |        3.42        |       10      |
| OK     | NUCLEO_F334R8 | uARM      | EXAMPLE_1 | /dev/null                       |        3.42        |       10      |
| OK     | NUCLEO_F334R8 | uARM      | MBED_24   | Timeout Int us                  |       11.47        |       15      |
| OK     | NUCLEO_F334R8 | uARM      | MBED_25   | Time us                         |       11.43        |       15      |
| OK     | NUCLEO_F334R8 | uARM      | MBED_26   | Integer constant division       |        3.37        |       10      |
| OK     | NUCLEO_F334R8 | uARM      | MBED_23   | Ticker Int us                   |       12.43        |       15      |
| OK     | NUCLEO_F334R8 | uARM      | MBED_A19  | I2C EEPROM read/write test      |       11.42        |       15      |
| OK     | NUCLEO_F334R8 | uARM      | MBED_11   | Ticker Int                      |       12.43        |       20      |
| OK     | NUCLEO_F334R8 | uARM      | MBED_10   | Hello World                     |        2.42        |       10      |
| OK     | NUCLEO_F334R8 | uARM      | MBED_12   | C++                             |        3.42        |       10      |
| OK     | NUCLEO_F334R8 | uARM      | MBED_16   | RTC                             |        4.76        |       15      |
| UNDEF  | NUCLEO_F334R8 | uARM      | MBED_2    | stdio                           |       20.42        |       20      |
| UNDEF  | NUCLEO_F334R8 | uARM      | MBED_A9   | Serial Echo at 115200           |       10.37        |       10      |
+--------+---------------+-----------+-----------+---------------------------------+--------------------+---------------+
Result: 2 UNDEF / 12 OK

Completed in 160 sec

If we want to get additional test summary with results in separate columns please use option -t.

singletest.py -i test_spec.json -M muts_all.json -t

Test summary:
+---------------+-----------+---------------------------------+-------+
| Target        | Test ID   | Test Description                | uARM  |
+---------------+-----------+---------------------------------+-------+
| NUCLEO_F334R8 | EXAMPLE_1 | /dev/null                       | OK    |
| NUCLEO_F334R8 | MBED_10   | Hello World                     | OK    |
| NUCLEO_F334R8 | MBED_11   | Ticker Int                      | OK    |
| NUCLEO_F334R8 | MBED_12   | C++                             | OK    |
| NUCLEO_F334R8 | MBED_16   | RTC                             | OK    |
| NUCLEO_F334R8 | MBED_2    | stdio                           | UNDEF |
| NUCLEO_F334R8 | MBED_23   | Ticker Int us                   | OK    |
| NUCLEO_F334R8 | MBED_24   | Timeout Int us                  | OK    |
| NUCLEO_F334R8 | MBED_25   | Time us                         | OK    |
| NUCLEO_F334R8 | MBED_26   | Integer constant division       | OK    |
| NUCLEO_F334R8 | MBED_A1   | Basic                           | OK    |
| NUCLEO_F334R8 | MBED_A19  | I2C EEPROM read/write test      | OK    |
| NUCLEO_F334R8 | MBED_A25  | I2C EEPROM line read/write test | OK    |
| NUCLEO_F334R8 | MBED_A9   | Serial Echo at 115200           | UNDEF |
+---------------+-----------+---------------------------------+-------+

To only compile all required libraries and tests please use option -O (capital 'o').

singletest.py -i test_spec.json -M muts_all.json -O

workspace_tools> singletest.py -i test_spec.json -M muts_all.json -O

>>> BUILD LIBRARY CMSIS (NUCLEO_F334R8, uARM)

>>> BUILD LIBRARY MBED (NUCLEO_F334R8, uARM)

>>> BUILD PROJECT: I2C_EEPROM_LINE (NUCLEO_F334R8, uARM)

>>> BUILD PROJECT: BASIC (NUCLEO_F334R8, uARM)

>>> BUILD PROJECT: DEV_NULL (NUCLEO_F334R8, uARM)

>>> BUILD PROJECT: TIMEOUT (NUCLEO_F334R8, uARM)

>>> BUILD PROJECT: TIME_US (NUCLEO_F334R8, uARM)

>>> BUILD PROJECT: DIV (NUCLEO_F334R8, uARM)

>>> BUILD PROJECT: TICKER_2 (NUCLEO_F334R8, uARM)

>>> BUILD PROJECT: I2C_EEPROM (NUCLEO_F334R8, uARM)

>>> BUILD PROJECT: TICKER (NUCLEO_F334R8, uARM)

>>> BUILD PROJECT: HELLO (NUCLEO_F334R8, uARM)

>>> BUILD PROJECT: CPP (NUCLEO_F334R8, uARM)

>>> BUILD PROJECT: RTC (NUCLEO_F334R8, uARM)

>>> BUILD PROJECT: STDIO (NUCLEO_F334R8, uARM)

>>> BUILD PROJECT: ECHO (NUCLEO_F334R8, uARM)
Completed in 1 sec

To add extra verbose prints please use option -v.

singletest.py -i test_spec.json -M muts_all.json -v

~\workspace_tools> singletest.py -i test_spec.json -M muts_all.json -v
{   u'targets': {   u'NUCLEO_F334R8': [u'uARM']}}
{   u'1': {   u'disk': u'E:\\',
              u'mcu': u'NUCLEO_F334R8',
              u'peripherals': [u'24LC256'],
              u'port': u'COM46'}}

>>> BUILD LIBRARY CMSIS (NUCLEO_F334R8, uARM)

>>> BUILD LIBRARY MBED (NUCLEO_F334R8, uARM)
TargetTest::NUCLEO_F334R8::TestSkipped(ethernet)
TargetTest::NUCLEO_F334R8::TestSkipped(ethernet)
TargetTest::NUCLEO_F334R8::TestSkipped(ethernet)
TargetTest::NUCLEO_F334R8::TestSkipped(ethernet)
TargetTest::NUCLEO_F334R8::TestSkipped(ethernet)
TargetTest::NUCLEO_F334R8::TestSkipped(ethernet)
TargetTest::NUCLEO_F334R8::TestSkipped(ethernet)
TargetTest::NUCLEO_F334R8::TestSkipped(ethernet)
TargetTest::NUCLEO_F334R8::TestSkipped(ethernet)

>>> BUILD PROJECT: I2C_EEPROM_LINE (NUCLEO_F334R8, uARM)
Test::Output::Start
Mbed: "COM46" "E:\"
I2C: I2C Frequency: 400000 Hz
I2C: Lines pattern write test ... [OK]
I2C: Write errors: 0 ... [OK]
I2C: Lines pattern read test ... [OK]
I2C: Read errors: 0/1000 ... [OK]
EEPROM: Pattern match errors: 0/1000 ... [OK]
{{success}}
{{end}}
Test::Output::Finish
TargetTest::NUCLEO_F334R8::uARM::MBED_A25::I2C EEPROM line read/write test [OK] in 12.40 of 15 sec

>>> BUILD PROJECT: BASIC (NUCLEO_F334R8, uARM)
Test::Output::Start
Mbed: "COM46" "E:\"
{{success}}
{{end}}
Test::Output::Finish
TargetTest::NUCLEO_F334R8::uARM::MBED_A1::Basic [OK] in 3.42 of 10 sec

>>> BUILD PROJECT: DEV_NULL (NUCLEO_F334R8, uARM)
Test::Output::Start
Mbed: "COM46" "E:\"
Received 33 bytes

{success}
{end}
Test::Output::Finish
TargetTest::NUCLEO_F334R8::uARM::EXAMPLE_1::/dev/null [OK] in 3.42 of 10 sec
TargetTest::NUCLEO_F334R8::TestSkipped(digital_loop)
TargetTest::NUCLEO_F334R8::TestSkipped(TMP102)
TargetTest::NUCLEO_F334R8::TestSkipped(digital_loop)

>>> BUILD PROJECT: TIMEOUT (NUCLEO_F334R8, uARM)
Test::Output::Start
Mbed: "COM46" "E:\"
Test started
. skipped
. skipped
. skipped
. in 1.01 sec (0.01) [OK]
. in 1.00 sec (0.00) [OK]
. in 1.01 sec (0.01) [OK]
. in 1.01 sec (0.01) [OK]
. in 1.01 sec (0.01) [OK]
. in 1.01 sec (0.01) [OK]
. in 1.01 sec (0.01) [OK]
Completed in 9.08 sec

{success}
{end}
Test::Output::Finish
TargetTest::NUCLEO_F334R8::uARM::MBED_24::Timeout Int us [OK] in 11.54 of 15 sec

>>> BUILD PROJECT: TIME_US (NUCLEO_F334R8, uARM)
Test::Output::Start
Mbed: "COM46" "E:\"
Test started
. skipped
. skipped
. skipped
. in 1.00 sec (0.00) [OK]
. in 1.00 sec (0.00) [OK]
. in 1.00 sec (0.00) [OK]
. in 1.00 sec (0.00) [OK]
. in 1.00 sec (0.00) [OK]
. in 1.00 sec (0.00) [OK]
. in 1.00 sec (0.00) [OK]
Completed in 9.01 sec

{success}
{end}
Test::Output::Finish
TargetTest::NUCLEO_F334R8::uARM::MBED_25::Time us [OK] in 11.48 of 15 sec

>>> BUILD PROJECT: DIV (NUCLEO_F334R8, uARM)
Test::Output::Start
Mbed: "COM46" "E:\"
64bit: 0x7FFFFFFF8: expected 0x55555555 got 0x55555555 ... [OK]
64bit: 0x17FFFFFFE8: expected 0xFFFFFFFF got 0xFFFFFFFF ... [OK]
{{success}}
{{end}}
Test::Output::Finish
TargetTest::NUCLEO_F334R8::uARM::MBED_26::Integer constant division [OK] in 3.43 of 10 sec

>>> BUILD PROJECT: TICKER_2 (NUCLEO_F334R8, uARM)
Test::Output::Start
Mbed: "COM46" "E:\"
Test started
. skipped
. skipped
. skipped
. in 1.00 sec (0.00) [OK]
. in 1.00 sec (0.00) [OK]
. in 1.00 sec (0.00) [OK]
. in 1.00 sec (0.00) [OK]
. in 1.00 sec (0.00) [OK]
. in 1.00 sec (0.00) [OK]
. in 1.00 sec (0.00) [OK]
Completed in 9.00 sec

{success}
{end}
Test::Output::Finish
TargetTest::NUCLEO_F334R8::uARM::MBED_23::Ticker Int us [OK] in 11.45 of 15 sec

>>> BUILD PROJECT: I2C_EEPROM (NUCLEO_F334R8, uARM)
Test::Output::Start
Mbed: "COM46" "E:\"
I2C: I2C Frequency: 400000 Hz
I2C: Write 0x66 at address 0x0000 test ...
Unable to write data to EEPROM (i2c_stat = 0x01), aborting
{{failure}}
{{end}}
Test::Output::Finish
TargetTest::NUCLEO_F334R8::uARM::MBED_A19::I2C EEPROM read/write test [FAIL] in 3.43 of 15 sec
TargetTest::NUCLEO_F334R8::TestSkipped(port_loop)
TargetTest::NUCLEO_F334R8::TestSkipped(port_loop)
TargetTest::NUCLEO_F334R8::TestSkipped(SD)

>>> BUILD PROJECT: TICKER (NUCLEO_F334R8, uARM)
Test::Output::Start
Mbed: "COM46" "E:\"
Test started
. skipped
. skipped
. skipped
. in 1.00 sec (0.00) [OK]
. in 1.00 sec (0.00) [OK]
. in 1.00 sec (0.00) [OK]
. in 1.00 sec (0.00) [OK]
. in 1.00 sec (0.00) [OK]
. in 1.00 sec (0.00) [OK]
. in 1.00 sec (0.00) [OK]
Completed in 9.00 sec

{success}
{end}
Test::Output::Finish
TargetTest::NUCLEO_F334R8::uARM::MBED_11::Ticker Int [OK] in 11.43 of 20 sec

>>> BUILD PROJECT: HELLO (NUCLEO_F334R8, uARM)
Test::Output::Start
Mbed: "COM46" "E:\"
Hello World

{success}
{end}
Test::Output::Finish
TargetTest::NUCLEO_F334R8::uARM::MBED_10::Hello World [OK] in 2.43 of 10 sec

>>> BUILD PROJECT: CPP (NUCLEO_F334R8, uARM)
Test::Output::Start
Mbed: "COM46" "E:\"
Static::init
Static::stack_test
Stack::init
Stack::hello
Stack::destroy
Static::check_init: OK
Heap::init
Heap::hello
Heap::check_init: OK
Heap::destroy
{{success}}
{{end}}
Test::Output::Finish
TargetTest::NUCLEO_F334R8::uARM::MBED_12::C++ [OK] in 3.43 of 10 sec

>>> BUILD PROJECT: RTC (NUCLEO_F334R8, uARM)
Test::Output::Start
Mbed: "COM46" "E:\"
Got RTC time: [1256729737] [2009-10-28 11:35:37 AM] ... OK
Got RTC time: [1256729738] [2009-10-28 11:35:38 AM] ... OK
Got RTC time: [1256729739] [2009-10-28 11:35:39 AM] ... OK
Got RTC time: [1256729740] [2009-10-28 11:35:40 AM] ... OK
Got RTC time: [1256729741] [2009-10-28 11:35:41 AM] ... OK

{success}
{end}
Test::Output::Finish
TargetTest::NUCLEO_F334R8::uARM::MBED_16::RTC [OK] in 4.72 of 15 sec

>>> BUILD PROJECT: STDIO (NUCLEO_F334R8, uARM)
Test::Output::Start
Mbed: "COM46" "E:\"
Generated number: 3112

Test::Output::Finish
TargetTest::NUCLEO_F334R8::uARM::MBED_2::stdio [UNDEF] in 20.38 of 20 sec
TargetTest::NUCLEO_F334R8::TestSkipped(digital_loop)

>>> BUILD PROJECT: ECHO (NUCLEO_F334R8, uARM)
Test::Output::Start
Mbed: "COM46" "E:\"
Starting the ECHO test

Test::Output::Finish
TargetTest::NUCLEO_F334R8::uARM::MBED_A9::Serial Echo at 115200 [UNDEF] in 10.43 of 10 sec
Test summary:
+--------+---------------+-----------+-----------+---------------------------------+--------------------+---------------+
| Result | Target        | Toolchain | Test ID   | Test Description                | Elapsed Time (sec) | Timeout (sec) |
+--------+---------------+-----------+-----------+---------------------------------+--------------------+---------------+
| OK     | NUCLEO_F334R8 | uARM      | MBED_A25  | I2C EEPROM line read/write test |        12.4        |       15      |
| OK     | NUCLEO_F334R8 | uARM      | MBED_A1   | Basic                           |        3.42        |       10      |
| OK     | NUCLEO_F334R8 | uARM      | EXAMPLE_1 | /dev/null                       |        3.42        |       10      |
| OK     | NUCLEO_F334R8 | uARM      | MBED_24   | Timeout Int us                  |       11.54        |       15      |
| OK     | NUCLEO_F334R8 | uARM      | MBED_25   | Time us                         |       11.48        |       15      |
| OK     | NUCLEO_F334R8 | uARM      | MBED_26   | Integer constant division       |        3.43        |       10      |
| OK     | NUCLEO_F334R8 | uARM      | MBED_23   | Ticker Int us                   |       11.45        |       15      |
| FAIL   | NUCLEO_F334R8 | uARM      | MBED_A19  | I2C EEPROM read/write test      |        3.43        |       15      |
| OK     | NUCLEO_F334R8 | uARM      | MBED_11   | Ticker Int                      |       11.43        |       20      |
| OK     | NUCLEO_F334R8 | uARM      | MBED_10   | Hello World                     |        2.43        |       10      |
| OK     | NUCLEO_F334R8 | uARM      | MBED_12   | C++                             |        3.43        |       10      |
| OK     | NUCLEO_F334R8 | uARM      | MBED_16   | RTC                             |        4.72        |       15      |
| UNDEF  | NUCLEO_F334R8 | uARM      | MBED_2    | stdio                           |       20.38        |       20      |
| UNDEF  | NUCLEO_F334R8 | uARM      | MBED_A9   | Serial Echo at 115200           |       10.43        |       10      |
+--------+---------------+-----------+-----------+---------------------------------+--------------------+---------------+
Result: 1 FAIL / 2 UNDEF / 11 OK

Completed in 150 sec

What is host test?

Test suite support test supervisor concept. This concept is realized by separate Python script called "host test". Host tests can be found in workspace_tools/host_tests/ directory. Host test script is executed in parallel with test runner to monitor test execution. Basic host test just monitors device's default serial port for test results returned by test runner. Simple tests will print test result on serial port. In other cases host tests can for example judge by test results returned by test runner is test passed or failed. It all depends on test itself. In some cases host test can be TCP server echoing packets from test runner and judging packet loss. In other cases it can just check if values returned from accelerometer are actually valid (sane).

Peripherals testing

- only peripherals -P switch

MUTs specification

muts_all.json

Test specification

test_spec.JSON xxx

CI integration / verbose modes

xxx

Vocabulary

Test runner

A test runner is an executable program that runs tests implemented using test framework and reports the test results.

Appendix A

Example muts_all.json file

muts_all.json

{
    "1" : {"mcu" : "LPC1768",
           "port" : "COM4",
           "disk" : "J:\\",
           "peripherals" : ["TMP102", "digital_loop", "port_loop", "analog_loop", "SD"]},

    "2" : {"mcu" : "KL25Z",
           "port" : "COM39",
           "disk" : "E:\\",
           "peripherals" : ["SD", "24LC256", "KL25Z"]},

    "3" : {"mcu" : "LPC11U24",
           "port" : "COM5",
           "disk" : "E:\\",
           "peripherals" : ["TMP102", "digital_loop", "port_loop", "analog_loop"]},

    "4" : {"mcu" : "NUCLEO_F103RB",
           "port" : "COM11",
           "disk" : "I:\\",
           "peripherals" : ["24LC256"]},

    "5" : {"mcu" : "NRF51822",
           "port" : "COM21",
           "disk" : "I:\\",
           "peripherals" : []},

    "6" : {"mcu" : "UBLOX_C027",
           "port" : "COM14",
           "disk" : "F:\\",
           "peripherals" : []},

    "7" : {"mcu" : "LPC2368",
           "port" : "COM15",
           "disk" : "G:\\",
           "peripherals" : ["TMP102", "digital_loop", "port_loop", "analog_loop", "SD"]},

    "8" : {"mcu" : "LPC812",
           "port" : "COM6",
           "disk" : "E:\\",
           "peripherals" : ["24LC256"]},

    "9" : {"mcu" : "LPC1549",
           "port" : "COM32",
           "disk" : "I:\\",
           "peripherals" : ["24LC256"]},

    "10" : {"mcu" : "LPC4088",
            "port" : "COM17",
            "disk" : "E:\\",
            "peripherals" : []},

    "11" : {"mcu" : "KL46Z",
            "port" : "COM40",
            "disk" : "F:\\",
            "peripherals" : ["SD", "24LC256", "KL46Z"]},

    "12" : {"mcu" : "K64F",
            "port" : "COM18",
            "disk" : "E:\\",
            "peripherals" : ["ethernet"]},

    "13" : {"mcu" : "NUCLEO_L152RE",
            "port" : "COM9",
            "disk" : "E:\\",
            "peripherals" : []},

    "14" : {"mcu" : "NUCLEO_F302R8",
            "port" : "COM34",
            "disk" : "E:\\",
            "peripherals" : ["24LC256"]},

    "15" : {"mcu" : "NUCLEO_F401RE",
            "port" : "COM47",
            "disk" : "E:\\",
            "peripherals" : ["SD"]},

    "16" : {"mcu" : "NUCLEO_F030R8",
            "port" : "COM12",
            "disk" : "H:\\",
            "peripherals" : ["24LC256"]},

    "17" : {"mcu" : "NUCLEO_L053R8",
            "port" : "COM35",
            "disk" : "E:\\",
            "peripherals" : ["24LC256"]},

    "18" : {"mcu" : "LPC11U68",
            "port" : "COM36",
            "disk" : "E:\\",
            "peripherals" : ["24LC256"]},

    "19" : {"mcu" : "NUCLEO_F411RE",
            "port" : "COM44",
            "disk" : "F:\\",
            "peripherals" : ["24LC256"]},

    "20" : {"mcu" : "NUCLEO_F072RB",
            "port" : "COM45",
            "disk" : "F:\\",
            "peripherals" : ["24LC256"]},

    "21" : {"mcu" : "NUCLEO_F334R8",
            "port" : "COM46",
            "disk" : "E:\\",
            "peripherals" : ["24LC256"]},

    "22" : {"mcu" : "XADOW_M0",
            "port" : "COM51",
            "disk" : "E:\\",
            "peripherals" : []}
}

Example test_spec.json file

test_spec.json

{
    "targets": {
        "KL25Z": ["ARM", "GCC_ARM"],
        "LPC1768": ["ARM", "GCC_ARM", "GCC_CR", "GCC_CS", "IAR"],
        "LPC11U24": ["uARM"],
        "NRF51822": ["ARM"],
        "LPC2368": ["ARM"],
        "LPC4088": ["ARM"],
        "NUCLEO_F334R8" : ["uARM"]
    }
}

Please log in to post comments.

Mbed SDK - automated test suite

Abstract about how to build Mbed SDK tests on your local machine and how to take advantage from Mbed SDK test suite and how to use it in your local CI system.

Automation, CI, continuous, integration, test, tools, workspace, workspace_tools

Page owner: Przemek Wirkus

Created 10 Jun 2014.

Last updated 08 Jul 2014