Mbed SDK automated test suite
Mbed OS 2 and Mbed OS 5
This is a guide for Mbed OS 2. If you’re working with Mbed OS 5, please see the porting guide and Testing guide in the new handbook.
Mbed SDK automated test suite¶
Introduction¶
Test suit allows users to run locally on their machines Mbed SDK’s tests included in Mbed SDK repository. It also allows users to create their own tests and for example add new tests to test set as they progress with their project. If test is generic enough it could be included into official Mbed SDK test pool just do it via normal pull request!
Each test is supervised by python script called “host test” which will at least Test suite is using build script API to compile and build test source together with required by test libraries like CMSIS, Mbed, Ethernet, USB etc.
Prerequisites¶
- Installed Python 2.7.6 programming language: https://www.python.org/download/releases/2.7.
- Installed pyserial module for Python 2.7: https://pypi.python.org/pypi/pyserial
- Installed prettytable module for Python 2.7: https://code.google.com/p/prettytable/wiki/Installation
- You can check if you have correctly installed the above modules (or you already have them) by starting Python and importing both modules:
- Installed Git open source distributed version control system.
- Installed at least one of the supported by Mbed SDK workspace tools compilers:
| Compiler | Mbed SDK Abbreviation | Example Version |
|---|---|---|
| Keil ARM Compiler | ARM, uARM | ARM C/C++ Compiler, 5.03 [Build 117] |
| GCC ARM | GCC_ARM | gcc version 4.8.3 20131129 (release) [ARM/embedded-4_8-branch revision 205641] (GNU Tools for ARM Embedded Processors) |
| GCC CodeSourcery | GCC_CS | gcc version 4.8.1 (Sourcery CodeBench Lite 2013.11-24) |
| GCC CodeRed | GCC_CR | gcc version 4.6.2 20121016 (release) [ARM/embedded-4_6-branch revision 192487] (GNU Tools for ARM Embedded Processors) |
| IAR Embedded Workbench | IAR | IAR ANSI C/C++ Compiler V6.70.1.5641/W32 for ARM |
- Mbed board. You can find list of supported platforms here: https://mbed.org/platforms/
Test suite core: singletest.py script¶
singletest.py is a test suite script which allows users to compile, build tests and test runners (also supports CppUTest UT library). Script also is responsible for test execution on devices selected by configuration files.
Parameters of singletest.py¶
Test execution script singletest.py is a powerful tool to run tests for Mbed SDK platform. It is flexible and allows users to configure test execution process and define which mbed platforms will be tested.
By specifying external configuration files (in JSON format) you can gain flexibility and prepare many different test scenarios. Just pass configuration file names to your script and run it.
MUTs Specification¶
You can easily configure your MUTs (Mbed Under Test) by creating configuration file with MUTs' description.
Note: This configuration file must be in JSON format (http://www.w3schools.com/json/).
Note: Unfortunately JSON format is not allowing you to have comments inside JSON code.
Example 1 - three MUTs configuration (no peripherals)¶
Lets see some example and let's try to configure small "test farm" with three devices connected to your host computer. In this example no peripherals (like SD card or EEPROM) are connected to our Mbed boards. We will use three platforms in this example:
- NXP LPC1768 board https://mbed.org/platforms/mbed-LPC1768.
- Freescale KL25Z board https://mbed.org/platforms/KL25Z and
- STMicroelectronics NUCLEO_F103RB board https://mbed.org/platforms/ST-Nucleo-F103RB.
After connecting boards to our host machine (PC) we can check which serial ports and disks they occupy. For our example let's assume that:
- LPC1768 is on COM4 and disk drive is J:.
- KL25Z is on COM39 and disk drive is E:.
- NUCLEO_F103RB is on COM11 and disk drive is I:.
If you are working under Linux your port and disk could look like /dev/ttyACM5 and /media/usb5.
This information is needed to create muts_all.json configuration file. Its name will be passed to singletest.py script after -M (MUTs specification) switch. let's see how this file's content would look like in our example below:
muts_all.json
{
"1" : {"mcu": "LPC1768",
"port":"COM4",
"disk":"J:\\",
"peripherals": []
},
"2" : {"mcu": "KL25Z",
"port":"COM39",
"disk":"E:\\",
"peripherals": []
},
"3" : {"mcu": "NUCLEO_F103RB",
"port":"COM11",
"disk":"I:\\",
"peripherals": []
}
}
Note: We will leave field "peripherals" empty for the sake of this example. We will explain it later. All you need to do now is to properly fill fields "mcu", "port" and "disk".
Note: Please make sure files muts_all.json and test_spec.json are in workspace_tools/ directory. We will assume in this example they are.
Where to find "mcu" names? You can use option -S of build script build.py to check all supported off-line MCUs' names. Calling build.py script with -S option in workspace_tools/ directory should give you below list:
build.py -S
~\workspace_tools> build.py -S +---------------+-----------+-----------+-----------+-----------+-----------+-----------+------------+---------------+ | Platform | ARM | uARM | GCC_ARM | GCC_CS | GCC_CR | IAR | GCC_CW_EWL | GCC_CW_NEWLIB | +---------------+-----------+-----------+-----------+-----------+-----------+-----------+------------+---------------+ | ARCH_BLE | Default | - | Supported | - | - | - | - | - | | ARCH_PRO | Default | Supported | Supported | Supported | Supported | Supported | - | - | | ARM_MPS2 | Default | - | Supported | - | - | - | - | - | | DISCO_F051R8 | - | Default | Supported | - | - | - | - | - | | DISCO_F100RB | - | Default | Supported | - | - | - | - | - | | DISCO_F303VC | - | Default | Supported | - | - | - | - | - | | DISCO_F407VG | - | Default | Supported | - | - | - | - | - | | HRM1017 | Default | - | Supported | - | - | - | - | - | | K20D5M | Default | - | Supported | - | - | - | - | - | | K64F | Default | - | Supported | - | - | - | - | - | | KL05Z | Supported | Default | Supported | - | - | - | - | - | | KL25Z | Default | - | Supported | - | - | - | Supported | Supported | | KL46Z | Default | - | Supported | - | - | - | - | - | | LPC1114 | Supported | Default | Supported | - | Supported | - | - | - | | LPC11C24 | Default | Supported | Supported | - | - | - | - | - | | LPC11U24 | Supported | Default | Supported | - | - | - | - | - | | LPC11U24_301 | Default | Supported | Supported | - | - | - | - | - | | LPC11U35_401 | Supported | Default | Supported | - | Supported | - | - | - | | LPC11U35_501 | Supported | Default | Supported | - | Supported | - | - | - | | LPC11U68 | - | Default | Supported | - | Supported | - | - | - | | LPC1347 | Default | - | Supported | - | - | - | - | - | | LPC1549 | - | Default | - | - | Supported | - | - | - | | LPC1768 | Default | Supported | Supported | Supported | Supported | Supported | - | - | | LPC2368 | Default | - | Supported | - | Supported | - | - | - | | LPC4088 | Default | - | Supported | - | Supported | - | - | - | | LPC4330_M4 | Default | - | Supported | - | Supported | Supported | - | - | | LPC810 | - | Default | - | - | - | - | - | - | | LPC812 | - | Default | - | - | - | - | - | - | | LPCCAPPUCCINO | - | Default | Supported | - | Supported | - | - | - | | NRF51822 | Default | - | Supported | - | - | - | - | - | | NUCLEO_F030R8 | Supported | Default | - | - | - | - | - | - | | NUCLEO_F072RB | Supported | Default | - | - | - | - | - | - | | NUCLEO_F103RB | Supported | Default | Supported | - | - | - | - | - | | NUCLEO_F302R8 | Supported | Default | - | - | - | - | - | - | | NUCLEO_F334R8 | Supported | Default | - | - | - | - | - | - | | NUCLEO_F401RE | Supported | Default | Supported | - | - | - | - | - | | NUCLEO_F411RE | Supported | Default | - | - | - | - | - | - | | NUCLEO_L053R8 | Supported | Default | - | - | - | - | - | - | | NUCLEO_L152RE | Supported | Default | - | - | - | - | - | - | | STM32F3XX | Supported | Default | Supported | - | - | - | - | - | | STM32F407 | Default | - | Supported | - | - | - | - | - | | UBLOX_C027 | Default | Supported | Supported | Supported | Supported | Supported | - | - | | XADOW_M0 | Supported | Default | Supported | - | Supported | - | - | - | +---------------+-----------+-----------+-----------+-----------+-----------+-----------+------------+---------------+ *Default - default on-line compiler *Supported - supported off-line compiler Total permutations: 119
Note: If you update devices' firmware or even disconnect / reconnect Mbed device you may find that serial port / disk configuration changed. You need to update configuration file accordingly or you will face connection problems and obviously tests will not run.
Now we've already defined how our devices are connected to our host PC. We can continue and define which of this MUTs will be tested and which compilers we will use to compile and build Mbed SDK and tests. To do so we need to create test specification file (let's call it test_spec.json) and put inside our configuration file information about which MUTs we actually want to test. We will pass this file's name to singletest.py script using -i switch.
Below we can see how sample test_spec.json file content could look like. (I've included all possible toolchains, we will change it in a moment).
test_spec.json
{
"targets": {
"LPC1768" : ["ARM", "uARM", "GCC_ARM", "GCC_CS", "GCC_CR", "IAR"],
"KL25Z" : ["ARM", "GCC_ARM"],
"NUCLEO_F103RB" : ["ARM", "uARM"]
}
}
Above example configuration will force tests for LPC1768, KL25Z, NUCLEO_F103RB platforms and:
- Compilers: ARM, uARM, GCC_ARM, GCC_CS, GCC_CR and IAR will be used to compile tests for LPC1768.
- Compilers: ARM and GCC_ARM will be used for KL25Z platform.
- Compilers: ARM and uARM will be used for NUCLEO_F103RB platform.
For our example purposes let's assume we only have Keil ARM compiler, so let's change configuration in test_spec.json file and reduce number of compiler to those we actually have:
test_spec.json
{
"targets": {
"LPC1768" : ["ARM", "uARM"],
"KL25Z" : ["ARM"],
"NUCLEO_F103RB" : ["ARM", "uARM"]
}
}
Example 1 - Configuration summary¶
We've successfully connected all parts of our small "test farm" in two files test_spec.json and muts_all.json. Let's see how configuration logic looks like in Example 1:
Note: uARM is flavor of ARM compiler. In case of uARM we will link with microlib so we can save some space on your Mbed flash. There are other limitations concerning using of microlib (e.g. it does not support C++) but we will not discuss them here further.
Just a reminder. If you are unsure which Mbed platform supports which toolchains please go to workspace_tools/ directory in Mbed SDK repository and call build.py script with option -S:
c:\temp\repos\mbed\workspace_tools> python build.py -S +---------------+-----------+-----------+-----------+-----------+-----------+-----------+------------+---------------+ | Platform | ARM | uARM | GCC_ARM | GCC_CS | GCC_CR | IAR | GCC_CW_EWL | GCC_CW_NEWLIB | +---------------+-----------+-----------+-----------+-----------+-----------+-----------+------------+---------------+ ... | KL25Z | Supported | - | Supported | - | - | - | Supported | Supported | ... | LPC1768 | Supported | Supported | Supported | Supported | Supported | Supported | - | - | ... | NUCLEO_F103RB | Supported | Supported | Supported | - | - | - | - | - | ... +---------------+-----------+-----------+-----------+-----------+-----------+-----------+------------+---------------+ Total permutations: 101
Example 1 - Run your tests¶
After you configure all your MUTs and compilers you are ready to run tests. Make sure your devices are connected and your configuration files reflect your current configuration (serial ports, devices). Go to workspace_tools directory in your mbed location.
cd workspace_tools/
and execute test script.
singletest.py -i test_spec.json -M muts_all.json
Example 1 - summary¶
Having multiple configuration files allows you to manage your test scenarios in more flexible manner. You can:
- set up all platforms and toolchains used during testing,
- define (using script's -n switch) which tests you want to run during testing or
- just run regression (all tests). Regression is default setting for test script.
You can also force singletest.py script to:
- run only peripherals' tests (switch -P) or
- just skip peripherals' tests (switch -C).
Below you can see full singletes.py help command (-h switch) listing:
c:\temp\repos\mbed\workspace_tools> singletest.py -h
Usage: singletest.py [options]
This script allows you to run mbed defined test cases for particular MCU(s)
and corresponding toolchain(s).
Options:
-h, --help show this help message and exit
-i FILE, --tests=FILE
Points to file with test specification
-M FILE, --MUTS=FILE Points to file with MUTs specification (overwrites
settings.py and private_settings.py)
-j NUMBER, --jobs=NUMBER
Define number of compilation jobs. Default value is 1
-g, --goanna-for-tests
Run Goanna static analyse tool for tests. (Project
will be rebuilded)
-G, --goanna-for-sdk Run Goanna static analyse tool for mbed SDK (Project
will be rebuilded)
-s, --suppress-summary
Suppresses display of wellformatted table with test
results
-t, --test-summary Displays wellformatted table with test x toolchain
test result per target
-r, --test-automation-report
Prints information about all tests and exits
-R, --test-case-report
Prints information about all test cases and exits
-P, --only-peripherals
Test only peripheral declared for MUT and skip common
tests
-C, --only-commons Test only board internals. Skip perpherials tests and
perform common tests.
-c COPY_METHOD, --copy-method=COPY_METHOD
You can choose which copy method you want to use put
bin in mbed. You can choose from 'cp', 'copy',
'xcopy'. Default is python shutils.copy method.
-n TEST_BY_NAMES, --test-by-names=TEST_BY_NAMES
Runs only test enumerated it this switch
-S, --supported-toolchains
Displays supported matrix of MCUs and toolchains
-O, --only-build Only build tests, skips actual test procedures
(flashing etc.)
--config Displays full test specification and MUTs configration
and exits
--loops=TEST_LOOPS_LIST
Set no. of loops per test. Format:
TEST_1=1,TEST_2=2,TEST_3=3
--global-loops=TEST_GLOBAL_LOOPS_VALUE
Set global number of test loops per test. Default
value is set 1
--firmware-name=FIRMWARE_GLOBAL_NAME
Set global name for all produced projects. Note,
proper file extension will be added by buid scripts.
-u, --shuffle Shuffles test execution order
--shuffle-seed=SHUFFLE_TEST_SEED
Shuffle seed (If you want to reproduce your shuffle
order please use seed provided in test summary)
--reset-type=MUT_RESET_TYPE
Extra reset method used to reset MUT by host test
script
-f GENERAL_FILTER_REGEX, --filter=GENERAL_FILTER_REGEX
For some commands you can use filter to filter out
results
--inc-timeout=NUMBER You can increase global timeout for each test by
specifying additional test timeout in seconds
-l LOG_FILE_NAME, --log=LOG_FILE_NAME
Log events to external file (note not all console
entries may be visible in log file)
--verbose-skipped Prints some extra information about skipped tests
-V, --verbose-test-result
Prints test serial output
-v, --verbose Verbose mode (prints some extra information)
--version Prints script version and exits
Example: singletest.py -i test_spec.json -M muts_all.json
Example 2¶
This example will show you how to configure single device, run general tests or only peripheral tests. We will also show some real test result examples.
- We will test only one board ST Nucleo board F334R8 connected to our PC (port COM 46 and disk is E:).
- We will also connect EEPROM 24LC256 to SDA, SCL pins of our Nucleo board and define 24LC256 peripheral to make sure our test suite will run all available tests for 24LC256.
- Let's configure our one MUT and set uARM as the only compiler we will use to compiler Mbed SDK and tests.
- We also need to create two configuration files muts_all.json and test_spec.json to pass our small testbed configuration to test script.
muts_all.json
{
"1" : {
"mcu": "NUCLEO_F334R8",
"port":"COM46",
"disk":"E:\\",
"peripherals": ["24LC256"]
}
}
Note: By defining "peripherals": ["24LC256"] we are passing to test suite information that this particular board has EEPROM 24LC256 connected to our board.
test_spec.json
{
"targets": {
"NUCLEO_F334R8" : ["uARM"]
}
}
Above configuration now looks like this:
- Please make sure device is connected before we will start running tests.
- Please make sure files muts_all.json and test_spec.json are in workspace_tools/ directory. We will assume in this example they are.
To build Mbed SDK, compile, build and run tests please execute below command in workspace_tools/ directory.
singletest.py -i test_spec.json -M muts_all.json
~\workspace_tools> singletest.py -i test_spec.json -M muts_all.json >>> BUILD LIBRARY CMSIS (NUCLEO_F334R8, uARM) >>> BUILD LIBRARY MBED (NUCLEO_F334R8, uARM) >>> BUILD PROJECT: I2C_EEPROM_LINE (NUCLEO_F334R8, uARM) TargetTest::NUCLEO_F334R8::uARM::MBED_A25::I2C EEPROM line read/write test [OK] in 12.41 of 15 sec >>> BUILD PROJECT: BASIC (NUCLEO_F334R8, uARM) TargetTest::NUCLEO_F334R8::uARM::MBED_A1::Basic [OK] in 3.42 of 10 sec >>> BUILD PROJECT: DEV_NULL (NUCLEO_F334R8, uARM) TargetTest::NUCLEO_F334R8::uARM::EXAMPLE_1::/dev/null [OK] in 3.42 of 10 sec >>> BUILD PROJECT: TIMEOUT (NUCLEO_F334R8, uARM) TargetTest::NUCLEO_F334R8::uARM::MBED_24::Timeout Int us [OK] in 11.47 of 15 sec >>> BUILD PROJECT: TIME_US (NUCLEO_F334R8, uARM) TargetTest::NUCLEO_F334R8::uARM::MBED_25::Time us [OK] in 11.43 of 15 sec >>> BUILD PROJECT: DIV (NUCLEO_F334R8, uARM) TargetTest::NUCLEO_F334R8::uARM::MBED_26::Integer constant division [OK] in 3.37 of 10 sec >>> BUILD PROJECT: TICKER_2 (NUCLEO_F334R8, uARM) TargetTest::NUCLEO_F334R8::uARM::MBED_23::Ticker Int us [OK] in 12.43 of 15 sec >>> BUILD PROJECT: I2C_EEPROM (NUCLEO_F334R8, uARM) TargetTest::NUCLEO_F334R8::uARM::MBED_A19::I2C EEPROM read/write test [OK] in 11.42 of 15 sec >>> BUILD PROJECT: TICKER (NUCLEO_F334R8, uARM) TargetTest::NUCLEO_F334R8::uARM::MBED_11::Ticker Int [OK] in 12.43 of 20 sec >>> BUILD PROJECT: HELLO (NUCLEO_F334R8, uARM) TargetTest::NUCLEO_F334R8::uARM::MBED_10::Hello World [OK] in 2.42 of 10 sec >>> BUILD PROJECT: CPP (NUCLEO_F334R8, uARM) TargetTest::NUCLEO_F334R8::uARM::MBED_12::C++ [OK] in 3.42 of 10 sec >>> BUILD PROJECT: RTC (NUCLEO_F334R8, uARM) TargetTest::NUCLEO_F334R8::uARM::MBED_16::RTC [OK] in 4.76 of 15 sec >>> BUILD PROJECT: STDIO (NUCLEO_F334R8, uARM) TargetTest::NUCLEO_F334R8::uARM::MBED_2::stdio [UNDEF] in 20.42 of 20 sec >>> BUILD PROJECT: ECHO (NUCLEO_F334R8, uARM) TargetTest::NUCLEO_F334R8::uARM::MBED_A9::Serial Echo at 115200 [UNDEF] in 10.37 of 10 sec Test summary: +--------+---------------+-----------+-----------+---------------------------------+--------------------+---------------+ | Result | Target | Toolchain | Test ID | Test Description | Elapsed Time (sec) | Timeout (sec) | +--------+---------------+-----------+-----------+---------------------------------+--------------------+---------------+ | OK | NUCLEO_F334R8 | uARM | MBED_A25 | I2C EEPROM line read/write test | 12.41 | 15 | | OK | NUCLEO_F334R8 | uARM | MBED_A1 | Basic | 3.42 | 10 | | OK | NUCLEO_F334R8 | uARM | EXAMPLE_1 | /dev/null | 3.42 | 10 | | OK | NUCLEO_F334R8 | uARM | MBED_24 | Timeout Int us | 11.47 | 15 | | OK | NUCLEO_F334R8 | uARM | MBED_25 | Time us | 11.43 | 15 | | OK | NUCLEO_F334R8 | uARM | MBED_26 | Integer constant division | 3.37 | 10 | | OK | NUCLEO_F334R8 | uARM | MBED_23 | Ticker Int us | 12.43 | 15 | | OK | NUCLEO_F334R8 | uARM | MBED_A19 | I2C EEPROM read/write test | 11.42 | 15 | | OK | NUCLEO_F334R8 | uARM | MBED_11 | Ticker Int | 12.43 | 20 | | OK | NUCLEO_F334R8 | uARM | MBED_10 | Hello World | 2.42 | 10 | | OK | NUCLEO_F334R8 | uARM | MBED_12 | C++ | 3.42 | 10 | | OK | NUCLEO_F334R8 | uARM | MBED_16 | RTC | 4.76 | 15 | | UNDEF | NUCLEO_F334R8 | uARM | MBED_2 | stdio | 20.42 | 20 | | UNDEF | NUCLEO_F334R8 | uARM | MBED_A9 | Serial Echo at 115200 | 10.37 | 10 | +--------+---------------+-----------+-----------+---------------------------------+--------------------+---------------+ Result: 2 UNDEF / 12 OK Completed in 160 sec
If we want to get additional test summary with results in separate columns please use option -t.
singletest.py -i test_spec.json -M muts_all.json -t
Test summary: +---------------+-----------+---------------------------------+-------+ | Target | Test ID | Test Description | uARM | +---------------+-----------+---------------------------------+-------+ | NUCLEO_F334R8 | EXAMPLE_1 | /dev/null | OK | | NUCLEO_F334R8 | MBED_10 | Hello World | OK | | NUCLEO_F334R8 | MBED_11 | Ticker Int | OK | | NUCLEO_F334R8 | MBED_12 | C++ | OK | | NUCLEO_F334R8 | MBED_16 | RTC | OK | | NUCLEO_F334R8 | MBED_2 | stdio | UNDEF | | NUCLEO_F334R8 | MBED_23 | Ticker Int us | OK | | NUCLEO_F334R8 | MBED_24 | Timeout Int us | OK | | NUCLEO_F334R8 | MBED_25 | Time us | OK | | NUCLEO_F334R8 | MBED_26 | Integer constant division | OK | | NUCLEO_F334R8 | MBED_A1 | Basic | OK | | NUCLEO_F334R8 | MBED_A19 | I2C EEPROM read/write test | OK | | NUCLEO_F334R8 | MBED_A25 | I2C EEPROM line read/write test | OK | | NUCLEO_F334R8 | MBED_A9 | Serial Echo at 115200 | UNDEF | +---------------+-----------+---------------------------------+-------+
To only compile, build and link all required libraries and tests' dependencies please use option -O (capital 'o').
singletest.py -i test_spec.json -M muts_all.json -O
workspace_tools> singletest.py -i test_spec.json -M muts_all.json -O >>> BUILD LIBRARY CMSIS (NUCLEO_F334R8, uARM) >>> BUILD LIBRARY MBED (NUCLEO_F334R8, uARM) >>> BUILD PROJECT: I2C_EEPROM_LINE (NUCLEO_F334R8, uARM) >>> BUILD PROJECT: BASIC (NUCLEO_F334R8, uARM) >>> BUILD PROJECT: DEV_NULL (NUCLEO_F334R8, uARM) >>> BUILD PROJECT: TIMEOUT (NUCLEO_F334R8, uARM) >>> BUILD PROJECT: TIME_US (NUCLEO_F334R8, uARM) >>> BUILD PROJECT: DIV (NUCLEO_F334R8, uARM) >>> BUILD PROJECT: TICKER_2 (NUCLEO_F334R8, uARM) >>> BUILD PROJECT: I2C_EEPROM (NUCLEO_F334R8, uARM) >>> BUILD PROJECT: TICKER (NUCLEO_F334R8, uARM) >>> BUILD PROJECT: HELLO (NUCLEO_F334R8, uARM) >>> BUILD PROJECT: CPP (NUCLEO_F334R8, uARM) >>> BUILD PROJECT: RTC (NUCLEO_F334R8, uARM) >>> BUILD PROJECT: STDIO (NUCLEO_F334R8, uARM) >>> BUILD PROJECT: ECHO (NUCLEO_F334R8, uARM) Completed in 1 sec
To add extra verbose prints please use option -v.
singletest.py -i test_spec.json -M muts_all.json -v
~\workspace_tools> singletest.py -i test_spec.json -M muts_all.json -v
{ u'targets': { u'NUCLEO_F334R8': [u'uARM']}}
{ u'1': { u'disk': u'E:\\',
u'mcu': u'NUCLEO_F334R8',
u'peripherals': [u'24LC256'],
u'port': u'COM46'}}
>>> BUILD LIBRARY CMSIS (NUCLEO_F334R8, uARM)
>>> BUILD LIBRARY MBED (NUCLEO_F334R8, uARM)
TargetTest::NUCLEO_F334R8::TestSkipped(ethernet)
TargetTest::NUCLEO_F334R8::TestSkipped(ethernet)
TargetTest::NUCLEO_F334R8::TestSkipped(ethernet)
TargetTest::NUCLEO_F334R8::TestSkipped(ethernet)
TargetTest::NUCLEO_F334R8::TestSkipped(ethernet)
TargetTest::NUCLEO_F334R8::TestSkipped(ethernet)
TargetTest::NUCLEO_F334R8::TestSkipped(ethernet)
TargetTest::NUCLEO_F334R8::TestSkipped(ethernet)
TargetTest::NUCLEO_F334R8::TestSkipped(ethernet)
>>> BUILD PROJECT: I2C_EEPROM_LINE (NUCLEO_F334R8, uARM)
Test::Output::Start
Mbed: "COM46" "E:\"
I2C: I2C Frequency: 400000 Hz
I2C: Lines pattern write test ... [OK]
I2C: Write errors: 0 ... [OK]
I2C: Lines pattern read test ... [OK]
I2C: Read errors: 0/1000 ... [OK]
EEPROM: Pattern match errors: 0/1000 ... [OK]
{{success}}
{{end}}
Test::Output::Finish
TargetTest::NUCLEO_F334R8::uARM::MBED_A25::I2C EEPROM line read/write test [OK] in 12.40 of 15 sec
>>> BUILD PROJECT: BASIC (NUCLEO_F334R8, uARM)
Test::Output::Start
Mbed: "COM46" "E:\"
{{success}}
{{end}}
Test::Output::Finish
TargetTest::NUCLEO_F334R8::uARM::MBED_A1::Basic [OK] in 3.42 of 10 sec
>>> BUILD PROJECT: DEV_NULL (NUCLEO_F334R8, uARM)
Test::Output::Start
Mbed: "COM46" "E:\"
Received 33 bytes
{success}
{end}
Test::Output::Finish
TargetTest::NUCLEO_F334R8::uARM::EXAMPLE_1::/dev/null [OK] in 3.42 of 10 sec
TargetTest::NUCLEO_F334R8::TestSkipped(digital_loop)
TargetTest::NUCLEO_F334R8::TestSkipped(TMP102)
TargetTest::NUCLEO_F334R8::TestSkipped(digital_loop)
>>> BUILD PROJECT: TIMEOUT (NUCLEO_F334R8, uARM)
Test::Output::Start
Mbed: "COM46" "E:\"
Test started
. skipped
. skipped
. skipped
. in 1.01 sec (0.01) [OK]
. in 1.00 sec (0.00) [OK]
. in 1.01 sec (0.01) [OK]
. in 1.01 sec (0.01) [OK]
. in 1.01 sec (0.01) [OK]
. in 1.01 sec (0.01) [OK]
. in 1.01 sec (0.01) [OK]
Completed in 9.08 sec
{success}
{end}
Test::Output::Finish
TargetTest::NUCLEO_F334R8::uARM::MBED_24::Timeout Int us [OK] in 11.54 of 15 sec
>>> BUILD PROJECT: TIME_US (NUCLEO_F334R8, uARM)
Test::Output::Start
Mbed: "COM46" "E:\"
Test started
. skipped
. skipped
. skipped
. in 1.00 sec (0.00) [OK]
. in 1.00 sec (0.00) [OK]
. in 1.00 sec (0.00) [OK]
. in 1.00 sec (0.00) [OK]
. in 1.00 sec (0.00) [OK]
. in 1.00 sec (0.00) [OK]
. in 1.00 sec (0.00) [OK]
Completed in 9.01 sec
{success}
{end}
Test::Output::Finish
TargetTest::NUCLEO_F334R8::uARM::MBED_25::Time us [OK] in 11.48 of 15 sec
>>> BUILD PROJECT: DIV (NUCLEO_F334R8, uARM)
Test::Output::Start
Mbed: "COM46" "E:\"
64bit: 0x7FFFFFFF8: expected 0x55555555 got 0x55555555 ... [OK]
64bit: 0x17FFFFFFE8: expected 0xFFFFFFFF got 0xFFFFFFFF ... [OK]
{{success}}
{{end}}
Test::Output::Finish
TargetTest::NUCLEO_F334R8::uARM::MBED_26::Integer constant division [OK] in 3.43 of 10 sec
>>> BUILD PROJECT: TICKER_2 (NUCLEO_F334R8, uARM)
Test::Output::Start
Mbed: "COM46" "E:\"
Test started
. skipped
. skipped
. skipped
. in 1.00 sec (0.00) [OK]
. in 1.00 sec (0.00) [OK]
. in 1.00 sec (0.00) [OK]
. in 1.00 sec (0.00) [OK]
. in 1.00 sec (0.00) [OK]
. in 1.00 sec (0.00) [OK]
. in 1.00 sec (0.00) [OK]
Completed in 9.00 sec
{success}
{end}
Test::Output::Finish
TargetTest::NUCLEO_F334R8::uARM::MBED_23::Ticker Int us [OK] in 11.45 of 15 sec
>>> BUILD PROJECT: I2C_EEPROM (NUCLEO_F334R8, uARM)
Test::Output::Start
Mbed: "COM46" "E:\"
I2C: I2C Frequency: 400000 Hz
I2C: Write 0x66 at address 0x0000 test ...
Unable to write data to EEPROM (i2c_stat = 0x01), aborting
{{failure}}
{{end}}
Test::Output::Finish
TargetTest::NUCLEO_F334R8::uARM::MBED_A19::I2C EEPROM read/write test [FAIL] in 3.43 of 15 sec
TargetTest::NUCLEO_F334R8::TestSkipped(port_loop)
TargetTest::NUCLEO_F334R8::TestSkipped(port_loop)
TargetTest::NUCLEO_F334R8::TestSkipped(SD)
>>> BUILD PROJECT: TICKER (NUCLEO_F334R8, uARM)
Test::Output::Start
Mbed: "COM46" "E:\"
Test started
. skipped
. skipped
. skipped
. in 1.00 sec (0.00) [OK]
. in 1.00 sec (0.00) [OK]
. in 1.00 sec (0.00) [OK]
. in 1.00 sec (0.00) [OK]
. in 1.00 sec (0.00) [OK]
. in 1.00 sec (0.00) [OK]
. in 1.00 sec (0.00) [OK]
Completed in 9.00 sec
{success}
{end}
Test::Output::Finish
TargetTest::NUCLEO_F334R8::uARM::MBED_11::Ticker Int [OK] in 11.43 of 20 sec
>>> BUILD PROJECT: HELLO (NUCLEO_F334R8, uARM)
Test::Output::Start
Mbed: "COM46" "E:\"
Hello World
{success}
{end}
Test::Output::Finish
TargetTest::NUCLEO_F334R8::uARM::MBED_10::Hello World [OK] in 2.43 of 10 sec
>>> BUILD PROJECT: CPP (NUCLEO_F334R8, uARM)
Test::Output::Start
Mbed: "COM46" "E:\"
Static::init
Static::stack_test
Stack::init
Stack::hello
Stack::destroy
Static::check_init: OK
Heap::init
Heap::hello
Heap::check_init: OK
Heap::destroy
{{success}}
{{end}}
Test::Output::Finish
TargetTest::NUCLEO_F334R8::uARM::MBED_12::C++ [OK] in 3.43 of 10 sec
>>> BUILD PROJECT: RTC (NUCLEO_F334R8, uARM)
Test::Output::Start
Mbed: "COM46" "E:\"
Got RTC time: [1256729737] [2009-10-28 11:35:37 AM] ... OK
Got RTC time: [1256729738] [2009-10-28 11:35:38 AM] ... OK
Got RTC time: [1256729739] [2009-10-28 11:35:39 AM] ... OK
Got RTC time: [1256729740] [2009-10-28 11:35:40 AM] ... OK
Got RTC time: [1256729741] [2009-10-28 11:35:41 AM] ... OK
{success}
{end}
Test::Output::Finish
TargetTest::NUCLEO_F334R8::uARM::MBED_16::RTC [OK] in 4.72 of 15 sec
>>> BUILD PROJECT: STDIO (NUCLEO_F334R8, uARM)
Test::Output::Start
Mbed: "COM46" "E:\"
Generated number: 3112
Test::Output::Finish
TargetTest::NUCLEO_F334R8::uARM::MBED_2::stdio [UNDEF] in 20.38 of 20 sec
TargetTest::NUCLEO_F334R8::TestSkipped(digital_loop)
>>> BUILD PROJECT: ECHO (NUCLEO_F334R8, uARM)
Test::Output::Start
Mbed: "COM46" "E:\"
Starting the ECHO test
Test::Output::Finish
TargetTest::NUCLEO_F334R8::uARM::MBED_A9::Serial Echo at 115200 [UNDEF] in 10.43 of 10 sec
Test summary:
+--------+---------------+-----------+-----------+---------------------------------+--------------------+---------------+
| Result | Target | Toolchain | Test ID | Test Description | Elapsed Time (sec) | Timeout (sec) |
+--------+---------------+-----------+-----------+---------------------------------+--------------------+---------------+
| OK | NUCLEO_F334R8 | uARM | MBED_A25 | I2C EEPROM line read/write test | 12.4 | 15 |
| OK | NUCLEO_F334R8 | uARM | MBED_A1 | Basic | 3.42 | 10 |
| OK | NUCLEO_F334R8 | uARM | EXAMPLE_1 | /dev/null | 3.42 | 10 |
| OK | NUCLEO_F334R8 | uARM | MBED_24 | Timeout Int us | 11.54 | 15 |
| OK | NUCLEO_F334R8 | uARM | MBED_25 | Time us | 11.48 | 15 |
| OK | NUCLEO_F334R8 | uARM | MBED_26 | Integer constant division | 3.43 | 10 |
| OK | NUCLEO_F334R8 | uARM | MBED_23 | Ticker Int us | 11.45 | 15 |
| FAIL | NUCLEO_F334R8 | uARM | MBED_A19 | I2C EEPROM read/write test | 3.43 | 15 |
| OK | NUCLEO_F334R8 | uARM | MBED_11 | Ticker Int | 11.43 | 20 |
| OK | NUCLEO_F334R8 | uARM | MBED_10 | Hello World | 2.43 | 10 |
| OK | NUCLEO_F334R8 | uARM | MBED_12 | C++ | 3.43 | 10 |
| OK | NUCLEO_F334R8 | uARM | MBED_16 | RTC | 4.72 | 15 |
| UNDEF | NUCLEO_F334R8 | uARM | MBED_2 | stdio | 20.38 | 20 |
| UNDEF | NUCLEO_F334R8 | uARM | MBED_A9 | Serial Echo at 115200 | 10.43 | 10 |
+--------+---------------+-----------+-----------+---------------------------------+--------------------+---------------+
Result: 1 FAIL / 2 UNDEF / 11 OK
Completed in 150 sec
Run only selected tests¶
You can select which tests should be executed when you run test suite. Use -n switch to define tests by their ids you want to execute. Use comma to separate test ids:
singletest.py -i test_spec.json -M muts_all.json -n RTOS_1,RTOS_2,RTOS_3,MBED_10,MBED_16,MBED_11
Below you can see test summary only for selected tests:
Test summary: +--------+---------+-----------+---------+-------------------------+--------------------+---------------+-------+ | Result | Target | Toolchain | Test ID | Test Description | Elapsed Time (sec) | Timeout (sec) | Loops | +--------+---------+-----------+---------+-------------------------+--------------------+---------------+-------+ | OK | LPC1768 | ARM | MBED_11 | Ticker Int | 12.51 | 20 | 1/1 | | OK | LPC1768 | ARM | MBED_10 | Hello World | 2.42 | 10 | 1/1 | | OK | LPC1768 | ARM | MBED_16 | RTC | 5.67 | 15 | 1/1 | | OK | LPC1768 | ARM | RTOS_3 | Semaphore resource lock | 11.5 | 20 | 1/1 | | OK | LPC1768 | ARM | RTOS_2 | Mutex resource lock | 12.51 | 20 | 1/1 | | OK | LPC1768 | ARM | RTOS_1 | Basic thread | 12.58 | 15 | 1/1 | +--------+---------+-----------+---------+-------------------------+--------------------+---------------+-------+ Result: 6 OK Completed in 75.23 sec
Shuffle your tests¶
We strongly encourage you to shuffle your test order each time you execute test suite script. Rationale: It is probable that tests executed in one particular order will pass and in other will fail.
To shuffle your tests’ order please use option –u (or --shuffle):
singletest.py -i test_spec.json -M muts_all.json --shuffle
Above command with force test script to randomly generate shuffle seed and shuffle test order execution. Note: Shuffle seed is float in [0.0, 1.0).
You can always recreate particular test order by forcing shuffle (-u or } switch) and passing shuffle seed back to test suite using
--shuffle-seed switch:
singletest.py -i test_spec.json -M muts_all.json --shuffle --shuffle-seed 0.4041028336
Note: You can also supply your own randomly generated shuffle seed to drive particular test execution order scenarios. Just make sure shuffle seed is float in [0.0, 1.0).
You can find test shuffle seed in test summary:
... | OK | LPC1768 | ARM | RTOS_6 | Mail messaging | 3.42 | 10 | 1/1 | | OK | LPC1768 | ARM | RTOS_4 | Signals messaging | 7.45 | 10 | 1/1 | | OK | LPC1768 | ARM | MBED_A9 | Serial Echo at 115200 | 2.84 | 10 | 1/1 | +--------+---------+-----------+-----------+-----------------------------+--------------------+---------------+-------+ Result: 1 FAIL / 22 OK Shuffle Seed: 0.4041028336 Completed in 234.85 sec
Note: you can force shuffle seed (--shuffle-seed) will only be available when you use shuffle switch -u. In other case test suite will execute tests in one fixed order.
Loop over your tests¶
You can loop (execute few times) tests while test script is running.
- Use switch
--global-loopsto define global loop count or
singletest.py -i test_spec.json -M muts_all.json --global-loops 5
Example output, each test is repeated 5 times:
Building library CMSIS (LPC1768, ARM) Building library MBED (LPC1768, ARM) Building project TICKER (LPC1768, ARM) TargetTest::LPC1768::ARM::MBED_11::Ticker Int [OK] in 12.48 of 20 sec TargetTest::LPC1768::ARM::MBED_11::Ticker Int [OK] in 12.51 of 20 sec TargetTest::LPC1768::ARM::MBED_11::Ticker Int [OK] in 12.53 of 20 sec TargetTest::LPC1768::ARM::MBED_11::Ticker Int [OK] in 12.47 of 20 sec TargetTest::LPC1768::ARM::MBED_11::Ticker Int [OK] in 12.47 of 20 sec Building project HELLO (LPC1768, ARM) TargetTest::LPC1768::ARM::MBED_10::Hello World [OK] in 2.42 of 10 sec TargetTest::LPC1768::ARM::MBED_10::Hello World [OK] in 2.37 of 10 sec TargetTest::LPC1768::ARM::MBED_10::Hello World [OK] in 2.37 of 10 sec TargetTest::LPC1768::ARM::MBED_10::Hello World [OK] in 2.37 of 10 sec TargetTest::LPC1768::ARM::MBED_10::Hello World [OK] in 2.37 of 10 sec Building project RTC (LPC1768, ARM) TargetTest::LPC1768::ARM::MBED_16::RTC [OK] in 5.59 of 15 sec TargetTest::LPC1768::ARM::MBED_16::RTC [OK] in 5.67 of 15 sec TargetTest::LPC1768::ARM::MBED_16::RTC [OK] in 5.66 of 15 sec TargetTest::LPC1768::ARM::MBED_16::RTC [OK] in 5.67 of 15 sec TargetTest::LPC1768::ARM::MBED_16::RTC [OK] in 5.67 of 15 sec Building library RTX (LPC1768, ARM) Building library RTOS (LPC1768, ARM) Building project SEMAPHORE (LPC1768, ARM) TargetTest::LPC1768::ARM::RTOS_3::Semaphore resource lock [OK] in 11.51 of 20 sec TargetTest::LPC1768::ARM::RTOS_3::Semaphore resource lock [OK] in 11.49 of 20 sec TargetTest::LPC1768::ARM::RTOS_3::Semaphore resource lock [OK] in 11.48 of 20 sec TargetTest::LPC1768::ARM::RTOS_3::Semaphore resource lock [OK] in 11.49 of 20 sec TargetTest::LPC1768::ARM::RTOS_3::Semaphore resource lock [OK] in 11.50 of 20 sec Building library RTX (LPC1768, ARM) Building library RTOS (LPC1768, ARM) Building project MUTEX (LPC1768, ARM) TargetTest::LPC1768::ARM::RTOS_2::Mutex resource lock [OK] in 12.50 of 20 sec TargetTest::LPC1768::ARM::RTOS_2::Mutex resource lock [OK] in 12.50 of 20 sec TargetTest::LPC1768::ARM::RTOS_2::Mutex resource lock [OK] in 12.50 of 20 sec TargetTest::LPC1768::ARM::RTOS_2::Mutex resource lock [OK] in 12.51 of 20 sec TargetTest::LPC1768::ARM::RTOS_2::Mutex resource lock [OK] in 12.50 of 20 sec Building library RTX (LPC1768, ARM) Building library RTOS (LPC1768, ARM) Building project BASIC (LPC1768, ARM) TargetTest::LPC1768::ARM::RTOS_1::Basic thread [OK] in 12.51 of 15 sec TargetTest::LPC1768::ARM::RTOS_1::Basic thread [OK] in 12.52 of 15 sec TargetTest::LPC1768::ARM::RTOS_1::Basic thread [OK] in 12.58 of 15 sec TargetTest::LPC1768::ARM::RTOS_1::Basic thread [OK] in 12.51 of 15 sec TargetTest::LPC1768::ARM::RTOS_1::Basic thread [OK] in 12.57 of 15 sec Test summary: +--------+---------+-----------+---------+-------------------------+--------------------+---------------+-------+ | Result | Target | Toolchain | Test ID | Test Description | Elapsed Time (sec) | Timeout (sec) | Loops | +--------+---------+-----------+---------+-------------------------+--------------------+---------------+-------+ | OK | LPC1768 | ARM | MBED_11 | Ticker Int | 12.47 | 20 | 5/5 | | OK | LPC1768 | ARM | MBED_10 | Hello World | 2.37 | 10 | 5/5 | | OK | LPC1768 | ARM | MBED_16 | RTC | 5.67 | 15 | 5/5 | | OK | LPC1768 | ARM | RTOS_3 | Semaphore resource lock | 11.5 | 20 | 5/5 | | OK | LPC1768 | ARM | RTOS_2 | Mutex resource lock | 12.5 | 20 | 5/5 | | OK | LPC1768 | ARM | RTOS_1 | Basic thread | 12.57 | 15 | 5/5 | +--------+---------+-----------+---------+-------------------------+--------------------+---------------+-------+ Result: 6 OK Completed in 374.42 sec
Note: Default global loop value is 1.
- switch
--loopsto overwrite global loop count and redefine loop count for particular test(s). Define test loops as TEST_ID=X where X is integer and separate loops count definitions by comma if necessary. E.g. TEST_1=5,TEST_2=20,TEST_3=2.
singletest.py -i test_spec.json -M muts_all.json RTOS_1=10,RTOS_2=5
Example output, where test RTOS_1 will be executed 10 times and test RTOS_2 will be executed 5 times:
Building library CMSIS (LPC1768, ARM) Building library MBED (LPC1768, ARM) Building project TICKER (LPC1768, ARM) TargetTest::LPC1768::ARM::MBED_11::Ticker Int [OK] in 12.52 of 20 sec Building project HELLO (LPC1768, ARM) TargetTest::LPC1768::ARM::MBED_10::Hello World [OK] in 2.42 of 10 sec Building project RTC (LPC1768, ARM) TargetTest::LPC1768::ARM::MBED_16::RTC [OK] in 5.63 of 15 sec Building library RTX (LPC1768, ARM) Building library RTOS (LPC1768, ARM) Building project SEMAPHORE (LPC1768, ARM) TargetTest::LPC1768::ARM::RTOS_3::Semaphore resource lock [OK] in 11.50 of 20 sec Building library RTX (LPC1768, ARM) Building library RTOS (LPC1768, ARM) Building project MUTEX (LPC1768, ARM) TargetTest::LPC1768::ARM::RTOS_2::Mutex resource lock [OK] in 12.52 of 20 sec TargetTest::LPC1768::ARM::RTOS_2::Mutex resource lock [OK] in 12.49 of 20 sec TargetTest::LPC1768::ARM::RTOS_2::Mutex resource lock [OK] in 12.50 of 20 sec TargetTest::LPC1768::ARM::RTOS_2::Mutex resource lock [OK] in 12.49 of 20 sec TargetTest::LPC1768::ARM::RTOS_2::Mutex resource lock [OK] in 12.50 of 20 sec Building library RTX (LPC1768, ARM) Building library RTOS (LPC1768, ARM) Building project BASIC (LPC1768, ARM) TargetTest::LPC1768::ARM::RTOS_1::Basic thread [OK] in 12.52 of 15 sec TargetTest::LPC1768::ARM::RTOS_1::Basic thread [OK] in 12.48 of 15 sec TargetTest::LPC1768::ARM::RTOS_1::Basic thread [OK] in 12.63 of 15 sec TargetTest::LPC1768::ARM::RTOS_1::Basic thread [OK] in 12.54 of 15 sec TargetTest::LPC1768::ARM::RTOS_1::Basic thread [OK] in 12.52 of 15 sec TargetTest::LPC1768::ARM::RTOS_1::Basic thread [OK] in 12.58 of 15 sec TargetTest::LPC1768::ARM::RTOS_1::Basic thread [OK] in 12.51 of 15 sec TargetTest::LPC1768::ARM::RTOS_1::Basic thread [OK] in 12.58 of 15 sec TargetTest::LPC1768::ARM::RTOS_1::Basic thread [OK] in 12.58 of 15 sec TargetTest::LPC1768::ARM::RTOS_1::Basic thread [OK] in 12.58 of 15 sec Test summary: +--------+---------+-----------+---------+-------------------------+--------------------+---------------+-------+ | Result | Target | Toolchain | Test ID | Test Description | Elapsed Time (sec) | Timeout (sec) | Loops | +--------+---------+-----------+---------+-------------------------+--------------------+---------------+-------+ | OK | LPC1768 | ARM | MBED_11 | Ticker Int | 12.52 | 20 | 1/1 | | OK | LPC1768 | ARM | MBED_10 | Hello World | 2.42 | 10 | 1/1 | | OK | LPC1768 | ARM | MBED_16 | RTC | 5.63 | 15 | 1/1 | | OK | LPC1768 | ARM | RTOS_3 | Semaphore resource lock | 11.5 | 20 | 1/1 | | OK | LPC1768 | ARM | RTOS_2 | Mutex resource lock | 12.5 | 20 | 5/5 | | OK | LPC1768 | ARM | RTOS_1 | Basic thread | 12.58 | 15 | 10/10 | +--------+---------+-----------+---------+-------------------------+--------------------+---------------+-------+ Result: 6 OK Completed in 278.60 sec
Example 3¶
Please do not forget you can combine few options together to get result you want. For example you want to repeat few tests multiple number of times, shuffle test ids execution order and select only tests which are critical for you at this point. You can do it using switch -n, --global-loops with --loops and --shuffle:
Execute above command to:
- run only tests: RTOS_1, RTOS_2, RTOS_3, MBED_10, MBED_16, MBED_11.
- shuffle test execution order. Note tests in loops will not be shuffled.
- set global loop count to 3 - each test will repeated 3 times.
- overwrite global loop count (set above to 3) and:
- force to loop test RTOS_1 to execute 3 times.
- force to loop test RTOS_2 to execute 4 times.
- force to loop test RTOS_3 to execute 5 times.
- force to loop test MBED_11 to execute 5 times.
Note: In test summary you can find shuffle seed value for this test script execution: 0.8837216077.
singletest.py -i test_spec.json -M muts_all.json -n RTOS_1,RTOS_2,RTOS_3,MBED_10,MBED_16,MBED_11 --shuffle --global-loops 3 --loops RTOS_1=3,RTOS_2=4,RTOS_3=5,MBED_11=5
Example output:
Building library CMSIS (LPC1768, ARM) Building library MBED (LPC1768, ARM) Building library RTX (LPC1768, ARM) Building library RTOS (LPC1768, ARM) Building project BASIC (LPC1768, ARM) TargetTest::LPC1768::ARM::RTOS_1::Basic thread [OK] in 12.51 of 15 sec TargetTest::LPC1768::ARM::RTOS_1::Basic thread [OK] in 12.52 of 15 sec TargetTest::LPC1768::ARM::RTOS_1::Basic thread [OK] in 12.57 of 15 sec Building library RTX (LPC1768, ARM) Building library RTOS (LPC1768, ARM) Building project SEMAPHORE (LPC1768, ARM) TargetTest::LPC1768::ARM::RTOS_3::Semaphore resource lock [OK] in 11.49 of 20 sec TargetTest::LPC1768::ARM::RTOS_3::Semaphore resource lock [OK] in 11.49 of 20 sec TargetTest::LPC1768::ARM::RTOS_3::Semaphore resource lock [OK] in 11.49 of 20 sec TargetTest::LPC1768::ARM::RTOS_3::Semaphore resource lock [OK] in 11.50 of 20 sec TargetTest::LPC1768::ARM::RTOS_3::Semaphore resource lock [OK] in 11.49 of 20 sec Building project TICKER (LPC1768, ARM) TargetTest::LPC1768::ARM::MBED_11::Ticker Int [OK] in 12.47 of 20 sec TargetTest::LPC1768::ARM::MBED_11::Ticker Int [OK] in 12.51 of 20 sec TargetTest::LPC1768::ARM::MBED_11::Ticker Int [OK] in 12.47 of 20 sec TargetTest::LPC1768::ARM::MBED_11::Ticker Int [OK] in 12.42 of 20 sec TargetTest::LPC1768::ARM::MBED_11::Ticker Int [OK] in 12.51 of 20 sec Building project HELLO (LPC1768, ARM) TargetTest::LPC1768::ARM::MBED_10::Hello World [OK] in 2.37 of 10 sec TargetTest::LPC1768::ARM::MBED_10::Hello World [OK] in 2.42 of 10 sec TargetTest::LPC1768::ARM::MBED_10::Hello World [OK] in 2.37 of 10 sec Building project RTC (LPC1768, ARM) TargetTest::LPC1768::ARM::MBED_16::RTC [OK] in 5.64 of 15 sec TargetTest::LPC1768::ARM::MBED_16::RTC [OK] in 5.66 of 15 sec TargetTest::LPC1768::ARM::MBED_16::RTC [OK] in 5.67 of 15 sec Building library RTX (LPC1768, ARM) Building library RTOS (LPC1768, ARM) Building project MUTEX (LPC1768, ARM) TargetTest::LPC1768::ARM::RTOS_2::Mutex resource lock [OK] in 12.51 of 20 sec TargetTest::LPC1768::ARM::RTOS_2::Mutex resource lock [OK] in 12.50 of 20 sec TargetTest::LPC1768::ARM::RTOS_2::Mutex resource lock [OK] in 12.54 of 20 sec TargetTest::LPC1768::ARM::RTOS_2::Mutex resource lock [OK] in 12.51 of 20 sec Test summary: +--------+---------+-----------+---------+-------------------------+--------------------+---------------+-------+ | Result | Target | Toolchain | Test ID | Test Description | Elapsed Time (sec) | Timeout (sec) | Loops | +--------+---------+-----------+---------+-------------------------+--------------------+---------------+-------+ | OK | LPC1768 | ARM | RTOS_1 | Basic thread | 12.57 | 15 | 3/3 | | OK | LPC1768 | ARM | RTOS_3 | Semaphore resource lock | 11.49 | 20 | 5/5 | | OK | LPC1768 | ARM | MBED_11 | Ticker Int | 12.51 | 20 | 5/5 | | OK | LPC1768 | ARM | MBED_10 | Hello World | 2.37 | 10 | 3/3 | | OK | LPC1768 | ARM | MBED_16 | RTC | 5.67 | 15 | 3/3 | | OK | LPC1768 | ARM | RTOS_2 | Mutex resource lock | 12.51 | 20 | 4/4 | +--------+---------+-----------+---------+-------------------------+--------------------+---------------+-------+ Result: 6 OK Shuffle Seed: 0.8837216077 Completed in 300.14 sec
What is host test?¶
Test suite support test supervisor concept. This concept is realized by separate Python script called "host test". Host tests can be found in workspace_tools/host_tests/ directory. Host test script is executed in parallel with test runner to monitor test execution. Basic host test just monitors device's default serial port for test results returned by test runner. Simple tests will print test result on serial port. In other cases host tests can for example judge by test results returned by test runner is test passed or failed. It all depends on test itself. In some cases host test can be TCP server echoing packets from test runner and judging packet loss. In other cases it can just check if values returned from accelerometer are actually valid (sane).
Peripherals testing¶
When creating MUTs configuration file (switch -M) you can define in MUTs JSON file peripherals connected to your device:
muts_all.json
{
"1" : {"mcu" : "KL25Z",
"port" : "COM39",
"disk" : "E:\\",
"peripherals" : ["SD", "24LC256"]}
}
You can force test suite to run only common tests (switch -C) or only peripheral tests (switch -P).
singletest.py -i test_spec.json -M muts_all.json -C
will not include tests for SD card and EEPROM 24LC256.
singletest.py -i test_spec.json -M muts_all.json -P
will only run tests bind to SD card and EEPROM 24LC256.
Note: option -P is useful for example in cases when you have same platform and different shields you want to test. No need to test common part all the time (timers, RTC, RTOS etc.). You can force to test peripherals only on some devices and for example only common tests on other devices.
Additional MUTs configuration file settings¶
You can add extra information to each MUT configuration. In particular you can specify which flashing (binary copy method) should be used, how to reset target and for example set reset timeout (used to delay test execution just after reset).
muts_all.json
{
"1" : {"mcu" : "LPC1768",
"port" : "COM77",
"disk" : "G:\\",
"peripherals" : ["TMP102", "digital_loop", "port_loop", "analog_loop", "SD"]},
"2" : {"mcu" : "KL25Z",
"port" : "COM89",
"disk" : "F:\\",
"peripherals" : ["SD", "24LC256", "KL25Z"],
"copy_method" : "copy",
"reset_type" : "default",
"reset_tout" : "2"},
"3" : {"mcu" : "LPC11U24",
"port" : "COM76",
"disk" : "E:\\",
"peripherals" : []}
}
Please note that for MUT no. 2 few extra parameters were defined: copy_method, reset_type and reset_tout. Using this extra options you can tell test suite more about MUT you are using. This will allow you to be more flexible in terms of how you configure and use your MUTs.
- copy_method - STRING - tells test suite which binary copy method should be used.
- reset_type - STRING - some boards may require special reset handling, for example vendor specific command must be executed to reset device.
- reset_tout - INTEGER - extra timeout just after device is reseted. May be used to wait for few seconds so device may finish booting, flashing data internally etc.
You may notice that singletest.py command line help contains description about: - Option -c (in MUTs file called copy_method) with available copy methods supported by test suite plugin system. - Option -r (in MUTs file called reset_type) with available reset methods supported by test suite plugin system.
Part of help listing for singletest.py:
-c COPY_METHOD, --copy-method=COPY_METHOD
Select binary copy (flash) method. Default is Python's
shutil.copy() method. Plugin support: copy, cp,
default, eACommander, eACommander-usb, xcopy
-r MUT_RESET_TYPE, --reset-type=MUT_RESET_TYPE
Extra reset method used to reset MUT by host test
script. Plugin support: default, eACommander,
eACommander-usb
Set common output binary name for all tests¶
In some cases you would like to have the same name for all tests. You can use switch to specify (without extension) build script output binary name.
In below example we would like to have all test binaries called firmware.bin (or with other extension like .elf, .hex depending on target accepted format).
singletest.py -i test_spec.json -M muts_all.json --firmware-name firmware
Building library CMSIS (LPC1768, ARM) Building library MBED (LPC1768, ARM) Building project TICKER to firmware (LPC1768, ARM) Building project HELLO to firmware (LPC1768, ARM) Building project RTC to firmware (LPC1768, ARM) Building library RTX (LPC1768, ARM) Building library RTOS (LPC1768, ARM) Building project SEMAPHORE to firmware (LPC1768, ARM) Building library RTX (LPC1768, ARM) Building library RTOS (LPC1768, ARM) Building project MUTEX to firmware (LPC1768, ARM) Building library RTX (LPC1768, ARM) Building library RTOS (LPC1768, ARM) Building project BASIC to firmware (LPC1768, ARM) Completed in 0.39 sec
Only build libraries and tests¶
You can force test script to only build all libraries and tests:
singletest.py -i test_spec.json -M muts_all.json -O
Building library CMSIS (LPC1768, ARM) Building library MBED (LPC1768, ARM) Building project TICKER (LPC1768, ARM) Building project HELLO (LPC1768, ARM) Building project RTC (LPC1768, ARM) Building library RTX (LPC1768, ARM) Building library RTOS (LPC1768, ARM) Building project SEMAPHORE (LPC1768, ARM) Building library RTX (LPC1768, ARM) Building library RTOS (LPC1768, ARM) Building project MUTEX (LPC1768, ARM) Building library RTX (LPC1768, ARM) Building library RTOS (LPC1768, ARM) Building project BASIC (LPC1768, ARM) Completed in 0.37 sec
Note: this can be useful when you are integrating test script with CI (Continuous Integration system). You can make libraries and test building process as separate CI job step. And test execution as another separate step which will be not executed by CI system if e.g. building tests with dependencies fails.
Build your libraries and tests on multiple CPUs¶
Use option -j to build tests with dependencies on multiple cores.
Below example will use 4 (four) cores to build your dependencies to tests:
singletest.py -i test_spec.json -M muts_all.json -j 4
Check MUTs and test specification configuration¶
You can check MUTs configuration read from configuration file (switch -M) and test specification read from test specification file (switch -i) by setting switch ---config.
Below command will check configuration against configuration in test_spec.json and muts_all.json file.
singletest.py -i test_spec.json -M muts_all.json --config
MUTs configuration in muts_all.json: +-------+-------------+---------------+------+-------+ | index | peripherals | mcu | disk | port | +-------+-------------+---------------+------+-------+ | 13 | 24LC256 | NUCLEO_L152RE | F:\ | COM9 | | 15 | 24LC256 | NUCLEO_F401RE | G:\ | COM25 | | 14 | 24LC256 | NUCLEO_F302R8 | E:\ | COM34 | | 17 | 24LC256 | NUCLEO_L053R8 | E:\ | COM35 | | 16 | 24LC256 | NUCLEO_F030R8 | H:\ | COM12 | | 4 | 24LC256 | NUCLEO_F103RB | I:\ | COM11 | +-------+-------------+---------------+------+-------+ Test specification in test_spec.json: +---------------+-----+------+ | mcu | ARM | uARM | +---------------+-----+------+ | NUCLEO_F103RB | Yes | Yes | | NUCLEO_F030R8 | Yes | Yes | | NUCLEO_L152RE | Yes | Yes | | NUCLEO_F401RE | Yes | Yes | | NUCLEO_F302R8 | Yes | Yes | +---------------+-----+------+
Simple checks can be performed to see if you actually properly configured test specification: In below example user defined his/hers test specification file like this:
test_spec.json
{
"targets": {
"NUCLEO_F103RB": ["uARM", "ARM", "IAR"],
"NUCLEO_F030R8": ["uARM", "ARM"],
"NUCLEO_F401RE": ["uARM", "ARM"],
"NUCLEO_F302R8": ["uARM", "ARM", "GCC_ARM"],
"NUCLEO_L152RE": ["uARM", "ARM"]
}
}
Because NUCLEO platform do not support IAR or GCC_ARM toolchains "Toolchain conflicts" will be printed like below:
MUTs configuration in muts_all.json:
+-------+-------------+---------------+------+-------+
| index | peripherals | mcu | disk | port |
+-------+-------------+---------------+------+-------+
| 13 | 24LC256 | NUCLEO_L152RE | F:\ | COM9 |
| 15 | 24LC256 | NUCLEO_F401RE | G:\ | COM25 |
| 14 | 24LC256 | NUCLEO_F302R8 | E:\ | COM34 |
| 17 | 24LC256 | NUCLEO_L053R8 | E:\ | COM35 |
| 16 | 24LC256 | NUCLEO_F030R8 | H:\ | COM12 |
| 4 | 24LC256 | NUCLEO_F103RB | I:\ | COM11 |
+-------+-------------+---------------+------+-------+
Test specification in test_spec.json:
+---------------+-----+---------+------+------+
| mcu | ARM | GCC_ARM | IAR | uARM |
+---------------+-----+---------+------+------+
| NUCLEO_F103RB | Yes | - | Yes* | Yes |
| NUCLEO_F030R8 | Yes | - | - | Yes |
| NUCLEO_L152RE | Yes | - | - | Yes |
| NUCLEO_F401RE | Yes | - | - | Yes |
| NUCLEO_F302R8 | Yes | Yes* | - | Yes |
+---------------+-----+---------+------+------+
Toolchain conflicts:
* Target NUCLEO_F103RB does not support IAR toolchain
* Target NUCLEO_F302R8 does not support GCC_ARM toolchain
JSON configuration file parsing hints¶
Test suite script can supply you with simple configuration file JSON related error notifications. For example when you have a typo in your JSON configuration file:
test_spec.json
{
"targets": {
"NUCLEO_F103RB": ["uARM"],
"NUCLEO_F030R8": ["uARM"],
"NUCLEO_F401RE": ["uARM"],
"NUCLEO_F302R8": ["uARM"] typo!,
"NUCLEO_L152RE": ["uARM"]
}
}
You should be notified and pointed to place where correct JSON parsing was interrupted:
singletest.py -i test_spec.json -M muts_all.json
Results in:
JSON file test_spec.json parsing failed. Reason: Expecting , delimiter: line 6 column 35 (char 158)
Line 1: {
Line 2: "targets": {
Line 3: "NUCLEO_F103RB": ["uARM"],
Line 4: "NUCLEO_F030R8": ["uARM"],
Line 5: "NUCLEO_F401RE": ["uARM"],
Line 6: "NUCLEO_F302R8": ["uARM"] typo!,
----------------------------------^
Vocabulary¶
Test runner¶
A test runner is an executable program that runs tests implemented using test framework and reports the test results.
MUT¶
MUT stands for Mbed Under Test - a device we have had connected to our host computer and
Appendix A¶
Example muts_all.json file¶
muts_all.json
{
"1" : {"mcu" : "LPC1768",
"port" : "COM4",
"disk" : "J:\\",
"peripherals" : ["TMP102", "digital_loop", "port_loop", "analog_loop", "SD"]},
"2" : {"mcu" : "KL25Z",
"port" : "COM39",
"disk" : "E:\\",
"peripherals" : ["SD", "24LC256", "KL25Z"]},
"3" : {"mcu" : "LPC11U24",
"port" : "COM5",
"disk" : "E:\\",
"peripherals" : ["TMP102", "digital_loop", "port_loop", "analog_loop"]},
"4" : {"mcu" : "NUCLEO_F103RB",
"port" : "COM11",
"disk" : "I:\\",
"peripherals" : ["24LC256"]},
"5" : {"mcu" : "NRF51822",
"port" : "COM21",
"disk" : "I:\\",
"peripherals" : []},
"6" : {"mcu" : "UBLOX_C027",
"port" : "COM14",
"disk" : "F:\\",
"peripherals" : []},
"7" : {"mcu" : "LPC2368",
"port" : "COM15",
"disk" : "G:\\",
"peripherals" : ["TMP102", "digital_loop", "port_loop", "analog_loop", "SD"]},
"8" : {"mcu" : "LPC812",
"port" : "COM6",
"disk" : "E:\\",
"peripherals" : ["24LC256"]},
"9" : {"mcu" : "LPC1549",
"port" : "COM32",
"disk" : "I:\\",
"peripherals" : ["24LC256"]},
"10" : {"mcu" : "LPC4088",
"port" : "COM17",
"disk" : "E:\\",
"peripherals" : []},
"11" : {"mcu" : "KL46Z",
"port" : "COM40",
"disk" : "F:\\",
"peripherals" : ["SD", "24LC256", "KL46Z"]},
"12" : {"mcu" : "K64F",
"port" : "COM18",
"disk" : "E:\\",
"peripherals" : ["ethernet"]},
"13" : {"mcu" : "NUCLEO_L152RE",
"port" : "COM9",
"disk" : "E:\\",
"peripherals" : []},
"14" : {"mcu" : "NUCLEO_F302R8",
"port" : "COM34",
"disk" : "E:\\",
"peripherals" : ["24LC256"]},
"15" : {"mcu" : "NUCLEO_F401RE",
"port" : "COM47",
"disk" : "E:\\",
"peripherals" : ["SD"]},
"16" : {"mcu" : "NUCLEO_F030R8",
"port" : "COM12",
"disk" : "H:\\",
"peripherals" : ["24LC256"]},
"17" : {"mcu" : "NUCLEO_L053R8",
"port" : "COM35",
"disk" : "E:\\",
"peripherals" : ["24LC256"]},
"18" : {"mcu" : "LPC11U68",
"port" : "COM36",
"disk" : "E:\\",
"peripherals" : ["24LC256"]},
"19" : {"mcu" : "NUCLEO_F411RE",
"port" : "COM44",
"disk" : "F:\\",
"peripherals" : ["24LC256"]},
"20" : {"mcu" : "NUCLEO_F072RB",
"port" : "COM45",
"disk" : "F:\\",
"peripherals" : ["24LC256"]},
"21" : {"mcu" : "NUCLEO_F334R8",
"port" : "COM46",
"disk" : "E:\\",
"peripherals" : ["24LC256"]},
"22" : {"mcu" : "XADOW_M0",
"port" : "COM51",
"disk" : "E:\\",
"peripherals" : []}
}
Example test_spec.json file¶
test_spec.json
{
"targets": {
"KL25Z": ["ARM", "GCC_ARM"],
"LPC1768": ["ARM", "GCC_ARM", "GCC_CR", "GCC_CS", "IAR"],
"LPC11U24": ["uARM"],
"NRF51822": ["ARM"],
"LPC2368": ["ARM"],
"LPC4088": ["ARM"],
"NUCLEO_F334R8" : ["uARM"]
}
}