SDK Development


Discuss and develop the mbed SDK.

Mbed SDK automated test suite reporting

Table of Contents

  1. Rationale
  2. Reporting facilities
  3. Console output (default)
  4. HTML reports
  5. JUnit XML formated test reports

Mbed OS 2 and Mbed OS 5

This is a guide for Mbed OS 2. If you’re working with Mbed OS 5, please see the Contributing guide and Testing guide in the new handbook.

Rationale

Test suite and automation tools ought to report test data in at least few formats. Each format should satisfy different need. For example:

  • As console output while test suite is running as CLI (command line interface) application.
  • HTML report is human readable and can be presented to audience.
  • JUnit format (formatted XML) is used by various tools, test result parsers to generate reports, combine test data or just present it in Continuous Integration report page.
  • Other like database dump: export test result data directly to database located locally or remotely.

Reporting facilities

All mentioned above methods are supported by test suite at this time.

Console output (default)

Console output is by default printing compile, build and rest result data. Additionally switches like -v or -V can be used for command line to get extra prints from test suite configuration and diagnostic or MUT (mbed under test) console output respectively.

HTML reports

HTML reports are used to generate simple HTML page with intuitive and simple test result matrix. HTML report is a single page containing all test results and test console output data.

Command line switch:

--report-html

can be used to generate HTMLformatted test result

singletest.py -i test_spec.json -M muts_all.json --report-html myreport.hmtl

Below you can see example HTML report with active test result tool-tip: /media/uploads/PrzemekWirkus/test_report_html.png

JUnit XML formated test reports

http://junit.org/ JUnit test results reports can be used to generate JUnit compliant XML file with test suite test results. Users can export JUnit XML files to e.g. IDEs like Eclipse to see results directly or they can feed JUnit result reports to CI systems for further automated processing.

Command line switch:

--report-junit

can be used to generate JUnit formatted test result

singletest.py -i test_spec.json -M muts_all.json --report-junit myreport.xml

Below you can see example of JUnit XML report:

<?xml version="1.0" ?>
<testsuites>
	<testsuite errors="0" failures="0" name="test.suite.KL25Z.ARM" skipped="0" tests="1" time="0.55">
		<properties>
			<property name="skipped" value=""/>
			<property name="toolchain" value="ARM"/>
			<property name="target" value="KL25Z"/>
			<property name="shuffle_random_seed" value="0.9237068745"/>
			<property name="test.path.KL25Z.ARM.DTCT_1" value="C:\Work\winbb\slave\BuildRelease\build\build\test\KL25Z\ARM\DTCT_1\detect.bin"/>
			<property name="build_dir" value="C:\Work\winbb\slave\BuildRelease\build\build\test\KL25Z\ARM"/>
			<property name="build_mbed_libs_result" value="True"/>
			<property name="test_ids" value=""/>
			<property name="test.libs.KL25Z.ARM.DTCT_1" value=""/>
			<property name="clean" value="False"/>
			<property name="jobs" value="1"/>
			<property name="test.loops.KL25Z.ARM.DTCT_1" value="1"/>
			<property name="test.duration.KL25Z.ARM.DTCT_1" value="10"/>
		</properties>
		<testcase classname="test.KL25Z.ARM.DTCT_1" name="Simple detect test" time="0.550000">
			<system-out>Host test instrumentation on port: &quot;COM39&quot; and disk: &quot;F:\&quot;
{{start}}

{{start}}
HOST: Detecting target name...
MBED: Target 'KL25Z'
HOST: MUT Target name 'KL25Z', expected 'KL25Z'... [OK]
MBED: Test ID 'DTCT_1'
MBED: UUID '60bed2a6-60bb-44d9-909d-97acbc58f2ec'

{success}
{end}
</system-out>
		</testcase>
	</testsuite>
	<testsuite errors="0" failures="0" name="test.suite.LPC1768.ARM" skipped="0" tests="1" time="1.37">
		<properties>
			<property name="test.duration.LPC1768.ARM.DTCT_1" value="10"/>
			<property name="skipped" value=""/>
			<property name="toolchain" value="ARM"/>
			<property name="target" value="LPC1768"/>
			<property name="shuffle_random_seed" value="0.9237068745"/>
			<property name="test.path.LPC1768.ARM.DTCT_1" value="C:\Work\winbb\slave\BuildRelease\build\build\test\LPC1768\ARM\DTCT_1\detect.bin"/>
			<property name="build_dir" value="C:\Work\winbb\slave\BuildRelease\build\build\test\LPC1768\ARM"/>
			<property name="build_mbed_libs_result" value="True"/>
			<property name="test_ids" value=""/>
			<property name="clean" value="False"/>
			<property name="jobs" value="1"/>
			<property name="test.loops.LPC1768.ARM.DTCT_1" value="1"/>
			<property name="test.libs.LPC1768.ARM.DTCT_1" value=""/>
		</properties>
		<testcase classname="test.LPC1768.ARM.DTCT_1" name="Simple detect test" time="1.370000">
			<system-out>Host test instrumentation on port: &quot;COM77&quot; and disk: &quot;E:\&quot;
{{start}}

{{start}}
HOST: Detecting target name...
MBED: Target 'LPC1768'
HOST: MUT Target name 'LPC1768', expected 'LPC1768'... [OK]
MBED: Test ID 'DTCT_1'
MBED: UUID 'b3c29946-4a0d-4a54-92ba-817fb19a9284'

{success}
{end}
</system-out>
		</testcase>
	</testsuite>
	<testsuite errors="0" failures="0" name="test.suite.NRF51822.ARM" skipped="0" tests="1" time="0.9">
		<properties>
			<property name="skipped" value=""/>
			<property name="toolchain" value="ARM"/>
			<property name="target" value="NRF51822"/>
			<property name="shuffle_random_seed" value="0.9237068745"/>
			<property name="test.loops.NRF51822.ARM.DTCT_1" value="1"/>
			<property name="test.libs.NRF51822.ARM.DTCT_1" value=""/>
			<property name="build_dir" value="C:\Work\winbb\slave\BuildRelease\build\build\test\NRF51822\ARM"/>
			<property name="build_mbed_libs_result" value="True"/>
			<property name="test_ids" value=""/>
			<property name="test.duration.NRF51822.ARM.DTCT_1" value="10"/>
			<property name="clean" value="False"/>
			<property name="jobs" value="1"/>
			<property name="test.path.NRF51822.ARM.DTCT_1" value="C:\Work\winbb\slave\BuildRelease\build\build\test\NRF51822\ARM\DTCT_1\detect.hex"/>
		</properties>
		<testcase classname="test.NRF51822.ARM.DTCT_1" name="Simple detect test" time="0.900000">
			<system-out>Host test instrumentation on port: &quot;COM58&quot; and disk: &quot;H:\&quot;
{{start}}

{{start}}
HOST: Detecting target name...
MBED: Target 'NRF51822'
HOST: MUT Target name 'NRF51822', expected 'NRF51822'... [OK]
MBED: Test ID 'DTCT_1'
MBED: UUID 'faa0309c-35aa-4256-b0c1-80be362af5c1'

{success}
{end}
</system-out>
		</testcase>
	</testsuite>
	<testsuite errors="0" failures="0" name="test.suite.NRF51822.GCC_ARM" skipped="0" tests="1" time="0.91">
		<properties>
			<property name="test.duration.NRF51822.GCC_ARM.DTCT_1" value="10"/>
			<property name="skipped" value=""/>
			<property name="toolchain" value="GCC_ARM"/>
			<property name="target" value="NRF51822"/>
			<property name="shuffle_random_seed" value="0.9237068745"/>
			<property name="test.path.NRF51822.GCC_ARM.DTCT_1" value="C:\Work\winbb\slave\BuildRelease\build\build\test\NRF51822\GCC_ARM\DTCT_1\detect.hex"/>
			<property name="build_dir" value="C:\Work\winbb\slave\BuildRelease\build\build\test\NRF51822\GCC_ARM"/>
			<property name="build_mbed_libs_result" value="True"/>
			<property name="test_ids" value=""/>
			<property name="clean" value="False"/>
			<property name="test.loops.NRF51822.GCC_ARM.DTCT_1" value="1"/>
			<property name="jobs" value="1"/>
			<property name="test.libs.NRF51822.GCC_ARM.DTCT_1" value=""/>
		</properties>
		<testcase classname="test.NRF51822.GCC_ARM.DTCT_1" name="Simple detect test" time="0.910000">
			<system-out>Host test instrumentation on port: &quot;COM58&quot; and disk: &quot;H:\&quot;
{{start}}

{{start}}
HOST: Detecting target name...
MBED: Target 'NRF51822'
HOST: MUT Target name 'NRF51822', expected 'NRF51822'... [OK]
MBED: Test ID 'DTCT_1'
MBED: UUID '73ee32a1-8f2d-434c-bc44-2dc18d5e2d14'

{success}
{end}
</system-out>
		</testcase>
	</testsuite>
	<testsuite errors="0" failures="0" name="test.suite.LPC1768.uARM" skipped="0" tests="1" time="1.3">
		<properties>
			<property name="test.loops.LPC1768.uARM.DTCT_1" value="1"/>
			<property name="skipped" value=""/>
			<property name="toolchain" value="uARM"/>
			<property name="target" value="LPC1768"/>
			<property name="shuffle_random_seed" value="0.9237068745"/>
			<property name="build_dir" value="C:\Work\winbb\slave\BuildRelease\build\build\test\LPC1768\uARM"/>
			<property name="build_mbed_libs_result" value="True"/>
			<property name="test_ids" value=""/>
			<property name="test.path.LPC1768.uARM.DTCT_1" value="C:\Work\winbb\slave\BuildRelease\build\build\test\LPC1768\uARM\DTCT_1\detect.bin"/>
			<property name="clean" value="False"/>
			<property name="jobs" value="1"/>
			<property name="test.libs.LPC1768.uARM.DTCT_1" value=""/>
			<property name="test.duration.LPC1768.uARM.DTCT_1" value="10"/>
		</properties>
		<testcase classname="test.LPC1768.uARM.DTCT_1" name="Simple detect test" time="1.300000">
			<system-out>Host test instrumentation on port: &quot;COM77&quot; and disk: &quot;E:\&quot;
{{start}}

{{start}}
HOST: Detecting target name...
MBED: Target 'LPC1768'
HOST: MUT Target name 'LPC1768', expected 'LPC1768'... [OK]
MBED: Test ID 'DTCT_1'
MBED: UUID 'c8eac564-aa80-4cc1-a09f-b242751a6346'

{success}
{end}
</system-out>
		</testcase>
	</testsuite>
	<testsuite errors="0" failures="0" name="test.suite.NUCLEO_F103RB.uARM" skipped="0" tests="1" time="0.55">
		<properties>
			<property name="test.duration.NUCLEO_F103RB.uARM.DTCT_1" value="10"/>
			<property name="skipped" value=""/>
			<property name="toolchain" value="uARM"/>
			<property name="target" value="NUCLEO_F103RB"/>
			<property name="shuffle_random_seed" value="0.9237068745"/>
			<property name="test.libs.NUCLEO_F103RB.uARM.DTCT_1" value=""/>
			<property name="build_dir" value="C:\Work\winbb\slave\BuildRelease\build\build\test\NUCLEO_F103RB\uARM"/>
			<property name="build_mbed_libs_result" value="True"/>
			<property name="test_ids" value=""/>
			<property name="test.loops.NUCLEO_F103RB.uARM.DTCT_1" value="1"/>
			<property name="clean" value="False"/>
			<property name="test.path.NUCLEO_F103RB.uARM.DTCT_1" value="C:\Work\winbb\slave\BuildRelease\build\build\test\NUCLEO_F103RB\uARM\DTCT_1\detect.bin"/>
			<property name="jobs" value="1"/>
		</properties>
		<testcase classname="test.NUCLEO_F103RB.uARM.DTCT_1" name="Simple detect test" time="0.550000">
			<system-out>Host test instrumentation on port: &quot;COM64&quot; and disk: &quot;G:\&quot;
{{start}}

{{start}}
HOST: Detecting target name...
MBED: Target 'NUCLEO_F103RB'
HOST: MUT Target name 'NUCLEO_F103RB', expected 'NUCLEO_F103RB'... [OK]
MBED: Test ID 'DTCT_1'
MBED: UUID '234933ae-3849-4269-81d3-2f28744e1595'

{success}
{end}
</system-out>
		</testcase>
	</testsuite>
</testsuites>
  • You can also export test suite test results to database. Currently we are supporting MySQL database but nothings stands in the way to support other DB. There is a special switch: 
--db

used set MySQL DB connection string to mbed SDK test database.

Below example runs test suite and forces it to run two tests RTOS_7 and ROTS_2 for devices defined in muts_all.json and test_spec.json configuration files. In this example we are connecting to MySQL database names 'mbed_test' located locally: 127.0.0.1 (localhost). Note that user mbed_user (password: your_password) should have permissions to access to mbed_test database.

singletest.py -i test_spec.json -M muts_all.json -n RTOS_7,RTOS_2 --db 'mysql://mbed_user:your_password@127.0.0.1/mbed_test'

Above command will point test suite to database and test results will be dumped to mbed_test database.

To diagnose database connection problems please combine:

--db

switch with:

--config

to get database connection overview:

singletest.py -i test_spec.json -M muts_all.json --db mysql://user:password@127.0.0.1/mbed_test --config
Connecting to database 'mysql://user:password@127.0.0.1/mbed_test'... ok
Detecting database...
        innodb_version: 1.1.8
        protocol_version: 10
        slave_type_conversions:
        version: 5.5.24-log
        version_comment: MySQL Community Server (GPL)
        version_compile_machine: x86
        version_compile_os: Win32
Disconnecting... done

Below you can see database scheme which must be used to create mbed_test database for MySQL DB engine:

Database scheme used by test suite

-- phpMyAdmin SQL Dump
-- version 3.5.1
-- http://www.phpmyadmin.net
--
-- Host: localhost
-- Generation Time: Nov 21, 2014 at 09:57 AM
-- Server version: 5.5.24-log
-- PHP Version: 5.3.13

SET SQL_MODE="NO_AUTO_VALUE_ON_ZERO";
SET AUTOCOMMIT=0;
START TRANSACTION;
SET time_zone = "+00:00";


/*!40101 SET @OLD_CHARACTER_SET_CLIENT=@@CHARACTER_SET_CLIENT */;
/*!40101 SET @OLD_CHARACTER_SET_RESULTS=@@CHARACTER_SET_RESULTS */;
/*!40101 SET @OLD_COLLATION_CONNECTION=@@COLLATION_CONNECTION */;
/*!40101 SET NAMES utf8 */;

--
-- Database: `mbed_test`
--

-- --------------------------------------------------------

--
-- Table structure for table `mtest_build_id`
--

CREATE TABLE IF NOT EXISTS `mtest_build_id` (
  `mtest_build_id_pk` int(11) NOT NULL AUTO_INCREMENT,
  `mtest_build_id_name` varchar(64) COLLATE utf8_unicode_ci NOT NULL,
  `mtest_build_id_desc` varchar(512) COLLATE utf8_unicode_ci NOT NULL DEFAULT '',
  `mtest_build_id_location` text COLLATE utf8_unicode_ci NOT NULL,
  `mtest_build_id_muts` text COLLATE utf8_unicode_ci NOT NULL,
  `mtest_build_id_test_spec` text COLLATE utf8_unicode_ci NOT NULL,
  `mtest_build_id_type_fk` int(11) NOT NULL,
  `mtest_build_id_status_fk` int(11) NOT NULL,
  `mtest_build_id_shuffle_seed` varchar(32) COLLATE utf8_unicode_ci NOT NULL DEFAULT '0',
  `mtest_build_id_extra` text COLLATE utf8_unicode_ci NOT NULL,
  `mtest_build_id_timestamp` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP,
  PRIMARY KEY (`mtest_build_id_pk`)
) ENGINE=InnoDB  DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci AUTO_INCREMENT=5 ;

-- --------------------------------------------------------

--
-- Table structure for table `mtest_build_id_status`
--

CREATE TABLE IF NOT EXISTS `mtest_build_id_status` (
  `mtest_build_id_status_pk` int(11) NOT NULL AUTO_INCREMENT,
  `mtest_build_id_status_name` varchar(64) COLLATE utf8_unicode_ci NOT NULL,
  PRIMARY KEY (`mtest_build_id_status_pk`)
) ENGINE=InnoDB  DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci AUTO_INCREMENT=5 ;

--
-- Dumping data for table `mtest_build_id_status`
--

INSERT INTO `mtest_build_id_status` (`mtest_build_id_status_pk`, `mtest_build_id_status_name`) VALUES
(1, 'Started'),
(2, 'In Progress'),
(3, 'Completed'),
(4, 'Failed');

-- --------------------------------------------------------

--
-- Table structure for table `mtest_build_id_type`
--

CREATE TABLE IF NOT EXISTS `mtest_build_id_type` (
  `mtest_build_id_type_pk` int(11) NOT NULL AUTO_INCREMENT,
  `mtest_build_id_type_name` varchar(64) COLLATE utf8_unicode_ci NOT NULL,
  `mtest_build_id_type_desc` varchar(256) COLLATE utf8_unicode_ci NOT NULL DEFAULT '',
  PRIMARY KEY (`mtest_build_id_type_pk`)
) ENGINE=InnoDB  DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci AUTO_INCREMENT=3 ;

--
-- Dumping data for table `mtest_build_id_type`
--

INSERT INTO `mtest_build_id_type` (`mtest_build_id_type_pk`, `mtest_build_id_type_name`, `mtest_build_id_type_desc`) VALUES
(1, 'Test', ''),
(2, 'Build Only', '');

-- --------------------------------------------------------

--
-- Table structure for table `mtest_target`
--

CREATE TABLE IF NOT EXISTS `mtest_target` (
  `mtest_target_pk` int(11) NOT NULL AUTO_INCREMENT,
  `mtest_target_name` varchar(64) COLLATE utf8_unicode_ci NOT NULL,
  PRIMARY KEY (`mtest_target_pk`)
) ENGINE=InnoDB  DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci AUTO_INCREMENT=3 ;

-- --------------------------------------------------------

--
-- Table structure for table `mtest_test_entry`
--

CREATE TABLE IF NOT EXISTS `mtest_test_entry` (
  `mtest_test_result_pk` int(11) NOT NULL AUTO_INCREMENT,
  `mtest_build_id_fk` int(11) NOT NULL,
  `mtest_target_fk` int(11) NOT NULL,
  `mtest_toolchain_fk` int(11) NOT NULL,
  `mtest_test_type_fk` int(11) NOT NULL,
  `mtest_test_id_fk` int(11) NOT NULL,
  `mtest_test_result_fk` int(11) NOT NULL,
  `mtest_test_output` text COLLATE utf8_unicode_ci NOT NULL,
  `mtest_test_time` float NOT NULL,
  `mtest_test_timeout` float NOT NULL,
  `mtest_test_loop_no` int(11) NOT NULL DEFAULT '1',
  `mtest_test_result_extra` text COLLATE utf8_unicode_ci NOT NULL,
  `mtest_test_result_timestamp` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP,
  PRIMARY KEY (`mtest_test_result_pk`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci AUTO_INCREMENT=1 ;

-- --------------------------------------------------------

--
-- Table structure for table `mtest_test_id`
--

CREATE TABLE IF NOT EXISTS `mtest_test_id` (
  `mtest_test_id_pk` int(11) NOT NULL AUTO_INCREMENT,
  `mtest_test_id_name` varchar(64) COLLATE utf8_unicode_ci NOT NULL,
  `mtest_test_id_desc` varchar(256) COLLATE utf8_unicode_ci NOT NULL DEFAULT '',
  PRIMARY KEY (`mtest_test_id_pk`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci AUTO_INCREMENT=1 ;

-- --------------------------------------------------------

--
-- Table structure for table `mtest_test_result`
--

CREATE TABLE IF NOT EXISTS `mtest_test_result` (
  `mtest_test_result_pk` int(11) NOT NULL AUTO_INCREMENT,
  `mtest_test_result_name` varchar(64) COLLATE utf8_unicode_ci NOT NULL,
  PRIMARY KEY (`mtest_test_result_pk`)
) ENGINE=InnoDB  DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci AUTO_INCREMENT=5 ;

--
-- Dumping data for table `mtest_test_result`
--

INSERT INTO `mtest_test_result` (`mtest_test_result_pk`, `mtest_test_result_name`) VALUES
(1, 'OK'),
(2, 'FAIL'),
(3, 'TIMEOUT'),
(4, 'IOERR_COPY');

-- --------------------------------------------------------

--
-- Table structure for table `mtest_test_type`
--

CREATE TABLE IF NOT EXISTS `mtest_test_type` (
  `mtest_test_type_pk` int(11) NOT NULL AUTO_INCREMENT,
  `mtest_test_type_name` varchar(64) COLLATE utf8_unicode_ci NOT NULL,
  PRIMARY KEY (`mtest_test_type_pk`)
) ENGINE=InnoDB  DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci AUTO_INCREMENT=2 ;

--
-- Dumping data for table `mtest_test_type`
--

INSERT INTO `mtest_test_type` (`mtest_test_type_pk`, `mtest_test_type_name`) VALUES
(1, 'SingleTest');

-- --------------------------------------------------------

--
-- Table structure for table `mtest_toolchain`
--

CREATE TABLE IF NOT EXISTS `mtest_toolchain` (
  `mtest_toolchain_pk` int(11) NOT NULL AUTO_INCREMENT,
  `mtest_toolchain_name` varchar(64) COLLATE utf8_unicode_ci NOT NULL,
  PRIMARY KEY (`mtest_toolchain_pk`)
) ENGINE=InnoDB  DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci AUTO_INCREMENT=2 ;
COMMIT;

/*!40101 SET CHARACTER_SET_CLIENT=@OLD_CHARACTER_SET_CLIENT */;
/*!40101 SET CHARACTER_SET_RESULTS=@OLD_CHARACTER_SET_RESULTS */;
/*!40101 SET COLLATION_CONNECTION=@OLD_COLLATION_CONNECTION */;
All wikipages