Quality Assurance of mbed Libraries / Examples

09 Oct 2014

As a new member of the mbed community I was quite disappointed with the some of the software I would try and import ranging from compile errors to link errors.

Surely when software is posted onto the mbed developer community it would be possible to :-

1. Include unit tests 2. Include wrapper programs for testing 3. If a library relies on another library then it should automatically import that library (because tracking down library dependencies is a pain) 4. Automated testing of modules would be nice 5. More importantly if the program / library should fail then move onto the library / program into a "Here by Dragons" area or mark it in someway as not working

If this is covered already apologies ...

09 Oct 2014

Totally agree. Have a look at and a chat with:

http://developer.mbed.org/blog/entry/software-automation-mbed-testfarm/

:)

09 Oct 2014

Hi Simon,

I appreciate your comments. They represent what any good programmer would expect from a platform.

There is an ongoing effort to add unit tests for the core mbed libraries (or at least the ones directly maintained by the mbed team). Our automated tests are made up of Python scripts built on top of the cpputest unit-test framework; and you'll find mention of how to exercise these tests across various platforms.

The responsibility of testing component libraries submitted by users rests upon the original developers. The core mbed team doesn't have the resources for that.

If component libraries are tightly coupled with one another, they typically come combined within 'programs' which link the dependencies. Otherwise, it should be possible to combine components arbitrarily as long as they satisfy well defined interfaces. Mbed doesn't want to arbitrate in this. Users are free to create programs to demonstrate interworking components.

We don't have the resources to do all kinds of testing for all components on the large number of platforms available. Most users are able to tweak existing demos and make them work on their particular platform. This is a part of the current mbed experience; but you are correct in pointing out that it should all be seamless.

We're looking for enthusiastic and conscious engineers, such as yourself. Please consider applying for an open position at mbed.org; and help us improve mbed.

09 Oct 2014

Many libraries should have (almost) everything you ask for. Definately not all, but if you click on dependents of a library it should quickly be visible if it has a HelloWorld program. It can then happen that in newer versions of dependencies the library doesn't work anymore, but the HelloWorld program should import it from a state which always works, for the given target.

That is a second thing: Programs more than 1-2 years old are from the time when only the LPC1768/LPC11u24 mbeds existed, so sometimes they are optimized using some direct register accesses, which obviously wont compile for newer targets. Automated importing of dependencies should also happen (I admit it sometimes breaks, but in my experience that is less than 1% of the libs), a few libs don't automatically import their dependencies, which can have multiple reasons. One thing I would like to see is that the online compiler automatically handles importing duplicate libraries: The issue is if you have a library which relies on for example MODSERIAL, and a second one has the same, and they both automatically import it, the compiler will have issues with duplicate files.

For unit tests it also depends on the object the library is intended for. For example my libraries for I2C sensors preferably have the option to read out a WHOAMI register, or at least check if there is something at the I2C address. But if you have an LCD display with write-only access there is no way to do this from software.

I am curious though, could you link some of the libs you have had issues with?

09 Oct 2014

Hi guys,

For off-line mbed SDK test support we've got this (work in progress):

PS: I'm just implementing JUnit reporting so users can just "plug" out test suite to CI like Jenkins.

13 Oct 2014

All,

Looking at the various links, above, I have a few questions :-

1. The software automation test platform appears to be dated April 2014, whats the progress there ? 2. Why can't you integrate your test tools with all of the libraries and programs to see if they currently compile and link against the "head builds", when libraries are posted to the site require that they have unittests and requisite build dependencies. This would ensure you have a consistently clean environment (at least as far as library links) from the website developer end. 3. I tried to build the HTTPSClient program on an LPC1768

14 Oct 2014

Hello Simon, Thank you for your all questions.

1. Test automation suite is constantly under development. I keep adding new features. For example now I'm adding small features related to JUnit test result output so test automation CI users can directly export test results into JUnit XML formatted files and parse them using e.g. JUnit plugins in Jenkins CI or similar environment.

Please note that test automation suite is used only off-line which means it is not integrated with on-line IDE / compiler.

2. I never stated I can't integrate test tools with all libraries and programs to see if they currently compile and link against the "head builds". Actually we are performing similar automation in our internal CI. Test tools are using workspace_tools build scripts and we are building all libraries and link them with HEAD. We also do this for particular branches or pull requests. It depends what we want to achieve.

3. I think you had a problem with building HTTPSClient test on an LPC1768 ? If so, let me know and I will try to help you do that.

16 Nov 2014

I also am new to mbed. I have the same problems discussed above. I can't tell the difference between a users test script that was published and a supported library. Some indication of the source(who: mbed or user) and platform-set it was running on when published, would be very helpful.

Also, adequate documentation areas exist, but almost no one makes use of them. (wiki, readme, notebook,etc.) For code in a repository, there is no indication that a notebook page may reference it.

Almost none of the code is documented internally either! Every piece of code should have a comment header describing the intended use, example calling sequence and if it drives an external device example connections or wiring.

It also is VERY difficult for a mbed beginner to find "getting started" documentation and example usage. There are lots of different places with a little info, seldom dated in a way to determine usefulness and without reference to varying platforms. Many people have written up their starting experience, probably because they noticed the lack also, but they are often incomplete, aged, inept or even sometimes illiterate.

Also, there are too many places for questions/forum posts. I have noticed the same question posted in more than 4 different places wit differing sets of answers in each. There must be some way to restrict the question/answers to one searchable place. Fro example the same question is in: the questions at the bottom of the platform description page, the questions at the bottom of the localfilesystem library, cross posted in several forum Discussions.

Just today I searched (from the search bar at the top of most pages) for "Sparkfun" serial LCD. I have the very old one with the ATMEGA328 chip. I got 530 results, some of which looked like what I was after. The first 5 that I examined were obviously users test programs that only proved the device working. Not one of the first 10 had any kind of README or internal documentation to say which version of the SerialLCD it was for. None showed any hint of which wires to connect to which pins of any platform. I did not find one that implemented all of the commands the LCD accepts, which is OK but....

I had to open the main.cpp and read the un-commented code of each one to see if it even was written to address the same device I have.

This example maybe my fault for not searching correctly but, in 4 months I have not found the concise instructions for how to use this site.

I do like the eventual results. Mbed is much easier than coding the clocks by hand. But I get frustrated! In a prior life I managed software development and any of the above omissions were grounds for review/termination.

16 Nov 2014

After re-reading my above post, I realize I came across much too negative.

I apologize.

The intent of the post was to be constructive criticism. And a set of enhancement requests.

mbed is a fine product that I regularly enjoy. It could be better.

16 Nov 2014

The real offficial code all belongs to either mbed_official user, or the mbed team. There is generally not a reason to use a notebook to document code, since there are wikis for the code. That said in the past there weren't, so old libraries may have a different situation.

Every platform should have a getting started section. In general they promote documentation using the built-in doxygen, the components section which includes stuff like example programs, wiring, references, etc. But in the end they should not imo start rejecting programs from being public based on documentation. In the end it is mainly just hobbyist who just like to make their code available.

Now don't get me wrong, while I don't agree with everything you said, I do think there is alot to improve. The components section could be better. In platforms i want more search options (clock speed, ARM core, etc), and for example the handbook section should be completely redone since it is mainly from the time only the original two mbeds, and some with the KL25 also, existed. Targets wise mbed has made a very rapid growth, but it would be good if someone would just spend some time on making that up to date.

All questions are in the central question area which is also here on the homepage. (Well the developers section homepage these days). However people can link them to a program/page when making them, which is why it may happen. But in general it is simply forum for discussions and long questions, questions for questions. Those are the only places. If the search of the site would be a bit better that would be sufficient ;). It would already be alot better if handbook pages were prioritized, followed by cookbook, platform, components pages, and then everything else.

Now finally I am going to say something which is probably going to sound more harsh than intended, however: It is very easy to complain about someone elses code, but where is your own? You have 0 published libraries, programs, components, etc.

16 Nov 2014

Eric, Thank you for your reply. (it took 5 hours for your reply to show up here!) I did spent a significant portion of that time finding and reading the "Getting Started" info that is here. I found a few answers.

Is the proper method to register complaint or request enhancement to create a new topic in the "Bugs & Suggestions" forum?

What little unique code I have succeeded in creating is on my off-line laptop. When I get it tested and documented I plan to publish it. So far, only a device makefile for NUCLEO-F401 in gcc4mbed and a ported library for the ITG3200 3 axis compass to incorporate into another library(HK10DOF) for the 10DOF MEMS from DFRobot, etc. The code is published on my GitHub of the same userID. But is not adequately tested or documented yet.

I'm afraid the boat autopilot that this is to be a part of will take much longer than expected before it is publishable. ;^)

I would like to blame the documentation issues on my slow start and lack of published code, but it is only partially responsible. Due to my lack of internet access I spent a great deal of time trying to find a suitable off-line development scheme to use mbed without the internet. The platforms I chose (picked before I had ever heard of mbed), ST's NUCLEO-F401RE and DISCOVERY429, were not very supported in Aug '14 when I started. I am now using gcc4mbed and local linux utilities to develop and debug off-line and moving the code back and forth as internet availability exists. (living on a sailboat currently in rural Philippines, internet is by cell phone when I am close to a cell tower.)

Thank you, -Larry

16 Nov 2014

For site related bugs/suggestions, yes. For everything related to the official library code the mbed github is used (one of those things they don't bother writing down).

And I guess in your situation an offline tool indeed is pretty much a necessity :).