Improving firmware updates for Internet of Things devices – the IETF SUIT Hackathon in Berlin/Germany
IoT devices commonly require security patches and updates to protect against known vulnerabilities. In this blog post Hannes Tschofenig highlights a recently-launched standardization effort in the Internet Engineering Task Force (IETF): the "Software Updates group for Internet of Things (SUIT)" working group. The report shares experiences from a Hackathon in Berlin, where several working group members met to work on code for firmware updates.
Working Group Formation and IETF London Hackathon
The Internet Engineering Task Force (IETF) met in London from March 17th – 23rd for the first face-to-face meeting of the IETF Software Updates group for Internet of Things (SUIT). The SUIT working group is chartered to develop firmware update solutions that can be implemented into Internet of Things (IoT) devices, especially microcontrollers with limited RAM and flash memory, such as 10 KiB RAM and 100 KiB flash. The focus of the group is simple: since many IoT devices require software updates to fix security vulnerabilities, the group will develop and standardize a secure approach to these updates. For IoT devices, this software update often comes in the form of a monolithic block, where the entire codebase running on the device, i.e. the firmware, is replaced in one shot.
Prior to the meeting, a large number of the IETF participants gathered for a Hackathon at the meeting venue. It has become popular for engineers to hold a Hackathon the weekend before the start of the IETF meetings, to write code for the Internet protocols that will be discussed during the week. The IETF has long been known for a “rough consensus- running code“ mentality. This is a unique attitude meaning that the IETF is interested in practical, working systems that can be quickly implemented. As Charles Eckel, the organizer of the IETF Hackathons, explains in a blog post, one of the goals is to “advance the pace and relevance of IETF standard activities by bringing the speed and collaborative spirit of open-source development into the IETF.”
Engineers also met at the London IETF Hackathon to start implementing their work on the manifest format, i.e. the meta-data about a firmware image. The manifest format offers end-to-end security guarantees, including a security wrapper that is implemented via a recently standardized technology called CBOR Object Signing and Encryption (COSE). Firmware updates for IoT devices are important, but the update mechanism itself also needs to be resilient against attacks to prevent the situation from becoming worse.
A short report about their experience implementing the manifest format at the London IETF Hackathon was provided to the SUIT working group a few days later. The group was fairly small and learned quickly that the availability of an open-source implementation of COSE usable for embedded devices was unsatisfactory. Instead of working on the manifest format, the bootloader and the update process, the group spent their time writing the tools for the foundational security mechanisms. It was clear to the participants that another Hackathon was needed.
We set ourselves the following goals:
- First, we wanted to make sure that the specification was precise enough and easy to understand.
- Second, implementations by different parties needed to be interoperable.
- Third, we wanted to have at least one open-source implementation with product-level quality tailored to resource constrained devices.
During a pub visit at the London Hackathon, I had the chance to talk to Emmanuel Baccelli (Inria), who suggested that we schedule another Hackathon in Berlin before the summer IETF meeting (which will take place in Montreal, Canada). I liked the idea.
Emmanuel also promised to search for a developer at the university, who could construct an open-source COSE library that also supports embedded crypto libraries like Mbed TLS.
Emmanuel and his group were able to release their code, the libcose library, in time for the announcement of the Hackathon, which took place on June 5th and 6th and was hosted by the Freie Universität Berlin. Brendan Moran (Arm) also contributed an updated version of the manifest generator, which was developed for the London Hackathon.
Engineers at the Berlin SUIT Hackathon.
Developing a firmware update mechanism is a specialist task. It requires advanced computer science skills not typical among many developers, including knowledge of microcontrollers, security, and IoT operating systems. Emmanuel and I did not expect many participants. Yet, to our surprise, 20 people expressed an interest in participating, either on-site or remotely. A number of companies, including some of our silicon partners, as well as interested universities were involved. Many people talk about improving IoT device security, and fortunately, there are some actually doing something about it. The famous "Talk is cheap. Show me the code." quote by Linus Torvalds comes to mind. Hence, a big thanks to all the participants!
We offered remote participation opportunities, as some companies are more constrained in their travel budgets than others. Remote participation does, however, create challenges for the organizers and for the remote participants themselves. While we offered a conference bridge with decent video support, trying to follow discussions over a two-day Hackathon was strenuous. We are very interested to hear suggestions for ways to improve remote participation, but in-person attendance clearly has advantages.
The SUIT working group chairs, Russ Housley (Vigil Security), Dave Thaler (Microsoft) and Dave Waltermire (NIST), were very supportive of the Hackathon, and scheduled a conference call for the SUIT working group to take place at the end of the Hackathon. The IETF now calls these official conference calls "virtual interim meetings," and they have been used to report progress on documents and to resolve open issues. In my personal experience, they have helped re-draw the attention of the entire group to unresolved challenges.
Unlike other Hackathons, where the competitive nature of the event and winning a prize is the goal, we kept the Hackathon agenda flexible. Since face-to-face meetings give engineers the opportunity to talk about other related ongoing-activities, we scheduled an official social networking dinner, sponsored by Arm, as well as two informal evening activities. The meeting venue at the university was also ideal for casual conversations, as we had a canteen practically next door that kept us supplied with plenty of coffee.
Regarding hardware, most people brought their own microcontrollers to the Hackathon and this meant that we had a huge range of devices available: nRF52 DKs, various STM32 devices, the Texas Instruments LaunchPad, the Atmel SAM R21 and the Freedom-K64F boards.
There was no shortage of hardware at our meeting.
The participants organized themselves into the following groups according to their interests:
- Mbed OS group, which focused on putting the manifest parser with the libcose implementation into a K64F board. At Arm we already have a product-quality implementation of the manifest format as part of Mbed Cloud, which is based on our initially published contribution to the IETF. Our implementation will have to be updated to match the solution developed by the group, since it is based on an alternative serialization format. For this reason, we are interested in the new CBOR-based serialization format and the COSE security wrapper.
- RIOT group, which extended their bootloader to support various Mbed enabled boards, and implemented the SUIT update mechanism on the Atmel SAM R21 and on an STM32 board.
- Here.com group, which implemented the manifest parser on a Nucleo board.
- Infineon group, which focused on the CDDL-based manifest specification for a minimalistic COSE-signed profile.
- Fraunhofer group, which focused on the retrieval of firmware images and manifests using lightweight CoAP protocol on a Raspberry Pi.
- Nordic group, which focused on the evaluation of the manifest specification.
- Ackl.io group, which used STM32 low power boards for their manifest parsing.
- Linaro group, which looked at the libcose code.
- TZI group, which focused on learning about pain points and discussing the SUIT specifications.
So, what was accomplished during these two days? Of course, everyone had a slightly different experience depending on what topics they worked on. Yet, the following items are noteworthy:
- First, we discussed the firmware update architecture and the manifest information model documents that were adopted by the working group prior to the start of the Hackathon. We also spoke about the manifest specification itself, which was used as the basis for the implementations.
- Second, several groups made use of the libcose library and made updates to the library. The libcose code for integration with the Mbed TLS crypto and Mbed OS is available, as well as the code that integrates with the RIOT OS.
- Third, a first manifest parser (in C) is available (as developed by the RIOT OS guys).
- Forth, the manifest generator has been enhanced, and a tool for converting PEM files to x/y points, as used by libcose and Mbed TLS, is available.
Personally, what I found most rewarding was spending time with all the participants working on firmware updates. I hadn’t met most of them before, and I had the pleasure of learning more about what they are working on. Next time I see mail coming through from one of them, I will now have a face to associate with the name. And I’ll know who to call when I have questions.
A break from coding at the social event.
While working on the implementations, we noticed missing fields in the manifest and various places where additional background text is necessary to better explain the use cases. We also realized that the algorithms used by various groups differed, and because we have not decided on a mandatory-to-implement signing algorithm, we ran into interoperability problems. Currently, three types of algorithms are being considered by the group:
- the Elliptic Curve Digital Signature Algorithm (ECDSA) algorithm with the NIST approved P256r1 curve
- the Edwards-curve Digital Signature Algorithm (EdDSA) Ed25519
- the Hash-based Signature Algorithm
Only the first algorithm is currently an option for Mbed TLS users. Since the RIOT OS team used a different crypto implementation, they implemented Ed25519 instead. Needless to say, the two implementations did not work together. To make matters more complex, the newly proposed hash-based signature algorithm offers better security properties in a post-quantum crypto world, and might be ideally suited for an IoT firmware update environment. Yet, unfortunately, no one at the Hackathon had time to implement or integrate the hash-based signature algorithm into their project. So far, our algorithm discussions have centered around the signature algorithm, but we are looking into supporting encryption as well. Hence, the need for a set of mandatory-to-implement algorithms became clearer.
What’s Next? Montreal IETF Meeting
The Hackathon was pivotal in making progress towards accomplishing our goals, but we are not there yet. So, where do we go from here? The next important date for IETF participants is the upcoming face-to-face meeting in Montreal, from Saturday July 14th through Friday afternoon on July 20th. There will also be a Hackathon in Montreal. We are planning to continue our coding effort at that Hackathon and will even hand out IoT development boards to interested participants. As the milestones on the IETF SUIT working page indicate, the adoption of the manifest serialization format is long overdue, and therefore, needs to happen soon. In the meantime, discussions about the specifications will continue on the SUIT mailing list and during phone conference calls.
If you are new to IoT firmware updates, but want to help to improve the status quo of IoT security, then I would recommend:
- Joining the IETF SUIT mailing list. There are no membership fees for participating in the discussions.
- Reading through the available specifications, including the architecture, the information model and the manifest format.
- Contributing code to one of the above-mentioned projects.