We are excited to announce uTensor is joining forces with TensorFlow as part of the new TensorFlow Lite Micro project as announced today at Google I/O.

By Neil Tan, Pete Warden, Zach Shelby

uTensor was one of the first open source frameworks to bring machine learning onto microcontrollers. uTensor converts machine learning models to readable and self-contained C++ source files, to simplify the integration with any embedded project. It is especially designed for low-power, constrained embedded devices, and it has deep roots in TensorFlow and MbedOS. This merger is a great opportunity for us to bring uTensor’s innovations to TensorFlow and ensure it is easy for all developers to use and support a wide range of Arm Cortex-M hardware. All of us share a common vision to bring machine learning to the edge. We are looking forward to creating a state-of-the-art micro-inference framework together.

A micro-inference engine should be developed alongside a training framework. On resource-constrained devices based on micro-controllers, every bit of computational resource matters. The technologies used in both uTensor and TensorFlow Lite Micro such as FlatBuffer, micro-interpreter, quantization, SIMD, graph-rewriting, and code-generation have made neural-network deployment possible on MCUs. However, we can do a lot better by interacting with training pipelines. This opens additional possibilities for:

• Quantization-Aware Training
• Offline Execution-Planning
• Target-Aware Hyper-Parameter Tuning
• Sparsity and Lossy Compression
• Embedded optimized partial code generation

These techniques are best realized with the support of the training pipeline. We are looking forward to explore this potential with the TensorFlow team. Together, we can push the performance and efficiency of micro-inference engines beyond what can be achieved today.

Machine learning algorithms are evolving at lightning speed. New neural network accelerators are being introduced constantly. uTensor is developed to adapt to this changing landscape. With minimal code, developers can train and deploy TensorFlow models on a wide range of hardware supported by Mbed. It has features designed to be forward compatible with advances in embedded systems:

• Editable C++ model implementations generated from trained model files
• Extensibility for Tensorflow Lite files, RTL for FPGAs, MLIR, and, other generations
• The ability to place tensors in various memory devices
• Extensibility for optimized kernels, discrete accelerators, and remote services
• Python offline-optimization-tools to enable target-specific and data-driven optimization

These features have already helped us to integrate Arm’s CMSIS-NN with uTensor. This library helps neural networks to run faster and more energy-efficiently on Cortex-M MCUs. With the recent announcement of Helium vector extension for Cortex-M, we will see a new generation of computational kernels to come. We hope this work will inspire a whole generation of machine learning designs in TensorFlow Lite Micro.

The merger is also great news for the edge computing community. TensorFlow is Google’s widely used framework for machine learning. uTensor has a strong connection to Mbed’s communities, allowing us to bring ML to 100s of Mbed hardware targets and the 350k+ Mbed developer community. They each represent data science and embedded engineering communities in their own rights. By establishing this collaboration, we are bringing the people together, allowing us to gather more requirements, iterate faster and deliver better software to everyone. After a year of developing uTensor, we know it is extremely rewarding to work with experts from both sides.

The ability to deploy machine learning models on edge devices opens doors for smarter devices that are power- and bandwidth-efficient. We believe this will usher in an era of TinyML battery-powered devices that can take advantage of low-power wireless communication such as LPWANs. As discussed in Pete’s blog Why the Future of Deep Learning is Tiny, reducing the energy-cost of inferencing will introduce AI to new applications. In addition, it will improve privacy and cost efficiency of systems. Neil also has shared Why Machine Learning on the Edge is important. A widely supported open source micro-inference engine is a critical piece of technology to make AI ubiquitous.

Both uTensor and TensorFlow Lite Micro are at their early stages. uTensor is great for rapid prototyping. All the experimental and conceptual features are being matured in uTensor. We will share the same kernels, file-formats and optimization routines with Tensorflow Lite Micro. Over time, we will be merging the most useful features into TensorFlow Lite Micro. In the meantime, we are establishing the collaboration between the teams, and will transition to work on the same codebase. Stay tuned!

Further Resources

uTensor repository

TFLM documentation

TFLM repository

Mbed community

CMSIS-NN resources

Special thanks to Kazami Hsieh, Dboy Liao, Michael Bartling, Jan Jongboom, Austin Blackstone, Yue Zhao and other open-source contributors for their passions and contributions towards the uTensor project.