Mercedes-Benz Simulates Hardware Sensors with Deep Neural Networks
“This was the first time we were simulating sensors with neural networks on one of our powertrain ECUs. Without MATLAB and Simulink, we would have to use a tedious manual coding process that was very slow and error-prone.”
Challenge
Simulate automotive hardware sensors with deep neural networks.
Solution
Use MATLAB, Simulink, Deep Learning Toolbox, and Fixed-Point Designer to convert QKeras deep learning models into code that can be deployed to an automotive ECU.
Results
- CPU, memory, and performance requirements met
- Flexible process established
- Development speed increased 600%
Automated workflow for deploying virtual sensors to powertrain ECU.
Many automotive manufacturers equip their development cars with numerous additional hardware sensors to gather data for diagnosing issues and refining the design. Once the car transitions to production, many of these sensors are removed to reduce costs. In some cases, however, the hardware sensors can be replaced with virtual sensors, which are cheaper software alternatives that can further improve safety, efficiency, and driver comfort.
Mercedes-Benz recently used MATLAB® and Simulink® to establish a new workflow for deploying virtual sensors, such as those that simulate the functionality of a piston pressure sensor. These sensors are based on deep learning networks designed to run on resource-limited ECU microcontrollers. This automated workflow replaced a manual workflow that was both slower and relied on a trial-and-error approach.
“The MathWorks team helped us develop an easy-to-use pipeline for creating and integrating neural networks into our vehicle controller units,” says Katja Deuschl, AI developer at Mercedes-Benz. “With this pipeline, we are now able to create and deploy different types of neural networks for virtual sensors and a variety of other applications.”
Challenge
Most deep neural networks are designed to run on computers with significantly more processing power and memory capacity than an automotive ECU. Moreover, ECUs don’t support commonly used deep learning frameworks—such as TensorFlow™ and PyTorch®—or the floating-point operations they require.
To run models based on these frameworks on an ECU, developers must first translate the Python® code to C and transform the model parameters and calculations into fixed point operations. This process took Mercedes-Benz weeks to complete, produced unreliable results, and was difficult to teach to new team members.
As they worked through the many manual steps needed to translate the model from Python, the team had to ensure that the model implemented in C would fit within the limited memory space of the ECU and would run fast enough to provide real-time inference for time-sensitive operations such as sensing piston pressure. The team wanted to establish a more reliable and more automated process.
Solution
The Mercedes-Benz team worked with MathWorks engineers to implement an optimized workflow to convert Python deep learning models into code and parameters that can be transferred to their ECU integration pipeline.
In this workflow, the Mercedes-Benz team trains a quantized long short-term memory (LSTM) neural network in Python using the QKeras library.
They use Deep Learning Toolbox™ to import the trained network into MATLAB. They then run a custom MATLAB script to convert the imported neural network into a Simulink model.
Next, the team uses Fixed-Point Designer™ to convert all parameters in the model from floating-point to fixed-point data types in preparation for deployment to an automotive ECU.
After verifying the fixed-point model via simulation in Simulink, the team hands the model off to a third-party software integrator, who implements it on the ECU with other software components.
Having successfully implemented its first use case—a virtual sensor for piston pressure—with this workflow, the Mercedes-Benz team is now applying the same process to additional sensors and other deep learning applications for ECU targets.
Results
- CPU, memory, and performance requirements met. “With MATLAB and Simulink, we implemented a virtual sensor neural network that fits on the ECU and meets our benchmark requirements,” Deuschl says. “We would not have been able to produce a similar virtual sensor with classic software development.”
- Flexible process established. “We are already using the automated workflow we created with MATLAB and Simulink for other use cases,” says Deuschl. “We made small adaptations to support deployment on two different powertrain controllers, and the workflow is also applicable to other types of deep learning models such as gated recurrent units and fully-connected neural networks.”
- Development speed increased 600%. “With Deep Learning Toolbox and Fixed-Point Designer, we improved development speed by about six times compared to our previous manual development process,” Deuschl says. “And with less manual development, we committed fewer errors in creating the model and the code.”
