What Is Sensor Fusion? | Understanding Sensor Fusion and Tracking, Part 1

Sensor fusion is an integral part of the design of autonomous systems; things like self-driving cars, RADAR tracking stations, and the Internet of Things all rely on sensor fusion of one sort or another. So the questions I’d like to answer in this video are: "What is sensor fusion and how does it help in the design of autonomous systems?" This will be a good first video if you’re new to the topic because we’re going to go over a broad definition of what it is and then show a few scenarios that illustrate the various ways sensor fusion is used. So I hope you stick around. I’m Brian, and welcome to a MATLAB Tech Talk.

The high-level definition is that sensor fusion is combining two or more data sources in a way that generates a better understanding of the system. “Better” here refers to the solution being more consistent over time, more accurate, and more dependable than it would be with a single data source. For the most part, we can think of data as coming from sensors, and what they’re measuring provides the understanding of the system; for example, things like how fast it’s accelerating, or the distance to some object. But a data source could also be a mathematical model, because as designers we have some knowledge of the physical world and we can encode that knowledge into the fusion algorithm to improve the measurements from the sensors.

To understand why, let’s start with the big picture. Autonomous systems need to interact with the world around them and in order to be successful there are certain capabilities that the system needs to have. We can divide these into four main areas: sense, perceive, plan, and act.

Sense refers to directly measuring the environment with sensors. It’s collecting information from the system and the external world. For a self-driving car, the sensor suite might include RADAR, LIDAR, visible cameras, and many more. But simply gathering data with sensors isn’t good enough, because the system needs to be able to interpret the data and turn it into something that can be understood and acted on by the autonomous system. This is the role of the perceive step: to make sense of, well, the sensed data. For example, let’s say this is an image from a vehicle's camera sensor. The car has to ultimately interpret the blob of pixels as a road with lane lines, and that there’s something off to the side that could be a pedestrian about to cross the street or a stationary mailbox. This level of understanding is critical in order for the system to determine what to do next. This is the planning step, where it figures out what it would like to do and finds a path to get there. Lastly, the system calculates the best actions that get the system to follow that path. This last step is what the controller and the control system is doing.

All right, let’s go back to the perceive step because I want to go into a little more detail here. This step has two different but equally important responsibilities. It’s responsible for self-awareness, which is referred to as localization or positioning—you know, answering questions like where am I, what am I doing, and what state am I in? But it’s also responsible for situational awareness, things like detecting other objects in the environment and tracking them.

So where does sensor fusion come in? Well, it sort of straddles sense and perceive, as it has a hand in both of these capabilities. It’s the process of taking the multiple sensor measurements, combining them, and mixing in additional information from mathematical models with the goal of having a better understanding of the world with which the system can use to plan and act.

With that in mind, let’s walk through four different ways that sensor fusion can help us do a better job at localization and positioning of our own system, as well as detecting and tracking other objects.

All right, for the first scenario, let’s talk about possibly one of the more common reasons for sensor fusion and that’s to increase the quality of the data. We always want to work with data that has less noise, less uncertainty, and fewer deviations from the truth. Just overall nice, clean data.

As a simple example, let’s take a single accelerometer and place it on a stationary table so that it’s only measuring the acceleration due to gravity. If this was a perfect sensor, the output would read a constant 9.81 m/s^2. However, the actual measurement will be noisy - how noisy depends on the quality of the sensor. This is unpredictable noise so we can’t get rid of it through calibration, but we can reduce the overall noise in the signal if we add a second accelerometer and average the two readings. As long as the noise isn’t correlated across the sensors, fusing them together like this reduces the combined noise by a factor of the square root of the number of sensors. So four identical sensors fused together will have half the noise of a single one. In this case, all that makes up this very simple sensor fusion algorithm is an averaging function.

We can also reduce noise by combining measurements from two or more different types of sensors, and this can help if we have to deal with correlated noise sources. For example, let’s say we’re trying to measure the direction your phone is facing relative to north. We could use the phone magnetometer to measure the angle from magnetic north, easy. However, just like with the accelerometer, this sensor measurement will be noisy. And if we want to reduce that noise, then we may be tempted to add a second magnetometer. However, at least some contribution of noise is coming from the moving magnetic fields created by the electronics within the phone itself. This means that every magnetometer will be affected by this correlated noise source and so averaging the sensors won’t remove it.

Two ways to solve this problem are to simply move the sensors away from the corrupting magnetic fields—hard to do with a phone—or to filter the measurement through some form of a low-pass filter, which would add lag and make the measurement less responsive. But another option is to fuse the magnetometer with an angular rate sensor, a gyro. The gyro will be noisy as well, but by using two different sensor types, we’re reducing the likelihood that the noise is correlated and so they can be used to calibrate each other. The basic gist is that if the magnetometer measures a change in the magnetic field, the gyro can be used to confirm if that rotation came from the phone physically moving or if it’s just from noise.

There are several different sensor fusion algorithms that can accomplish this blending, but a Kalman filter is probably one of the more common methods. The interesting thing about Kalman filters is that a mathematical model of the system is already built into the filter. So you’re getting the benefit of fusing together sensor measurements and your knowledge of the physical world. If you want to learn more about Kalman filters, check out the MATLAB Tech Talk video that I’ve linked to in the description.

The second benefit of sensor fusion is that it can increase the reliability of the measurements. An obvious example is that if we have two identical sensors fused together, like we had with the averaged accelerometers, then we have a backup in case one fails. Of course, with this scenario, we lose quality if one sensor fails, but at least we don’t lose the whole measurement. We can also add a third sensor into the mix and the fusion algorithm could vote out the data of any single sensor that is producing a measurement that differs from the other two.

An example here could be using three pitot tubes to have a reliable measure of an aircraft’s air speed. If one breaks or reads incorrectly, then the airspeed is still known using the other two. So duplicating sensors is an effective way to increase reliability; however, we have to be careful of single failure modes that affect all of the sensors at the same time. An aircraft that flies through freezing rain might find that all three pitot tubes freeze up and no amount of voting or sensor fusion will save the measurement.

Again, this is where fusing together sensors that measure different quantities can help the situation. The aircraft could be set up to supplement the air speed measurements from the pitot tubes with an airspeed estimate using GPS and atmospheric wind models. In this case, air speed can still be estimated when the primary sensor suite is unavailable. Again, quality may be reduced, but the airspeed can still be determined, which is important for the safety of the aircraft.

Losing a sensor doesn’t always mean the sensor failed. It could mean that the quantity they’re measuring drops out momentarily. For example, take a RADAR system that is tracking the location of a small boat on the ocean. The RADAR station is sending out a radio signal which reflects off the boat and back and the round-trip travel time, the doppler shift of the signal, and the azimuth and elevation of the tracking station are all combined to estimate the location and range rate of the boat. However, if a larger cargo ship gets between the RADAR station and the smaller boat, then the measurement will shift instantly to the location and range rate of that blocking object.

So in this case, we don’t need an alternate sensor type or a secondary RADAR tracking station to help when the measurement drops out because we can use a model of the physical world. An algorithm could develop a speed and heading model of the object that’s being tracked. And then when the object is out of the RADAR line of sight, the model can take over and make predictions. This of course only works when the object you’re tracking is relatively predictable and you don’t have to rely on your predictions long term. Which is pretty much the case for slow-moving ships.

Okay, the third benefit of sensor fusion is that we can use it to estimate unmeasured states. Now, it’s important to recognize that unmeasured doesn’t mean unmeasurable; it just means that the system doesn’t have a sensor that can directly measure the state we’re interested in.

For example, a visible camera can’t measure the distance to an object in its field of view. A large object far away can take up the same number of pixels as a small but close object. However, we can add a second optical sensor and through sensor fusion, extract three-dimensional information. The fusion algorithm would compare the scene from the two different angles and measure the relative distances between the objects in the two images. So in this way, these two sensors can’t measure distance individually but they can when combined. In the next video, we’ll expand on this concept of using sensors to estimate unmeasured states by showing how we can estimate position using an accelerometer and a gyro.

For now, though, I want to move onto the last benefit I’m going to cover in this video. Sensor fusion can be used to increase the coverage area. Let’s imagine the short-range ultrasonic sensors on a car which are used for parking assist. These are the sensors that are measuring the distance to nearby objects, like other parked cars and the curb, to let you know when you’re close to impact. Each individual sensor may only have a range of a few feet and a narrow field of view. Therefore, if the car needs to have full coverage on all four sides, additional sensors need to be added and the measurements fused together to produce a larger total field of view. Now, more than likely, these measurements won’t be averaged or combined mathematically in any way since it’s usually helpful to know which sensor is registering an object so that you have an idea of where that object is relative to the car. But the algorithm that pulls all of these sensors together into one coherent system is still a form of sensor fusion.

So hopefully you can start to see that there are a lot of different ways to do sensor fusion, and even though the methods don’t necessarily share common algorithms or even have the same design objective, the general idea behind them is ubiquitous: Use multiple data sources to improve measurement quality, reliability, and coverage, as well as be able to estimate states that aren’t measured directly. The fact that sensor fusion has this broad appeal across completely different types of autonomous systems is what makes it an interesting and rewarding topic to learn.

In the next two videos we’re going to go into more detail on sensor fusion for localization and for multi-object tracking. In the next video in particular, we’ll show how we can combine an accelerometer, magnetometer, and a gyro to estimate orientation.

So if you don’t want to miss that and future Tech Talk videos, don’t forget to subscribe to this channel. Also, if you want to check out my channel, Control System Lectures, I cover more control topics there as well. I’ll see you next time.