Unmanned Aerial Vehicles using MATLAB and Simulink

Good afternoon, everyone. Welcome to this webinar on unmanned aerial vehicle design using MATLAB and Simulink. This is Naga Pemmaraju. I'm a senior application engineer with MathWorks India, mainly focused in the areas of [INAUDIBLE], modeling simulation controls, and real-time simulations. In today's webinar, I'll be walking you through how MATLAB and Simulink can be used for designing an unmanned aerial vehicle, and especially, I'll be focusing on some of the new capabilities that MathWorks has built in this area, especially with this related to UAV Toolbox. So let's start.

So we all know the UAV's development was, in the research phase, quite a long time. And you can see from this video, this is a video from one of the universities where they're trying to have these UAVs or drones building a small structure within a restricted area. As you can also see, the net, so that it doesn't go beyond that.

But this was in the beginning, or in that evaluation phase. But now, this has become quite mainstream, right? So we can see a lot of this in the news. Now, we have seen how we UAVs or drones have been using for agricultural purpose, and we know a few of the companies are working on [? open-air ?] mobility. And also, some of these are used for cargo delivery. And last but not least, mainly used in the military purpose.

Here are some applications where the UAVs are used, mainly for security and surveillance in the defense area. For some of this, search and rescue, in the consumer area, entertainment. I know one of the popular thing that is we see in India is the photography and filmography. Of course, the education and R&D is still a hotbed. We're trying to explore the usage of UAVs for various different applications, especially in the environmental research and wildlife monitoring. And a lot of other commercial applications like inventory management, oil and gas, construction, just to name a few.

I just want to touch base upon a few user stories, how MathWorks or MATLAB and Simulink have been used by some of our customers in building the UAVs. Here is one example from avionics that have used model-based design approach for developing their unmanned aerial camera motion system. One of the results, as you can see from the results, is they've been able to reduce their development time by over one year, and also to eliminate all the coding errors.

Here is one other example from Korean Air speeds, who developed the flight control software for UAVs. Again, the results are pretty much similar. And Intel used MATLAB tools, MATLAB and Simulink tools, for testing the GNC algorithms for the multi-rotor UAVs.

It's quite evident that MATLAB and Simulink are tools for simulation, and people are making use of these tools to test their design and do all the verification, validation, even before they get onto the prototype. And there is a lot of advantage in doing these simulations.

Here is one example I want to touch base upon where and how MATLAB and Simulink are used. You can see, this is an example of a UAV package delivery system. And the top one, what you're seeing is a Simulink model of how we have built the UAVs in MATLAB and Simulink, and the bottom one is the virtual realistic scenario on an Unreal gaming engine.

So this is in sync with the Simulink, so it is integrated with Simulink. And you can actually visualize how the UAV is actually flying within the bounds of-- there, you have a lot of these buildings, and how it's flying over or in between them. On the right side, we have simulated the LIDAR, and also the camera. And the output-- these two pictures are basically the outputs of those kind of camera, the bottom one being how the camera on the UAV is actually seeing the scenario in front of it.

So let's see how we can actually develop this and how MATLAB and Simulink can be used for designing the UAVs. I'll just talk about some of the steps that we followed, and the first one being first. And the first thing, we need to understand the flight dynamics of the UAV. Because it is flying, we need to ensure, what are the flight dynamics involved for the UAVs, right? Then, once we have this cleared, we focus on the autonomous algorithms. It can be related to the perception, planning, and controls, and waypoint navigation, or a lot of other things which fall under this bucket of autonomous algorithms.

Once we build these algorithms, we need to test them in a simulated world. And in this case, we need to ensure we tested more closer to the real world and we make use of this virtual environment-- in this case, an Unreal gaming engine-- when we can also simulate some of the sensor behaviors. Once we tested the algorithms and we have more fair amount of confidence, we will build our algorithms and deploy them onto the hardware. In this case, it's a Pixhawk PX4 hardware. And if you're talking about autopilot, it will be Pixhawk PX4. Or if you're talking about some other algorithms, it could be anything on the available onboard computer.

Finally, once test it, and deploy that, and you have this up and running, the last but also very important step that people follow in this is, how do we perform the flight data analysis? And this kind of data you're constantly getting in. It's late. There's a lot of data, I know, from various sensors. You have the telemetry data. And how do we make sense of that? And how MATLAB can be used in that particular case?

So in today's webinar, we will be focusing on all of these aspects of the UAV design. Before jumping into this, I have a poll question for you. Which area of UAV design are you working or are you interested in? You'll be getting a poll question on the right side, as you can see on my screen here.

You have five options. The first one, if you are interested in the modeling and simulation of the flight dynamics, or are you actually working on some autopilot or autonomous algorithms for your UAVs? Are you interested in the visualization or integration in the virtual environments like the one I showed you, which is related to the Unreal gaming engine?

The last one, the flight data analysis-- are you working on it, or is it of your interest? Last one, it may not be anything. You're just here to explore some of these capabilities.

Please answer this question, and it will be on your WebEx window on your right side. And this will be there for another 30 seconds. I will move on, in the interest of time.

So here is an example of a simulation model of the UAV, Unmanned Aerial Vehicle, wherein-- let me you know talk about what are these three different items. The leftmost subsystem, what you see is a ground station. The middle most subsystem is the autopilot, wherein we have considered the lateral and longitudinal controls and all of that. And the third one, being the actual plan model, wherein we are trying to model the dynamics of it.

If we look into inside this plan model, there are basically-- again, we have multiple subsystems where first, we need to capture the dynamics of the aircraft subsystem. Then, whatever the actuators or the motors available, we are trying to model that actuators and motors. And we also need to consider the weather, because now that the aircraft is, or UAV is flying, we need to ensure what happens if there is a wind, gust in the wind, and how does it impact my the flying of my UAV?

And we also need to ensure we have some sensor models available that can actually capture the right amount of details, and we can give that as an information back to our controllers, right? So as you can see from here, we're getting the GPS data and raw sensor data, which will be given back to our controller autopilot so that they can act accordingly.

If you further get into the aircraft dynamics, we need to model the aerodynamics forces and movements from various-- we will be capturing various forces and movements that are acting on this flight. And also, we need to consider the six degrees of freedom of this particular aircraft. And we'll be looking into the equations of motions for the six degrees of freedom.

Again, this is a block available as part of the aerospace blockset, wherein you can use all these blocks out of the box, out of the aerospace blockset. All you need to enter is just given the initial position of the various axes, the velocity information, the body rotation, mass, and inertia.

All right, so here is a simulation of how it works. I have not touched upon the autopilot, but I'll try to come back a little later. In this particular case, as you can see, here is one application of a waypoint navigation. We have about five waypoints, and we're trying to track a UAV which is trying to fly across all these waypoints, roughly. And this is what you can do in the simulation world. And the green one being the base station from which it has started-- the UAV started flying.

Great. So as I said, we have used the aerospace blockset for this, modeling the dynamics of the aircraft, or, in, this particular case, the UAV. It's an extension to the Simulink, whereby we are providing some cool vehicle functionality, like various kinds of equations of motion, some of the propulsion systems, actuators, and the pilot models, GNCs, environmental models, some of the aerodynamics, et cetera. So if some of you are new to this, I would encourage you to explore this so that you can have a very good high-fidelity model of your system.

Now, as I said, we have seen the aerospace tool box and blockset which we have used for flight dynamics. But then, the rest of the steps, we'll be using a UAV Toolbox, a tool box that was recently released in MATLAB-- 2020b version, that is, that got released a couple of months back. Let's look at some of these capabilities and how these capabilities are used in designing the UAVs.

As I said, UAV Toolbox was released in 2020b. It provides various algorithms for planning and control. It provides connectivity to a MAVLink and Pixhawk PX4 hardware. You can actually do some [INAUDIBLE] simulation with, also, very high fidelity, and, of course, simulation with Unreal gaming engine, which also provide some sensor models.

We can perform some flight data analysis using this toolbox. We provide a shipping, a flight log analyzer app as a part of this toolbox which eases your analysis of it in MATLAB and Simulink. And last, but not the least, we do provides shipping examples and reference applications in this particular toolbox. The one I showed you in the beginning is from this reference application. I will go into a little bit of detail about this by the end of the webinar.

All right, so let's look at the portion on how we can develop these autonomous UAV applications. So there are four different subsets, four different categories of which I have put everything to [INAUDIBLE] about the UAV algorithms, the second one being the scenario simulation-- how can we simulate the scenarios? Third one being connectivity and deployment, wherein, how do we access, how do we use the MAVLink protocol and connect to the various autopilots, especially in this particular case? Pixhawk PX4, and then, lastly, but not the least, how can we perform the flight data analysis?

Let's look at the first look, the UAV algorithms. Basically, UAV provides a set of guidance models. Some of these UAV algorithms, like waypoint navigation, orbit follower, which helps you to quickly assemble the model. And then, such algorithms can be put together, and you can actually do a quick prototype design and see how your design, how your concept is working.

The guidance models here we provide are related to both fixed-wing, as well as the multi-order wing. They're easier to configure. They can be used develop very high-level navigation and motion planning systems.

As I said, in the beginning, we have shown you how to model dynamics using the aerospace block set. That is a very high-fidelity model. But in case if you do not have all such details, you can still do with the blocks available from the UAV. These blocks gives you a low-fidelity model, but still, it will be helpful for you in designing your algorithm development and also seeing a complete system-level simulation, perform system-level simulation and understand how your model is working, and also your design is working. We can actually tune the guidance models to approximate the high-fidelity model. So you do not have all the details, but still we can fine-tune these guidance models so that it can behave close enough to the high-fidelity models.

With respect to the motion planning, as I said, we provides some of the blocks out of the library for orbit follower, waypoint follower, and path manager, which helps you to create and test your motion planning algorithms. Here is a model which we have developed using these low-fidelity blocks. As you can see from here, we have a UAV waypoint follower. And this, you can actually choose whether it is a fixed-wing or a multi-rotor wing. And similarly, whether it is for the co-generation, or are you just with the simulation purpose? Here, we have some heading control where we have used the MATLAB functions, and some of the-- written a small script to calculate the angles here.

And the UAV guidance model, again, provides the kinematics of the system. But then, again, here, also, we have an option to choose whether it's a fixed-wing or a multi-rotor. Actually, we can provide all of the initialized states and some of the configuration here, which makes it easy for us to get started with our model. And last, but not least, we also provide one block for UAV animation, which helps us to visualize how our algorithms are working in the simulation world.

So once you set this model, once you run this, once you run this [INAUDIBLE] model, you can actually visualize this using the UAV animation, which is available as a part of the UAV Toolbox. Here, I actually have speeded up this video, just in the interest of time. So as you can see here, you can actually visualize the animation of how it is going in all the three axes, both x, y, and z. And whenever it would be taking a turn, you can actually see the turn and how it is being impacted-- whether it is following the trajectory or not. So you can visualize it here.

When used with a Navigation Toolbox-- so again, a Navigation Toolbox, another toolbox from MathWorks-- people can design and simulate an autonomous motion planning algorithm, because they want to get in some more details, some more inbuilt details. We can use that. We also provide this RRT path planning algorithms. Basically, these had used to work with this kind of [INAUDIBLE] constraints, and we can actually understand if this can be-- if you can avoid obstacles that are used. But I know path-following, and also with all of these obstacles, and see whether we are-- and our flight is shorter and smoother, et cetera.

Coming back to the scenario simulation, this is one of the important one steps, I would say, because once you develop your algorithms, we need to test this in simulation so that we can understand how our design is working. And for this, we provide a rich set of features as a part of this UAV Toolbox. The first one, where we can quickly create a scenario [? of a world ?] to generate various test cases and send sensor data information from the environment.

So this can be either 2D or even a 3D. We can define and visualize the trajectories here. And as I said, we can quickly test UAV autonomous algorithm in a cuboid simulation. So you can create this scenario to simulate flights between a set of buildings. Just click here, turn on my video.

Here, as you see from the video, basically, we are trying to test an open loop simulation where the UAV is flying between the blocks, and now we are trying to simulate the flight with GPS, INS, and LIDAR sensor models. So these scenarios can be used to generate test cases and also validate some of these autonomous algorithms, such as obstacle avoidance, that matter.

And as we've seen in the beginning, we can alternatively use a course simulator designed with pre-generated US city environments, or any other scenarios. And the good part is you can also create your own custom scenic scenarios using Unreal Engine Editor, wherein you can also synthesize the sensor data like INS, GPS, LIDAR, and other camera sensors.

We actually provide a support package which can help you to customize these things. The blocks you can see at the top are actually shipped with a part of the UAV Toolbox-- like, for example, simulating a 3D camera, or a 3D LIDAR, or 3D scene configuration. You can use depth visualization, an use depth visualizations to validate depth estimation algorithms for your sensors. You can use semantic segmentation visualizations to analyze the classification scheme used for generating synthetic semantic segmentation data from the Unreal Engine.

These are the platforms for you to do co-simulation with Gazebo. A lot of our customers from the education or academia use Gazebo, which interacts with Simulink via ROS. As you can see from here, we have a model in Simulink. We are trying to send signals and visualize them on Gazebo environment.

And here, you can see the UAV flying, where in the algorithms, it basically detects a stop sign from the algorithm we put there. And then it stops, and then moves ahead-- or, sorry, moves on the side. So we can actually create such kind of scenarios and then also begin to see some of the sensor data from here, and we actually actuate all the model links with-- and joints in Gazebo simulator.

Now let's shift our gaze to the connectivity. One that we working algorithms which have been tested in the simulation world, now we need to validate this and validate this simulation. We need to deploy this either into an autopilot or into an onboard computer.

To provide that, we provide support for MAVLink. Basically, MAVLink is a communication protocol that enables a message exchange between a ground control station, autopilot, and onboard computer. So we have to see-- we have seen how important this is, and so we have used-- we have given this interface as a part of MATLAB and Simulink so that you can actually work with multiple other visualization tools and also talk to various autopilot and onboard computers.

Here is one simulation. What we have done here is a waypoint navigator algorithm, what we have done in MATLAB Simulink. Again, this is something similar to what you have seen in my earlier slide of the video here. You can see, we have created some nine waypoints, and on this video here, as you can see, maybe there's some issue. Sorry about that.

But here, you can actually visualize how my UAV-- or the red one, which represents a UAV-- can actually go to all these waypoints, which is in co-simulation with something. So all the commands are coming from the MATLAB and Simulink, and on the fly, you can actually change some of the settings in the [INAUDIBLE] ground control, which is given back to MATLAB and Simulink.

For example, if I click on the Continue Mission, or if I want to switch to an orbit guidance, I can get this command back to my algorithm which is running in Simulink, and accordingly, my algorithm will change and will give the right commands back to the visualization tool.

We provide extensive support for PX4. As of the first release, we started with support for PX4. We plan to support other autopilots as well.

And here is a kind of an architecture of how we can actually test our autopilot design. So you make your models in MATLAB and Simulink. You build and deploy them and run them on the host target. But you can constantly connect with Simulink with the UDP protocol.

And this host target is actually connected to the [INAUDIBLE] ground control. And also, you have a JMAVSim, which both are talking on the UDP protocols. And you can also connect your joystick to actually give the commands and see how your UAV is flying.

We also provide support for Jetson boards, for example, with the Simulink Coder Support Package. I know for everyday NVIDIA Jetson, you can actually integrate-- as you can see from the pictures here, you can connect the Pixhawk with the Jetson board and see how you can perform the complete simulation. So this is how-- autopilot versus your Jetson board, where your autonomous algorithms are working.

So here is a flight log analyzer app, which we ship along with the UAV Toolbox, where, then, you can load either ULOG, or TLOG, or even custom file types of your data, and then you can visualize and play back all this different flight telemetry data. You can also make use of some of our signal processing capabilities to analyze the signals coming from and also apply some of these techniques and understand better of the data that's coming from the UAVs.

I just want to quickly try the video which I want to play. Let me see if I can play this. Yes, this is a video of how-- I was talking about this particular environment, simulation environment, wherein we are connecting to Simulink with Pixhawk PX4 target, and also with jmap sem. And let's see that in a video here.

So here is my model, wherein we have this altitude and position tracking algorithms running in Simulink. And instead of having a joystick, we have made some dashboard within the Simulink itself. So what you see, the trackers and some of the changes what you've seen, the parameters, these are all from the Simulink dashboard, which we use. But with this, you can actually visualize how you can integrate and, basically, build algorithms for Pixhawk PX4 and then visualize this.

So here, I am actually changing the values of my yaw, and also my altitudes, and the directions. And you can see, on the right side, how in the simulation where my UAV is responding to the commands that are [INAUDIBLE] Simulink.

So this could be a very good way of prototyping your design-- your design, your algorithms-- and directly deploying them onto the UAVs, and understand how these are behaving. You can quickly test it in this kind of emulated world, and then try to see whether the algorithms are working as expected or not.

Moving on. This we have seen. I'll come back to the last part, which is a UAV package delivery. So this is what we have started, beginning with.

This is a Simulink model. As a part of the UAV Toolbox, we provide reference applications. And in this first release, we have given a UAV package delivery example, when we have a UAV model. I'll walk you through this now. And then, we have an Unreal Gaming Engine visualization aspect. And on the right side, you can see the sensor with outputs.

So let me go back to the model. So here is a model. Here is a model of the UAV package delivery. And in this particular case, as you can see, we have the ground control station. We have some external sensors, the onboard computer, and the multi-rotor.

So if I get into the multi-rotor, let me navigate this. So I hope you can see my full screen. So let's start with the multi-rotor, right? If I just go into the multi-rotor, we basically have the guidance logic, and we have the high [INAUDIBLE] [? plant ?] model. So if I get into the [? plant and ?] [? plan ?] model, I do have an option of whether it's a high-fidelity model or a low-fidelity model.

If I go to the high-fidelity model, again, here, we have the controls plus the [? plan ?] model. But I want to focus on the plan wherein we have the six degrees of freedom, all of this which I've shown earlier has been taken care.

Now, if I am not interested in this high fidelity because of various reasons, if I am interested in the low-fidelity model, I know it's carried it out, but basically, we have used the blocks from the UAV Toolbox, which is now-- for example, in this case, it's a guidance model wherein you can choose what kind of a model you wanted-- a multi-rotor or a fixed-wing-- and provide all the initial state and configuration parameters.

So again, once I go back, let me focus on the onboard computer. Again, this is mainly used for processing the various data that's coming from various sensors, in this particular case. And if I go to the simulation environment, again, I have multiple options, but I'll be interested in the probabilistic scenario of the quadcopter. And as you can see, again, we have a 3D simulation environment, wherein we provide a block for configuration, simulation 3D scene configuration, where I can choose what kind of a scene I want and all of those. I can set the basic configuration.

Then I have a UAV vehicle I can actually configure. I use two sensors. One is a camera sensor; the other one is a LIDAR sensor. For example, if I get into the camera sensor, I can actually see where am I actually mounting this, and what is the relative translation and rotational values, and what are some of the parameters that I'm interested in from the camera perspective-- like the focal length, the image size, optical sensor, et cetera, and some of the ground root capabilities.

Similarly, with respect to the LIDAR, we provide, again, where, exactly, they came out and some of these parameter details-- like OK, what is my range of this LIDAR, resolutions, the vertical and horizontal field of view, and the resolutions again. So all this information would be helpful for us to process the data that's coming from the LIDAR.

So once we get this data from the sensors, we try to send it to the processing, which is happening on my onboard computer. And ultimately, it's giving commands to the multi-rotor. And ground control station is just a simple flight mission which we have used. If you're trying to work with, say, a simulators, especially the QGroundControl, we can actually set some of these constraints here.

So let me run my simulation here. Right now, it's compiling. So now, you should see a window pop up for the Unreal environment. And I'm actually integrating with Unreal environment here. And then, you will also see a couple of windows which showcases the camera outputs coming from there. There you go. We have this visualization window, SKS again. I think it's-- there you go.

So we have visualization from the Unreal Gaming Engine, and all these commands are coming from the UAV. And you can see, my simulation is running down. My Simulink model is running, and all these commands are being given to the UAV that's in the Unreal Engine. Right, so I'm going to stop. Now, let me stop this simulation.

Let me hop back to my slides. So we have seen how various aspects of this design that can be within aspects of the design using MATLAB Simulink. If I just recall, most of these capabilities have from UAV Toolbox. And we have briefly touched on the Aerospace Toolbox in the beginning.

But having said that, there are a lot of other capabilities from other toolboxes from MathWorks. For example, I've just used Aerospace Toolbox for my flight dynamics. But if I want to have the real structural details-- for example, I have a CAD model of the UAV or the drone-- I can import that AeroCAD model into Simulink and get those capabilities.

So what we called it is a physical modeling, which we use Simscapes and [? scape multi, ?] Simscape Multibody, and Simscape Electrical if you want to model the motors of it, right? For the algorithmic part, we have the Navigation Toolbox and the sensor fusion and tracking. Of course, Computer Vision System Toolbox is also available.

For us to integrate with third-party simulators, we are providing Unreal with the UAV Toolbox. But along with that, for example, if you want to do the Gazebo [INAUDIBLE] simulation, we can make use of the Robotic System Toolbox, and we also provide some of the sensors from-- if you want more details, we are providing a LIDAR Toolbox, and for communication, we have a a ROS Toolbox.

For the deployment capabilities, we provide support for all the MATLAB and Simulink models, where we can generate code for any of the embedded targets, be it autopilot like the Pixhawk 4, or even onboard computer using the [? embedded ?] [? coder. ?] If some of your algorithms are targeted for any of the FPGAs or the GPUs, we actually also provide quotas for that, wherein we can automatically generate the quote from the Simulink models. Last but not the least, we have seen how we can perform flight data analysis using the app that is shipped as a part of the UAV Toolbox.

But along with that, I would encourage you to also make use lot of other capabilities of the base MATLAB, as well as some of the advanced capabilities in terms of machine learning and deep learning. So we have capabilities in all of this, and we can use these capabilities here. Here are some resources I would encourage you to-- feel free to reach out to us if you have any questions. We'd be happy to work with you. And if you're working on some prototypes where you need expertise from us, we're happy to talk to you and support you in that. All right, thank you for joining this webinar.