Enhancing Hands-On, Remote, and Virtual Control Engineering Laboratories with Digital Twins and Extended Reality at Tallinn University of Technology
By Saleh Alsaleh and Aleksei Tepljakov, Tallinn University of Technology
Control design is a fundamental skill in a broad range of engineering disciplines, including mechanical, electrical, fluid, chemical, and biological. As such, it has become a staple of the core curriculum in engineering programs worldwide. As with many engineering topics, instructors have long recognized the need for hands-on lab work and experimentation that can link the theoretical concepts students learn—in various areas of engineering, including control theory—with their practical application.
While in-person, hands-on labs engage students and enable them to develop a deeper, more intuitive understanding of control concepts, they also have drawbacks. First, they don’t scale well; lab setups—cranes, fluid-filled tanks, or robotic arms, for example—often require a sizeable work area. They can also be costly to acquire and maintain. As a result, many departments don’t have enough setups to support growing class sizes, resulting in scheduling hassles and long wait times for students. Second, in-person labs are not feasible for remote students, a pain point that was made clear during the pandemic.
At Tallinn University of Technology, our team has developed the ReImagine Lab framework to address these challenges. The framework combines digital twins built using MATLAB® and Simulink® with extended reality technologies to streamline the development and operation of hands-on, virtual, and remote laboratory setups (Figure 1). In contrast to remote labs, which make use of digital twins that are synchronized with laboratory equipment in real time, virtual labs use digital twins independently of the actual equipment. As such, virtual labs enable greater scalability to support large groups of students (particularly when the virtual environment is hosted in the cloud). Importantly, extended reality—supported by either virtual reality or augmented reality devices—provides students with an immersive, engaging experience in which they can visualize and interact with the digital twin (Figure 2).
Figure 1. The ReImagine Lab framework includes a physical lab asset, digital twins of that asset, and extended reality devices.
Figure 2. Augmented reality view of a tank system and its digital twin.
ReImagine Lab Framework Basics
At the heart of the ReImagine Lab framework is the digital twin, which is based on a data-driven mathematical model of a real laboratory asset (such as a gantry crane, for instance). To create the underlying mathematical models, we use a black-box approach, in which experimental data is fitted to the model. Alternatively, we can also use a gray-box approach, where the model is partially derived from physical laws and represented as a system of ordinary differential equations (ODEs) with parameters that are then estimated from experimental data (Figure 3). For both approaches, we use MATLAB and System Identification Toolbox™ to define the model and perform parameter estimation based on data we collect from the real system.
Figure 3. Black-box and grey-box approaches used to create a digital twin.
We integrate the identified mathematical models into Simulink with Instrument Control Toolbox™ blocks for real-time data communication via User Datagram Protocol (UDP). This enables communication with a 3D model of the asset in Unreal Engine® for visualization in an extended reality environment (Figure 4).
Figure 4. UDP communication between the MATLAB and Simulink environment and Unreal Engine.
An Example Instructional Use Case
The best way to understand how the ReImagine Lab framework works is in the context of an example use case that could be implemented with the framework.
In this example, student teams are tasked with developing a control system for a gantry crane. Also known as a 3D crane, this apparatus includes a movable rail, a cart that moves along the rail, and a payload suspended from the cart (Figure 5). Two encoder sensors are used to determine the position of the rail and cart, and two additional sensors measure the swing angle of the payload. The students’ objective is to design a controller to move the payload to a desired location as quickly as possible, while minimizing the swing angles.
Figure 5. (Left) A real gantry crane, (center) a physical model diagram of the crane, and (right) a VR-based representation of the crane.
For the first part of the lab assignment, each student team creates their own mathematical model of the crane. They begin by taking measurements of the crane (including its swing angles) as it moves. One team may work on the actual crane, while others take measurements interacting with a digital twin in a remote environment or independently in a fully virtual environment (Figure 6). Based on these measurements, the teams create plant models of the crane using MATLAB and System Identification Toolbox. We have the students create linear models instead of more complex nonlinear dynamics models while helping them understand the tradeoffs involved between simplicity and accuracy.
Once the students have created a plant model, they build a controller model using Simulink and Simulink Control Design™. The controller model typically contains four proportional, integral, and derivative (PID) loops: one for the rail position, one for the cart position, and two for the swing angles (Figure 6).
Figure 6. Simulink controller model for the gantry crane.
Students run closed-loop simulations in Simulink to test and tune their controllers. They then test them against the digital twin, interacting with the simulation in the virtual reality environment. When the student teams that have been working remotely or virtually have access to the real crane, they see how their designs perform on the lab equipment by using Simulink Desktop Real-Time™ to run their Simulink controller models in real time on a PC connected to the crane’s sensors and actuators.
It is important to emphasize that the ReImagine Lab framework is not limited to just a 3D crane in virtual reality. We have built a variety of digital twins, using both augmented reality and virtual reality (Figure 7). Students interact with the tank digital twin shown in Figure 2, for example, using a Microsoft® HoloLens headset. In the augmented reality environment, students see the physical system alongside the digital twin, a diagram of the control loop, and graphs that are updated in real time (Figure 8). In fact, students can adjust controller gains directly in the environment by resizing virtual spheres with their hands (Figure 9).
Figure 7. Interacting with a digital twin of the multi-tank system in a virtual reality environment.
Figure 8. A tank system control experiment as experienced in augmented reality.
Figure 9. Tuning controller gains dynamically in the augmented reality environment.
Next Steps
Our team is continuing to develop and improve the ReImagine Lab framework. As part of this work, we recently conducted a sytem usability study at Tallinn University of Technology. The study participants rated the VR experience higher than a standard desktop simulation. “The experience of working on this project was both fulfilling and enjoyable. The digital lab provided a unique and accessible way to gain deeper insights into control systems and processes that comprise them,” said Stanislav Jersov, a recent graduate of Tallinn University. “I truly believe that our work represents just the beginning of what is possible with mixed reality digital twins, and I am excited for their potential in education and beyond.”
However, students also provided feedback on usability that we are currently addressing. Some users found that the controls for interacting with the digital twin in the VR environment were not intuitive enough. We’ve been exploring ways to make the experience even more immersive. For example, as part of a master’s thesis, one Tallinn University student increased the size of the digital twin crane—so that the user could walk inside it within VR—and added audio to emulate the mechanical sounds the machinery would make as it moved. The effect of these additions was quite striking, as users at times moved quickly to get out of the way of the swinging virtual payload.
We also continue to add different types of lab experiments. We currently have eight different experiments, including the 3D crane, the tank system, an inverted pendulum, and a magnetic levitation system. Finally, in addition to integrating the ReImagine Lab framework more fully into our engineering curriculum, we are exploring ways to deploy our MATLAB and Simulink models directly to VR devices using automatic code generation.
Published 2023
