Main Content

Visualize Sensor Data from Android Device in RViz

This example uses:

Open Model

This example shows how to generate an Android® application to visualize live sensor data from an Android device on the ROS visualization (RViz) tool using Simulink® Support Package for Android Devices. In this example, sensor data such as orientation, angular velocity, and acceleration of the device are visualized. Your computer is configured as ROS master, and the Android device is configured as ROS client. The device connects to the computer through the Robot Operating System (ROS) interface. The sensor data from the device is visualized using the RViz tool running on the computer.

Requirements

Hardware Requirements

  • Android device

  • USB cable

Software Requirements

Install these products in your computer:

Prerequisites

Introduction

Robot Operating System (ROS) is a communication interface that enables different parts of a robot system to discover, send, and receive data. Using the ToApp block from the Simulink Support Package for Android Devices, you can exchange data with ROS-enabled physical robots or robot simulators such as Gazebo. For more information, see and the Concepts section on the ROS website.

Step 1: Run the Gazebo Simulator on the Computer

1. Complete the Download Virtual Machine step from the Get Started with Gazebo and Simulated TurtleBot (ROS Toolbox) example.

2. Open the ROS Melodic Terminal on the downloaded virtual machine.

3. To enable communication between ROS nodes, execute this command in ROS Melodic Terminal.

roscore

Step 2: Generate an Android Project Using Simulink

This section explains how to generate an Android project from a Simulink model.

Open the Simulink Model

1. Connect the Android device to your computer with a USB cable.

2. Install the Simulink Support Package for Android Devices. If you have already installed the support package, skip this step.

3. Open the IMUControl Simulink model. This model is preconfigured to run on the Android device.

The model has these blocks:

  • Orientation: The block reads the rotation of the device using three angular quantities: pitch (around x- axis), roll (around y- axis), and azimuth (around z- axis).

  • Accelerometer: The block reads acceleration along the x-, y-, and z- directions of the device.

  • Gyroscope: The block reads the rate of rotation along the x-, y-, and z- directions of the device.

  • MATLAB Function: The MATLAB function block converts the output of the Orientation block to quaternion angles.

  • ToApp: The block accepts the sensor data from the device and sends the data to the ROS Publish method. The ROS Publish method publishes the data to RViz.

  • Scope: The block displays signals generated during simulation.

Configure the Simulink Model

1. In the Simulink editor, select Simulation > Model Configuration Parameters.

2. In the Configuration Parameters dialog box, select Hardware Implementation.

3. Set the Hardware board parameter to Android Device. This selection automatically populates the parameters in the Hardware board settings with the default values for Android.

a. From the Groups list under Target hardware resources, select Device options.

b. From the Device list, select your Android device. If your device is not listed, click Refresh.

Note: If your device is not listed even after clicking the Refresh button, ensure that you have enabled the USB debugging option on your device. To enable USB debugging, enter androidhwsetup in the MATLAB Command Window and follow the onscreen instructions.

4. Click Apply. Click OK to close the dialog box.

Deploy Simulink Model on Android Device

On the Hardware tab of the Simulink model, in the Mode section, select Run on board and then click Build, Deploy & Start. The lower-left corner of the model window displays status while Simulink prepares, downloads, and deploys the model on the hardware. On successfully deploying the model, a project file is generated in the MATLAB build folder at MATLAB_build_folder/modelName_ert_rtw/modelName. For example, L:/workspace/IMUControl/IMUControl_ert_rtw/IMUControl.

Step 3: Import the Project and Add ROS Dependencies in Android Studio

1. Import the generated project to Android Studio by following the steps as described in Import Project to Google Android Studio.

2. Select the Project option from the top-left drop-down menu above the File Explorer.

3. Open the build.gradle file of your project and update the file by changing the code at the positions labeled in the image.

  • Position 1 — Add this line in the buildscript section.

apply from: "https://github.com/rosjava/android_core/raw/kinetic/buildscript.gradle"

  • Position 2 — Add these lines in the allprojects section.

apply plugin: 'ros-android'
afterEvaluate { project ->
android {
// Exclude a few files that are duplicated across our dependencies and prevent packaging Android applications.
           packagingOptions {
               exclude "META-INF/LICENSE.txt"
               exclude "META-INF/NOTICE.txt"
           }
       }
   }

  • Position 3 — Add these ROS dependencies below the listed dependencies.

implementation "org.ros.rosjava_core:rosjava:[0.3.2,0.4)"
implementation "org.ros.rosjava_messages:diagnostic_msgs:[1.12,1.13)"
implementation "org.ros.rosjava_messages:sensor_msgs:[1.12,1.13)"
implementation "org.ros.rosjava_messages:geometry_msgs:[1.12,1.13)"
implementation "org.ros.rosjava_core:rosjava_geometry:0.3.6"
implementation 'org.ros.rosjava_messages:visualization_msgs:[1.12,1.13)'
implementation "org.ros.rosjava_bootstrap:message_generation:[0.3,0.4)"
implementation 'org.ros.rosjava_messages:std_msgs:0.5.11'

4. Open the AndroidManifest.xml file located in the src > main folder of your project. Update the file by changing the code at the positions labeled in the image.

  • Position 1 — Add this code after the application tag.

xmlns:tools="http://schemas.android.com/tools" tools:replace="android:icon"

  • Position 2 — Replace the action and category sections.

<action android:name="com.android.rosupport.START_MODEL"/>
<category android:name="android.intent.category.DEFAULT" />

  • Position 3 — Add the activity and service sections immediately after the activity section.

<activity android:label="@string/app_name" android:name=".MainActivity"
   android:theme="@style/MWAppTheme" android:configChanges="orientation|screenSize">
   <intent-filter>
       <action android:name="android.intent.action.MAIN"/>
       <category android:name="android.intent.category.LAUNCHER"/>
   </intent-filter>
</activity>
<service android:name=".MWNodeMainExecutorService"
   tools:ignore="ExportedService">
   <intent-filter>
       <action android:name="org.ros.android.NodeMainExecutorService"/>
   </intent-filter>
</service>

  • Position 4 — Add these ROS uses-permissions below the listed uses-permission.

<uses-permission android:name="android.permission.FOREGROUND_SERVICE"/>
<uses-permission android:name="android.permission.WRITE_EXTERNAL_STORAGE"/>
<uses-permission android:name="android.permission.ACCESS_WIFI_STATE"/>
<uses-permission android:name="android.permission.CHANGE_WIFI_STATE"/>
<uses-permission android:name="android.permission.WAKE_LOCK"/>
<uses-permission android:name="android.permission.SYSTEM_ALERT_WINDOW"/>

5. Change the MATLAB Current Folder by using this command in the MATLAB Command Window:

cd(fullfile(matlabshared.supportpkg.getSupportPackageRoot,'toolbox','target','supportpackages','android','androidexamples','ROSonAndroid','IMUControl'))

a. Copy these files and paste them in the src > main > java > com.example.IMUControl folder of your project. Here, IMUControl is the name of your project.

  • ROSMainActivity.java

  • PublishNode.java

  • MainActivity.java

  • MWNodeMainExecutorListener.java

  • MWNodeMainExecutorService.java

  • MWNodeMainExecutorServiceListener.java

b. Save the main.xml file in the src > main > res > layout folder of your project.

c. Save the mwstyle.xml file in the src > main > res > values folder of your project.

6. Open the IMUControl.java file located in the src > main > java > com.example.IMUControl folder of your project. Here, IMUControl is the name of your project. Update the file by changing the code at the positions labeled in the image.

  • Position 1 — Add the ROS import statements.

import org.ros.node.NodeConfiguration;
import org.ros.node.NodeMainExecutor;

  • Position 2 — Replace AppCompactActivity with ROSMainActivity.

  • Position 3 — Add the PublishNode statement before the definition of the CountDownTimer method.

PublishNode pn;

  • Position 4 — Add the ROSPublish and init method definitions before the definition of the naMain method.

public void ROSPublish(double[] data)
{
   if(pn != null)
       pn.publishData(data);
}
protected  void init(NodeMainExecutor nodeMainExecutor) {
   pn = new PublishNode("/android/imu","sensor_msgs/Imu");
   NodeConfiguration nodeConfiguration = NodeConfiguration.newPublic(getRosHostname());
   nodeConfiguration.setMasterUri(getMasterUri());
   nodeMainExecutor.execute(pn, nodeConfiguration);
}

The ROSPublish method accepts the data from the Orientation, Accelerometer, and Gyroscope blocks of the Simulink model. The method then publishes the data to RViz (ROS master). The data type of the published data depends on the message type of the ROS topic. In this example, the publish topic is /android/imu and the message type is sensor_msg/Imu. You can use a different topic name and message type based on the requirement of your application. The init method is called after the ROS client is created. In this function, you can specify the number of publishers and subscribers required for the application. In this example, only one publisher is required.

Step 4: Sync Project Files in Android Studio

In Android Studio, click the Sync Now button in the top-right corner of the Android Studio editor.

When syncing your project, you may see the following error message. To resolve this issue, click the link that appears in the error message.

Note: The build tools version number specified in the error message may vary depending on the version of the Android gradle plugin installed in the computer.

Step 5: Configure and Run the Android Device as ROS Client

Note: Before proceeding, ensure that your computer and the Android device are connected to the same Wi-Fi® network.

1. In the Android Studio toolbar, select Run > Run 'IMUControl'. Android Studio installs the application on your device. The application starts running on the device and provides you the options to configure the device as ROS master or ROS client.

2. Select the ROS Client option and specify the IP address and the port number. In this example, the IP address is 192.255.88.128 and the port number is 11311.

3. To connect the device to the computer, click Connect. The device starts sending sensor data to computer. The application displays signals that represent the sensor data from the device over a certain amount of time. The y-axis of the application represents sensor data, whereas the x-axis represents time.

Step 6: Visualize Sensor Data in RViz

1. Open ROS Melodic Terminal on the VM installed on your computer and execute this command. The RViz tool is launched.

rviz

2. To add a display, click the Add button under the Displays sidebar.

3. In the Create Visualization pop-up window, scroll to Imu under the rviz_plugin_tutorial category, and then click OK. An Imu display is added to the Display sidebar.

4. In the Displays sidebar, click the Imu display. A list of properties associated with Imu is listed.

5. From the properties list, set Topic to /android/imu. In the Visualization window, you will now see an arrow representing the position and orientation of the device. Move the device and observe as the arrow changes its position.

This illustration shows the arrow in the Visualization window when the device is facing in the upward direction.

See Also

Set Position of Bebop Drone on Gazebo Simulator from Android Device Using ROS