Stream Camera, Depth and Semantic Segmentation Data from Unreal Engine to NVIDIA Jetson

Send Data Streams from Simulink

Open the model StreamFromUnrealToJetsonExampleModel.

open_system("StreamFromUnrealToJetsonExampleModel.slx")

This model simulates a US City Block scene in Unreal Engine, in which a UAV follows a trajectory defined by the position and orientation values defined in the model workspace. The UAV has an onboard camera. The camera is modeled by Simulation 3D Camera block that outputs the RGB image, depth map and semantic segmentation map of labels that correspond to the objects in the scene. The frame rate of the camera is configured to be 30 frames per second (fps) as defined by the Sample time parameter in the Simulation 3D Scene Configuration block. The three Video Send blocks stream corresponding data to the Jetson hardware. Change the IP address and port number values on these blocks to values that correspond to your Jetson hardware and network setup. Note that the Max frame time (ms) parameter in all three blocks is configured to be 34 milliseconds based on the input frame rate of 30 fps. All three blocks are configured to stream to different ports on the same remote IP address.

Inspect the block parameter configuration for the three data streams. The depth map is converted to uint16 data type using lossy rescaling and is configured to stream as 16-bit grayscale image with VP9 compression format. Additionally, note that the segLabels2RGBImage MATLAB Function block converts the segmentation label map to stream as a matrix of RGB values analogous to the camera image stream, with VP8 compression format. For more information about the colormap for label visualization, see Depth and Semantic Segmentation Visualization Using Unreal Engine Simulation.

Run StreamFromUnrealToJetsonExampleModel to start streaming data to the Jetson.

sim("StreamFromUnrealToJetsonExampleModel.slx");

Receive and Visualize Data Streams on Jetson

To visualize the data now streaming to Jetson, you must deploy three separate models individually to the Jetson hardware. This is because a Simulink model can only contain one SDL Video Display block and there are three separate data streams. First, open the model, VideoStreamReceiveOnJetsonExampleModel.

open_system("VideoStreamReceiveOnJetsonExampleModel.slx")

Note that the port number and compression parameters in the Network Video Receive block are same as those of the corresponding Video Send block streaming camera images. The sample time can be any value that is less than or equal to the sample time of the data stream. On the Hardware tab in the toolstrip, select Hardware Settings. Under Hardware board settings > Target hardware resources, populate the Device Address, Username and Password fields with values corresponding to your Jetson board. Then, under Deploy section of the Hardware tab, select Build, Deploy & Start to generate code for the model and deploy it to the Jetson. The SDL Video Display window opens up and visualizes the RGB image stream on the monitor connected to the Jetson.

To restart streaming after the simulation is complete, re-run StreamFromUnrealToJetsonExampleModel. Then, open and deploy the model, DepthStreamReceiveOnJetsonExampleModel, to visualize the depth map stream on the monitor connected to the Jetson.

sim("StreamFromUnrealToJetsonExampleModel.slx");
open_system("DepthStreamReceiveOnJetsonExampleModel.slx")

To restart streaming after the simulation is complete, re-run StreamFromUnrealToJetsonExampleModel. Then, open and deploy the model, LabelStreamReceiveOnJetsonExampleModel, to visualize the semantic segmentation label map stream on the monitor connected to the Jetson.

sim("StreamFromUnrealToJetsonExampleModel.slx");
open_system("LabelStreamReceiveOnJetsonExampleModel.slx")