Hello Andreas,
My suspicion of what is happening here is library path conflict. ROS Toolbox ships with ROS libraries, which probably overlap with the libraries that rviz uses. When you start MATLAB, the environment within MATLAB is setup to have MATLAB's libraries on the environment path. If you compare the environment variables from inside of MATLAB with those from a standard terminal, you can see the difference.
I would generally suggest starting rviz from outside MATLAB. If your workflow requires that it is started from within MATLAB, you could temporarily set the relevant environment variables to hardcoded values that are copied from outside of MATLAB, call the system command to start rviz, then reset the environment variables to their original values. The variables I would look at first are PATH and LD_LIBRARY_PATH, though there may be others.
Something like this:
% Prepare to reset the environment to MATLAB's current values
inMATLABPath = getenv("PATH");
inMATLABLDPath = getenv("LD_LIBRARY_PATH");
cleanPath = onCleanup(@() setenv("PATH", inMATLABPath));
cleanLDPath = onCleanup(@() setenv("LD_LIBRARY_PATH", inMATLABLDPath));
% Set environment to match outside of MATLAB
setenv("PATH", outsideMATLABPath)
setenv("LD_LIBRARY_PATH", outsideMATLABLDPath)
% Start rviz
[status,cmdout] = system('rviz rviz -d runMyRvizLaunchFile.rviz');
% Reset environment to default
clear cleanPath cleanLDPath
-Cam
