The documentation page for the sim function in Reinforcement Learning Toolbox states that the env input argument must be a "reinforcement learning environment object" and offers some suggestions on functions that you can use to create such an object. The agents input argument must be a "reinforcement learning agent object or array of agent objects" and suggests some functions that you can use to create those objects. The simOptions input argument must be an rlSimulationOptions object.
Are the inputs in your sim call the correct type of objects? If you're not sure set a breakpoint on the line where you call sim or set an error breakpoint. When MATLAB reaches the breakpoint, look at the variables with which you call sim. Show us the output of the whos function when you're stopped at the breakpoint if you're not certain.
