Depends on when the plant and controller start running there can be an offset between controller and plant. And as per the current design plant will run two times for each controller loop, hence typically plant simulation time will be two times controller simulation time plus additional offset. I dont think this might be the reson behind the issue you are facing. There might be some issues in the plant that is getting exposed with your controller design. I recommend you to give a try with the plant model used for HITL (example link) which is more robust. You need to replace the serial based MAVLink Bridge Source and Sink blocks in this model with TCP block to establish connection with PX4 SITL.
To answer your question on why it worked in MIL, I need to know how did you run the MIL. Did you run the MIL with the same plant you are using for SITL? PX4 have multiple module like navigator commander estimator and its only a part is designed in Simulink so technically just with MIL, you cant really verify how the controller runs with other modules in PX4. in SITL along with the PX4 desined controller other modules also will be runnign and they will interact and its lot different from MIL.
