Hello Czakó,
Unfortunately, it is currently not possible to stop model simulations in SimBiology programatically. A better error handling has been requested by multiple users and we are investigating the options. But this is not available at this point (R2020b). Currently, the only way to explore different starting points is to run sbiofitmixed with different initial values, which may not prevent running into parameter region where model simulations fail.
Since there is not much you can do with tweaking the ode integration or sbiofitmixed, you could try to find values for fixed / random effects / covariates where the model simulation returns nan or complex values by looking at the progress plot. It is very technical, but you could also set a breakpoint in the Matlab debugger (run dbstop if caught to set, and dbclear if caught to clear the breakpoint) before running the fit. This will stop the Matlab debugger in the internal functions of SimBiology; the failing set of parameter values is likely in the phi or initialValues input argument of the functions that is executed.
This may give you some information about parameter values for the fixed / random effects where the fit fails. Running a scan task by sampling from the random effects could give some insight which model dynamics and parameter values are the culprit for failing simulations.
Best,
Florian
