When there is a expression that might produce a complex result, the input signals for the calculation must be changed to Complex, or you must use complex() on the real signals that might produce a complex result. For example if you have sqrt() then if the input might go negative then you need to complex() the input before taking the sqrt.
I gather the error shows up at the time the need for a complex result is detected, so if you are not expecting the calculation to produce a complex result you may need to add debugging to see why it does produce such a result.
MATLAB silently transforms values to complex at need, but Simulink does not.
When you have a loop, Simulink has to solve the loop to establish the initial conditions for self-consistency because everything in one step has to be true simultaneously (if that is not needed for you then send your signal through a time delay so the rest of the calculation can be performed in the next step) . The result can include values outside the range you expect.
