This happens when your model simulation violates your design ranges (min, max specifications). Fxpopt throws this error, as it considers design ranges to be a hard requirement for your system specification and when the simulation violates the specification, fxpopt considers your model to be in faulty state.
A quick way to test this is to turn on the signal range checking by running the following command:
sim(model, 'SignalRangeChecking', 'error')
You can either update the design ranges or investigate why the simulation violates these ranges.