Detect precision loss

The Detect precision loss parameter specifies the diagnostic action to take if the software detects parameter precision loss. Precision loss occurs when the data type of a parameter does not have enough precision to represent the parameter value exactly. As a result, the value of the parameter in the model differs from the ideal value.

Both parameter precision loss and parameter overflow are quantization errors. While parameter precision loss differs from parameter overflow, the distinction can be nuanced. Parameter precision loss occurs when the ideal parameter value is within the range of the data type and scaling, but the data type does not have an exact representation of the parameter value.

The Detect overflow diagnostic reports all quantization errors greater than the specified Bits of error threshold, which has a default setting of one bit. For very small parameter quantization errors and when the Bits of error threshold is set to One bit, precision loss is reported rather than an overflow when:

where: