Understand Unexpected Results from Data Type Conversion

Open Live Script

Simulink® models that use fixed-point data types can sometimes yield unexpected values due to precision loss from quantization. This example shows how quantization occurs when you calculate fixed-point values. While this example uses the Data Type Conversion block, similar calculations can lead to quantization in other Simulink blocks, such as Gain and Product.

Consider a simple data type conversion operation in Simulink. The model shown here converts the value 0.5 from sfix16_Sp1 to sfix16_En7. Both of these data types are able to represent the value 0.5. However, quantization occurs during data type conversion, resulting in the observed value 0.4921875.

Data Type Conversion Calculation

To understand how quantization occurs during this operation, consider the data type conversion formula

$Q_{i n} \times F_{i n} \times 2^{E_{i n}} + B_{i n} = Q_{o u t} \times F_{o u t} \times 2^{E_{o u t}} + B_{o u t}$

where

$Q_{in}$ , $Q_{out}$ are stored integers.
$F_{in}$ , $F_{out}$ are the Slope Adjustment Factor of their respective data type.
$E_{in}$ , $E_{out}$ are the Fixed Exponent of their respective data type.
$B_{in}$ , $B_{out}$ are the Bias of their respective data type.

Each side of the equation represents a real-world value. The left side represents the value of the block input. The right side represents the value of the block's output. The formula contains the data type specifications that translate the stored integers, $Q_{in}$ and $Q_{out}$ , to real-world values. The goal of the data type conversion equation is to retain the real-world value of the input signal while changing the data type.

When converting data types, the input stored integer value, $Q_{in}$ , and input data type are known and give the values for the input variables, including the stored integer value. The output data type is also known, as it is specified in the Data Type Conversion block. The unknown variable is the output stored integer $Q_{out}$ . Solve for $Q_{out}$ to find the stored integer value that best represents the input value with the new data type specifications.

To solve for $Q_{out}$ , gather the known values of the input and output variables from the Simulink data types in the model.

Gather Known Values for Input Calculation

At the command line, create a fi object a_in that has the same numeric type as the input data type, sfix16_Sp1.

T_in = numerictype("sfix16_Sp1");
a_in = fi(0.5,T_in)

a_in = 
0.5


          DataTypeMode: Fixed-point: slope and bias scaling
            Signedness: Signed
            WordLength: 16
                 Slope: 0.1
                  Bias: 0

The data type has a Bias value of 0. To gather the remaining known values for the input data type, query the int, Slope Adjustment Factor, and Fixed-Exponent parameters of a_in.

Q_in = a_in.int;
F_in = a_in.SlopeAdjustmentFactor;
E_in = a_in.FixedExponent;

The values for the input equation are:

$Q_{in}$ = 5
$F_{in}$ = 1.6
$E_{in}$ = –4
$B_{in}$ = 0

Gather Known Values for Output Calculation

Gather the known values for the output data type. Create a fi object a_out that has the same numeric type as the output data type, sfix16_En7.

T_out = numerictype("sfix16_En7");
a_out = fi(0.5,T_out)

a_out = 
    0.5000

          DataTypeMode: Fixed-point: binary point scaling
            Signedness: Signed
            WordLength: 16
        FractionLength: 7

The data type has a Bias value of 0. Query the Slope Adjustment Factor, and Fixed-Exponent parameters of a_out to gather the remaining known values.

F_out = a_out.SlopeAdjustmentFactor;
E_out = a_out.FixedExponent;

The values for the output equation are:

$F_{out}$ = 1
$E_{out}$ = –7
$B_{out}$ = 0

Solve for Stored Integer Value

Solve the data type conversion equation for $Q_{out}$ . In this example, the bias of each data type is equal. The remaining values result in the equation

$Q_{o u t} = Q_{i n} \times \frac{F_{i n}}{F_{o u t}} \times 2^{E_{i n} - E_{o u t}}$

Estimate the quantized Slope Adjustment Factor $\frac{F_{i n}}{F_{o u t}}$ using a fi object with the signedness and word length of the output value.

initFormat = get(0, 'Format');
format longg
quantSlopeAdj = fi(F_in/F_out,true,16)

quantSlopeAdj = 
           1.5999755859375

          DataTypeMode: Fixed-point: binary point scaling
            Signedness: Signed
            WordLength: 16
        FractionLength: 14

The quantized Slope Adjustment Factor value is slightly less than the ideal value 1.6.

You can now substitute the known values into the equation to find $Q_{out}$ .

Q_in = 5;
F_net = quantSlopeAdj;
E_net = E_in-E_out;
Q_out = Q_in*F_net*2^E_net

Q_out = 
63.9990234375


          DataTypeMode: Fixed-point: binary point scaling
            Signedness: Signed
            WordLength: 64
        FractionLength: 53

Since $Q_{out}$ is a stored integer, it must be rounded to an integer value. The default integer rounding mode is Floor.

Q_out = floor(Q_out)

Q_out = 
    63

          DataTypeMode: Fixed-point: binary point scaling
            Signedness: Signed
            WordLength: 11
        FractionLength: 0

The stored integer value for the output after data type conversion is 63.

Calculate Real-World Value from Data Type Conversion

To see the real-world value that would be represented by the stored integer $Q_{out}$ with the Simulink data type sfix16_En7, plug $Q_{out}$ back into the right side of the original equation. You would see this value displayed as the output after data type conversion.

RWV = Q_out * F_out * 2^E_out

RWV = 
                 0.4921875

          DataTypeMode: Fixed-point: binary point scaling
            Signedness: Signed
            WordLength: 44
        FractionLength: 36

The result of the Data Type Conversion block calculation is, 0.4921875. Although the data type sfix16_En7 can represent the value 0.5 exactly, these calculations show how quantization can occur when you use a Data Type Conversion block. The principles of quantization also apply to calculations of fixed-point values in other Simulink blocks. To learn more about how to adjust settings in your Simulink model to improve accuracy of fixed-point values, see Resolve Unexpected Values from Data Type Conversion.

% Restore initial format
format(initFormat);