Code Generation for Quantized LiteRT Models (Tech Preview)

R2026b
Since R2026b

Note

Code generation for quantized LiteRT models is a tech preview feature. This feature is in active development and might change between the tech preview and the general release. The primary purpose of the tech preview is to solicit feedback from users. To enable this feature, enter enableCodegenForQuantizedLiteRTModels at the command line before calling the loadLiteRTModel function or loading a model by using the LiteRT block. To provide feedback, email the development team or participate in a survey.

Quantized LiteRT models use reduced-precision integer representations for weights and activations, which results in smaller deployment artifacts and faster inference on hardware that uses native integer compute units. You can generate C, C++, or CUDA code for quantized LiteRT models.

Supported Quantization Schemes

You can generate code from quantized models that use the int8 quantization scheme. In this scheme, weights and activations are quantized to 8-bit integers, which results in an approximately fourfold reduction in model size compared to fp32.

For models quantized using dynamic range quantization, the generated code dequantizes weights at runtime and performs computation in floating point (fp32). Unlike full integer quantization, dynamic range quantization does not quantize activations at runtime in the generated code. This behavior is identical to weights-only quantization and provides model size reduction but does not accelerate inference through integer compute.

Deployment Targets

You can deploy generated code for quantized LiteRT models to embedded targets such as:

  • Raspberry Pi®

  • ARM® Cortex®- M processors

  • Custom SoCs

Generate Code for Quantized Models

To generate code for a quantized LiteRT model:

  1. Enable code generation for quantized LiteRT models by entering this code at the command line:

    enableCodegenForQuantizedLiteRTModels();

  2. Load the quantized model by using the loadLiteRTModel function or the LiteRT block.

  3. To verify the input and output data types, inspect the model by using the summary, inputSpecifications, and outputSpecifications functions.

    By default, LiteRT preserves model inputs and outputs as fp32 floating-point values and performs quantization internally. If the model uses integer inputs and outputs, quantize the input data to match the expected input format.

  4. Generate code by using codegen with a coder.CodeConfig or coder.GpuCodeConfig (GPU Coder) object.

Usage Notes and Limitations

  • Code generation for quantized LiteRT models is a Tech Preview feature and might change in future releases.

  • The generated code does not use hardware-native integer acceleration, such as tensor cores or int8 DSPs. Quantization reduces the deployment size, but does not improve the inference speed.

  • For operators that do not have quantized code generation support, the code generator dequantizes inputs, performs the computation in floating point, and requantizes the output.

  • Dynamic range quantized models do not perform int8 compute in the generated code. The generated code dequantizes weights at runtime and performs computation in fp32. This behavior is the same as weights-only quantization.

See Also

| | | | |

Topics