Parquet File Logging Format for Simulation Data
You can export simulation data to a Parquet file using the Record block or the Simulation Data Inspector. Parquet is an open-source file format with efficient compression and encoding of column-oriented data that can be useful for processing big data. When you export data to a Parquet file, the way the data is stored in the file depends on the type of data that you export.
You can export real or complex scalar and multidimensional data from a signal, bus, or array of buses to a Parquet file. You can also export messages to a Parquet file as double values. Exporting variable-size signals to a Parquet file is not supported.
Data Types in Parquet File
The Record block and Simulation Data Inspector support some data types
that are not supported by a Parquet file. Most data types, such as
double, int, or string, do not
change. This table lists data types supported by the software and how these data types are
represented in the Parquet file.
| Simulink Data Type | Parquet File Logical Data Type |
|---|---|
double | double |
single | single |
int8 | int8 |
int16 | int16 |
int32 | int32 |
int64 | int64 |
uint8 | uint8 |
uint16 | uint16 |
uint32 | uint32 |
uint64 | uint64 |
string | string |
Boolean | Boolean |
half | double |
fixed point | double (fixed-point data is stored in the JSON
sidecar) |
enum | int32 |
image | Data type of underlying image data |
datetime | double representation of epoch time |
For more information about Parquet file data types, see Apache Parquet Data Type Mappings.
Data Format in Parquet File
How the data is formatted in the Parquet file depends on the type of signal being recorded. This table shows how each type of Simulink® signal is recorded in the Parquet file.
| Simulink Signal Type | Parquet File Logging Format |
|---|---|
| Scalar signal | Single column with a scalar value at each time step |
| Scalar signal with complex data | Single column with a |
| Nonscalar signal | Single column with sample values in the form of a vector, list of column vectors, or a nested list of column vectors for each time step |
| Nonscalar signal with complex data | Single column containing |
| Virtual or nonvirtual bus | Separate columns for each element in the bus or bus hierarchy |
| Array of buses | Separate columns for each element in the array of buses |
| Variable-size signal | Not supported |
Single-Rate and Multirate Data
You can save data to a Parquet file using shared or individual time columns. When you save single-rate data with a shared time column, the first column in the file contains time data, followed by columns containing signal data. The Record block and the Simulation Data Inspector export data to a Parquet file using shared time columns by default.
When you save multirate data using shared time columns, signals that have identical time data are grouped by shared time vectors. Time columns specify the sample times for signals to the right, up to the next time vector.
When you save data using individual time columns, the software saves data in pairs of time and signal data columns.
A Parquet file requires that all columns be of equal length. When you record signals that
are not of equal length to the same Parquet file, the software appends
NULL to any empty cells.
Complex Signals
The Record block and the Simulation Data Inspector export complex sample
values to a Parquet file as a 1-by-2 vector, where the first element is the real part and
the second element is the imaginary part of the complex value. For example, a scalar signal
value of 0.3973 + 0.5960i is saved as [0.3973,
.5960].
Multidimensional Signal Data
Multidimensional signal data with fixed dimensions can be represented in the Record block or the Simulation Data Inspector in two ways:
A single signal with multidimensional sample values
A set of signals with scalar sample values: one signal, called a channel, for each element of the multidimensional data
For both representations, the data for each time step is stored in the Parquet file as vectors for one-dimensional arrays, a list of column vectors for two-dimensional arrays, or as a nested list of column vectors for arrays with more than two dimensions. For instance, a 2-by-3 matrix-valued signal is recorded as a column of data where each entry consists of a vector of three 2-element vectors.
To export individual channels to a Parquet file, in the Simulation Data Inspector, use
the signal table to select only the channels to be exported. By default, signals with
samples that contain fewer than five elements are represented as channels. To represent a
multidimensional signal with five or more elements as channels, use the expand
function or click the signal dimension in the signal table and select Convert to
channels. When you export only selected runs or signals and you select
individual channels, rather than the parent signal, of an expanded multidimensional signal,
the Parquet file allocates a separate column for each channel. For more information about
multidimensional signals, see Analyze Multidimensional Signal Data.
Complex Multidimensional Signal Data
When you save multidimensional signals that contain complex data to a Parquet file,
each sample element is a nested 1-by-2 vector, where the first element is the real part
and the second element is the imaginary part of the complex value. For real values, the
second element is 0.
![A model logs a 2-by-3 matrix signal containing complex data using a Record block. In the saved Parquet file, there are two columns: time and signal data grouped into three 1-by-2 vectors of the 2-by-3 matrix signal at each sample time. Each element of the sample values is represented as a pair of real and imaginary components in the form ([real, imaginary]).](record-parquet-matrix-complex.png)
Buses
You can export data logged from virtual or nonvirtual buses to a Parquet file. In the Parquet file, dots in signal names specify the bus hierarchy.
You can also export arrays of buses to a Parquet file. In the Parquet file, each element in the array of buses is stored in a separate column.
Enumerated Data
When you save enumerated data to a Parquet file, the software exports only the
underlying integer data as int32 values.
For example, the MyColors class in this model defines a set of
enumerated values consisting of six colors, each associated with an integer value between
0 and 5.
When you save the enumerated data to a Parquet file, only the underlying integer values associated with each enumerated value are saved in the file.
