Hi Owen,
The issue you're encountering stems from the design of MATLAB's MAT-file formats and the inherent inefficiencies of the -v7.3 format for your specific use case.
- With the ‘-v7.3’ flag you can store variables with size greater than 2GB, with compression.
- Without the ‘-v7.3’ flag (e.g. if the default version is set to -v7 or lower) there is no (or less) compression, but we cannot store large arrays.
MAT-file structure: The ‘-v7.3’ format uses HDF5 as its backend, which is highly versatile but not optimized for cases where you have a single, large variable. HDF5 is designed for general-purpose storage, including metadata and other overheads that can lead to excessive file sizes.
Serialization limitations: Large and complex data structures like ‘struct’ can incur significant overhead because every field and subfield is treated as a separate dataset in HDF5.
‘getByteStreamFromArray’ is limited to serializing objects that MATLAB's internal serializer can handle. Structures or arrays with greater than 4 GB of data often hit limitations in MATLAB's serialization mechanism.
Some of the possible solutions that could resolve this issue are as follows:
Split into multiple variables and use -v7 format
- If feasible, divide your large structure into several smaller variables, each <2GB.
- Save these in the older ‘-v7’ format, which is more space-efficient for such cases.
fields = fieldnames(myStruct);
save(['part_' fields{i} '.mat'], 'myStruct', '-v7');
Writing custom serializer
- If the structure doesn't contain complex objects, you can recursively serialize it into a binary file with custom MATLAB code.
fid = fopen('large_struct.bin', 'w'); fwrite(fid, myStruct, 'uint8');
Use low-level HDF5 tools
- If you need to stick to ‘-v7.3’, you can consider using MATLAB's low-level HDF5 functions to write the structure directly without unnecessary overhead.
h5create('large_struct.h5', '/myStruct', size(myStruct));
h5write('large_struct.h5', '/myStruct', myStruct);
If performance is your priority and you don't need the ‘-v7.3’ features, you can split the data into smaller parts and use the -v7 format.
Moreover, if your workstation supports parallel computing, you can consider using MATLAB's Parallel Computing Toolbox to parallelize the saving process. This might help speed up the process, especially if your structure can be split into independent parts.
To know more about the usage of HDF5 functions used in the above code, refer to the documentation link mentioned below:
Best!
