Hello everyone,
Using MATLAB R2022a.
I was using my own HDF5 function and was trying to figure out why save with '-v7.3' without compression was giving me always larger output files. After a while I noticed that, most of the time, the "ChunkSize" for the MATFILE was not fitting well the size of the input data (size of the input divided by ChunkSize is not an integer most of the time). This results in, most of the time, matfiles that are 10% to 25% larger compared to my own HDF5 files.
Here is an simplified example compared to HDF5 (the input data size is a multiple of 128 so the chunk size is easy to compute for HDF5, just did that here so it's easy to understand for most people):
Creating some int16 matrix for example:
targetChunk = [128 1 256];
data = int16((2^16-1)*2*(rand(6144, 256, 800)-0.5));
Creating the HDF5 and MATFILE:
chunkSize = dataSize./(ceil(dataSize./targetChunk));
h5create('.\test.h5', ...
'Deflate', deflateValue, ...
h5write('test.h5', '/data', data);
save('test.mat', 'data', '-v7.3', '-nocompression');
On my computer, creating the HDF5 is about two times faster; but this does not matter much to me.
Comparating the size of the two output files:
fileHDF5 = dir('.\test.h5');
fileMAT = dir('.\test.mat');
fprintf('HDF5: Total size %.2f MB - %0.3f bytes/sample\n', ...
(fileHDF5.bytes./(1024*1024)), fileHDF5.bytes./(numel(data)));
fprintf('MATFILE: Total size %.2f MB - %0.3f bytes/sample\n', ...
(fileMAT.bytes./(1024*1024)), fileMAT.bytes./(numel(data)));
For that example I am getting:
HDF5: Total size 2403.15 MB - 2.003 bytes/sample
MATFILE: Total size 2739.15 MB - 2.283 bytes/sample
The HDF5 is correct, there is of course some headers/overhead (which are almost empty here, having only a single dataset in those files), but still the MATFILE is way too big (about 12% here).
If we check the metadata for both files (hdf5 and matfile):
hHDF5 = h5info('.\test.h5');
hMATFILE = h5info('.\test.mat');
hMATFILE.Datasets.ChunkSize
hHDF5.Datasets.ChunkSize is "128 1 200" which is expected.
hMATFILE.Datasets.ChunkSize is "1 256 114" which is weird to me.
size(data, 3) being equal to 800 I think this is why I am getting extra bytes with the matfile. In my opinion, this should not happen.
I simplified the input datasize, sometimes getting a more accurate chunksize for the HDF5 is not easy, but almost every time 'save' has a chunksize which is off.
This results in my function using HDF5 being better 95% of the time and 5% of the time having the same output as 'save' (including some headers/overhead).
Any idea why is that happening?
Thank you.
