h5read being slow for files written by matlab

13 次查看(过去 30 天)
I have some raw data in hdf5 file format (lz4 compressed) this is really quick to read, whcih is nice ;-).
I then do some stuff to the data, and save it in new hdf5 files (noncompressed) which is unreasonable slow to read in comparison:
>> tic;rawpix = h5read('raw_data.h5','/h5/data/path',[1 1 1], [1 1 1]);toc
Elapsed time is 0.154878 seconds.
>> tic;manpix = h5read('treated_data.h5','/h5/data/path', [1 1 1], [1 1 1]);toc
Elapsed time is 0.400602 seconds.
The raw data file is 3.8GiB (including 8 billion uint32's)
the treated one is 5GiB (including 0.7 billion doubles)
Is there a way to get theese speeds the same?
I think lz4 compressing as I'm writing the files would solve the problem, But I don't know how to make matlab use lz4 commpresion while writing, as it's not documented on The h5write documentation site, any ideas?

回答(1 个)

Preethi Ayyamperumal
Currently, MATLAB supports only gzip compression for creating HDF5 dataset. Th compression levels are between 0-9.
Refer to the following link for applying gzip compression:https://www.mathworks.com/help/matlab/ref/h5create.html#buvazss-2
Please try this compression method to see if it resolves the performance issue.

产品


版本

R2018b

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by