h5read being slow for files written by matlab
5 次查看(过去 30 天)
显示 更早的评论
I have some raw data in hdf5 file format (lz4 compressed) this is really quick to read, whcih is nice ;-).
I then do some stuff to the data, and save it in new hdf5 files (noncompressed) which is unreasonable slow to read in comparison:
>> tic;rawpix = h5read('raw_data.h5','/h5/data/path',[1 1 1], [1 1 1]);toc
Elapsed time is 0.154878 seconds.
>> tic;manpix = h5read('treated_data.h5','/h5/data/path', [1 1 1], [1 1 1]);toc
Elapsed time is 0.400602 seconds.
The raw data file is 3.8GiB (including 8 billion uint32's)
the treated one is 5GiB (including 0.7 billion doubles)
Is there a way to get theese speeds the same?
I think lz4 compressing as I'm writing the files would solve the problem, But I don't know how to make matlab use lz4 commpresion while writing, as it's not documented on The h5write documentation site, any ideas?
0 个评论
回答(1 个)
Preethi Ayyamperumal
2018-12-5
Currently, MATLAB supports only gzip compression for creating HDF5 dataset. Th compression levels are between 0-9.
Refer to the following link for applying gzip compression:https://www.mathworks.com/help/matlab/ref/h5create.html#buvazss-2
Please try this compression method to see if it resolves the performance issue.
0 个评论
另请参阅
类别
在 Help Center 和 File Exchange 中查找有关 HDF5 的更多信息
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!