How read extremely large HDF5 format data and resolve out of memory issue?

24 次查看（过去 30 天）

显示更早的评论

Trista Ni 2022-5-3

0
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/1710775-how-read-extremely-large-hdf5-format-data-and-resolve-out-of-memory-issue

回答： ROSEMARIE MURRAY 2022-5-3

I need to read several datasets from a 2TB HDF5 file which are needed for further computation.

If I simply coding as following,

varible1=h5read('path to .h5 file', 'path to dataset')

It would require ~500 GB array memory.

Is there any good way to solve this problem?

Thanks!

1 个评论
显示 -1更早的评论隐藏 -1更早的评论

Jonas 2022-5-3

编辑：Jonas 2022-5-3

does this help? https://de.mathworks.com/matlabcentral/answers/423693-read-and-divide-hdf5-data-into-chunks

maybe you could also spit data using oython beforehand, i saw multiple those scripts when googling

请先登录，再进行评论。

请先登录，再回答此问题。

回答（1 个）

ROSEMARIE MURRAY 2022-5-3

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/1710775-how-read-extremely-large-hdf5-format-data-and-resolve-out-of-memory-issue#answer_956405

You could use a fileDatastore with the read function h5read, which would allow you to specify a certain amount to read at a time.

Or you could try this datastore: https://www.mathworks.com/matlabcentral/fileexchange/64919-hdf5-custom-file-datastore-for-timeseries-in-matlab

0 个评论
显示 -2更早的评论隐藏 -2更早的评论

请先登录，再进行评论。

请先登录，再回答此问题。

类别

MATLAB Data Import and Analysis Data Import and Export Standard File Formats HDF5

在 Help Center 和 File Exchange 中查找有关 HDF5 的更多信息

标签

产品

MATLAB

版本

R2022a

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Translated by