Import large .par file

Question

Carlo Zarattini 2023-5-24

0
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/1972149-import-large-par-file

回答： Saffan 2023-5-29

I need to import a .par file with 2.5gb of dimension. The .par file is a numerical table with 10 columns and few millions of rows.

I tried to used the parquetread function but i got an error message ("Unable to read Parquet"). A reduced version of the parquet file, 500mb, is uploaded successfully. My conclusion is that the original .par file is too large.

Any suggestion on how to import large .par file?

0 个评论
显示 -2更早的评论隐藏 -2更早的评论

请先登录，再进行评论。

请先登录，再回答此问题。

Answer 1

Saffan 2023-5-29

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/1972149-import-large-par-file#answer_1246479

Hi Carlo,

One way to solve this issue could be by splitting the original parquet file into multiple smaller files and create “parquetDatastore” of them. The data can then be obtained from the datastore using the “read” function. A datastore is a repository of files where each individual file can fit in the memory but the entire collection of files may not fit. They are used to work with the data incrementally and hence reduces the memory usage.

You can refer to the following link for more information on datastores: https://www.mathworks.com/help/releases/R2022b/matlab/ref/datastore.html