Does matlab support parquet partitions

Question

Jerry Duggan 2022-12-30

0
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/1886507-does-matlab-support-parquet-partitions

回答： Sudarshan 2023-1-2

I have a large data set written using parquet partitioning. The partition variable is called 'mdRun', and I have 10 parquet files created in 10 directories as follows:

.../events/mdRun=0/events-0.parquet

../events/mdRun=1/events-0.parquet

and so on. I created these files using pyarrow Hive partitioning.

Using pyarrow, I can read the parquet file corresponding to a single partition using the filter argument, which will read only the parquet file stored in the appropriate directory. As a nice side effect, the mdRun column is not stored in the parquet file, but it is automatically included when I read a partition file(s).

Is it possible to read a parquet partitioned dataset in matlab in the same way?

Thank you!

0 个评论
显示 -2更早的评论隐藏 -2更早的评论

请先登录，再进行评论。

请先登录，再回答此问题。

Answer 1

Sudarshan 2023-1-2

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/1886507-does-matlab-support-parquet-partitions#answer_1139582

Hi Jerry,

As per my knowledge, the feature is not supported by MATLAB in R2022b. This request has already been forwarded to the relevant team.

However, MATLAB R2022b does support parquet file reading and writing. I have attached a few documentation links that may help you in working with parquet functions.

You can refer to the link below for various functions that could be useful in your case:

https://in.mathworks.com/help/matlab/parquet-files.html

You can refer to link below for the detailed documentation of the data type mappings:

https://in.mathworks.com/help/matlab/import_export/datatype-mappings-matlab-parquet.html

To help you read parquet files, you can refer the link below:

https://in.mathworks.com/help/matlab/ref/parquetread.html

I hope that this helps!