How to load parquet files where some columns are lists

Question

Enea Ceolini 2021-12-13

1
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/1610010-how-to-load-parquet-files-where-some-columns-are-lists

回答： Divyanshu 2024-2-26

I am trying to work with parquet files between python and MATLAB. Unfortutanely it seems matlab does not support reading parquet files if some of the columns are "unsupported types" such as lists/arrays/JSON (see https://ch.mathworks.com/help/releases/R2021a/matlab/import_export/datatype-mappings-matlab-parquet.html).

Is there maybe a way to force MATLAB to read in those columns as strings?

I thoughts about getting the parquet file header info with `parquetinfo` and force the VariableTypes of the columns with lists to be "string" but it seems the returned info file is read-only.

Thanks,

+Enea

1 个评论
显示 -1更早的评论隐藏 -1更早的评论

Maik Berchten 2021-12-17

We're faced a similar problem and ended up converting the array to a CSV string and storing that as UTF8 string while generating the parquet file. Thanks for reporting this (and we're looking forward to a solution that doesn't need a workaround)!

--maik

请先登录，再进行评论。

请先登录，再回答此问题。

Answer 1

Divyanshu 2024-2-26

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/1610010-how-to-load-parquet-files-where-some-columns-are-lists#answer_1416578

Hi Enea,

The Parquet Files or Parquet Formatted data was not supported in the MATLAB version R2021a. However, in the later versions these file formats are supported as mentioned in the release notes of R2021b:

https://www.mathworks.com/help/releases/R2021b/matlab/import_export/supported-file-formats-for-import-and-export.html

Also there are two functions 'parquetread' and 'parquetwrite' through which you can work with parquet files from MATLAB. Please refer the following documentation links to read more about these functions: