您正在关注此问题

您将在关注的内容源中看到更新。
您可能会收到电子邮件，具体取决于您的通信预设项。

Alternative to csv and parquet for arrays

6 次查看（过去 30 天）

显示更早的评论

Pelajar UM 2022-3-13

0
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/1670629-alternative-to-csv-and-parquet-for-arrays

⋮

0
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/1670629-alternative-to-csv-and-parquet-for-arrays

评论： Walter Roberson 2022-3-15

I have pretty massive csv files. It's a pain for both transfer and also in terms of read time.

I have been using parquet but this is only for tables and my functions only work with double arrays. So whenever I load the file, I have to use table2array to create the proper variables. This takes some extra time. Still much better than using csv, but I am wondering if there are any light and efficient alternatives to csv for arrays....

14 个评论
显示 12更早的评论隐藏 12更早的评论

Jan 2022-3-13

此评论的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/1670629-alternative-to-csv-and-parquet-for-arrays#comment_2038744

⋮

链接

此评论的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/1670629-alternative-to-csv-and-parquet-for-arrays#comment_2038744

The question is not clear. Why do you use a text file to store large data sets? Text files are useful, if they are read and manipulated by human. The conversion of floating point numbers to strings and back to numbers can cause rounding effects. Therefore using a binary format is recommended and much more efficient.

How do you work with parquet in Matlab?

Walter Roberson 2022-3-14

此评论的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/1670629-alternative-to-csv-and-parquet-for-arrays#comment_2038764

⋮

链接

此评论的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/1670629-alternative-to-csv-and-parquet-for-arrays#comment_2038764

How often is each file read compared to being written?

Pelajar UM 2022-3-14

此评论的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/1670629-alternative-to-csv-and-parquet-for-arrays#comment_2039039

⋮

链接

此评论的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/1670629-alternative-to-csv-and-parquet-for-arrays#comment_2039039

在 MATLAB Online 中打开

Thanks guys.

@Jan The data is prepared using another program (written in python). It is a mesh data with various values assinged to each one of the element (all numerical values). The MATLAB code is used to visualize this data. The file does not have to be human readable (parquet is not human readable).

Example:

[filename, folder] = uigetfile ({'*parquet'});
if ~ischar(filename); return; end   %user cancel
filename = fullfile(folder, filename);
input = parquetread(filename);
app.UITable2.Data = table2array(input);
nodes = [app.UITable2.Data(:,1),app.UITable2.Data(:,2),app.UITable2.Data(:,3)];
elements = [app.UITable2.Data(:,4),app.UITable2.Data(:,5),app.UITable2.Data(:,6),app.UITable2.Data(:,7)];
elements=rmmissing(elements);
TR = triangulation(elements,nodes); %generating triangular mesh
[F,P] = freeBoundary(TR); %extracting the surface

@Walter Roberson It's written only once (in Python), and every time you open a new session of MATLAB, you have to load the data. So I would reading time is more than important than writing time.

Walter Roberson 2022-3-14

此评论的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/1670629-alternative-to-csv-and-parquet-for-arrays#comment_2039059

⋮

链接

此评论的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/1670629-alternative-to-csv-and-parquet-for-arrays#comment_2039059

I suggest writing it as a binary file, with a header indicating the size. You might want to make it compatible with https://www.mathworks.com/help/matlab/ref/multibandread.html

Pelajar UM 2022-3-14

此评论的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/1670629-alternative-to-csv-and-parquet-for-arrays#comment_2039074

⋮

链接

此评论的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/1670629-alternative-to-csv-and-parquet-for-arrays#comment_2039074

@Walter Roberson using fwrite / multibandwrite, it's only slightly lighter:

CSV: 25 mb

fwrite/multibandwrite: 23 mb

Parquet: 12 mb

Walter Roberson 2022-3-14

此评论的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/1670629-alternative-to-csv-and-parquet-for-arrays#comment_2039094

⋮

链接

此评论的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/1670629-alternative-to-csv-and-parquet-for-arrays#comment_2039094

That hints to me that your data might perhaps only justify single precision but that you are using double precision.

Sarah Gilmore 2022-3-14

此评论的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/1670629-alternative-to-csv-and-parquet-for-arrays#comment_2040189

⋮

链接

此评论的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/1670629-alternative-to-csv-and-parquet-for-arrays#comment_2040189

Hi @Pelajar UM,

Do you mind answering a few questons that may help narrow down the issue?

Do you know if it's parquetread or table2array that's taking the most time? You can use the performance profiler to determine which lines are causing the issue.
How wide is the table in the Parquet file. Is it just 7 columns?
Which version of MATLAB are you running?

Thanks,

Sarah

Pelajar UM 2022-3-15

此评论的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/1670629-alternative-to-csv-and-parquet-for-arrays#comment_2041619

⋮

链接

此评论的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/1670629-alternative-to-csv-and-parquet-for-arrays#comment_2041619

Hi Sarah

parquetread takes 0.11 s and table2array 0.034 s. No issues per se and relatively speaking, not long at all. But it is just an extra step and I was wondering if it could be avoided.
This particular dataset is 23 columns wide with ~114,000 rows.
R2021a

Walter Roberson 2022-3-15

此评论的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/1670629-alternative-to-csv-and-parquet-for-arrays#comment_2041659

⋮

链接

此评论的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/1670629-alternative-to-csv-and-parquet-for-arrays#comment_2041659

If the original data is 23 columns but you only need 7 of them, then you can improve reading speed by writing in binary one column at a time with a header indicating how many rows are present; then by knowing the size of the header and the number of rows, you can fseek() directly to the beginning of any particular column. Or in your case, since you are using the first 7 columns, just ask to fread [nrow 7] after you have positioned past the header, leaving the other 16 columns unread.

Walter Roberson 2022-3-15

此评论的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/1670629-alternative-to-csv-and-parquet-for-arrays#comment_2041689

⋮

链接

此评论的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/1670629-alternative-to-csv-and-parquet-for-arrays#comment_2041689

If those file sizes are a problem, then have the Python write each column into a separate binary file and zip the set of files together. Transfer the zip. Unzip at the destination. open and read only the files corresponding to the columns you want to read.

Pelajar UM 2022-3-15