Processing Tall Arrays Taking Too Long

Question

Avinash Rajendra 2017-11-1

0
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/364565-processing-tall-arrays-taking-too-long

评论： Dominique Ingala 2021-4-7

I have a tall array with about 1.7 billion rows of data and 14 columns. I want to be able to process this data in the same way that several examples (with airline data) do it. I am just trying to extract one column and find the mean. My code is something like:

ds = datastore('some-file.csv');
tt = tall(ds); %Mx14 tall table (M should be about 1.7 billion)
a = tt.V;      %Mx1 tall double %(M should be the same as above)
m = mean(a);   %One integer
gather_m = gather(m);

The gather step is taking way too much time. I haven't seen it complete at all. In the examples I have seen, this step is shown to be completed in a few seconds. Eventually, I want to be able to make calculations and plots, but I want to start by making this simple step work first. Can anyone recognize the problem and recommend a solution? I have parallel pool turned on and there are two workers.

Thank you very much.

2 个评论
显示无隐藏无

Walter Roberson 2017-11-1

I had not realized that tall arrays used parallel if available, but I see that they do; https://www.mathworks.com/help/distcomp/run-tall-arrays-on-a-parallel-pool.html

Avinash Rajendra 2017-11-1

They do, but it still takes way too long to run. I feel like I'd be in good shape if the time to run the program was more manageable.

请先登录，再进行评论。

请先登录，再回答此问题。

Answer 1

Kojiro Saito 2017-11-2

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/364565-processing-tall-arrays-taking-too-long#answer_289036

在 MATLAB Online 中打开

It may speed up by configuring read size of datastore. You can know the default read size by

ds.ReadSize

This is the data size which MATLAB reads from the file at one time. You can set higher size than your default and this will reduce file I/O. Please add ds.ReadSize setting, for example,

ds = datastore('some-file.csv');
ds.ReadSize = 100000; % Or higher
tt = tall(ds); %Mx14 tall table (M should be about 1.7 billion)
a = tt.V; Mx1 tall double %(M should be the same as above)
m = mean(a); %One integer
gather_m = gather(m);