large file slow running

Question

Joydeb Saha 2021-9-20

0
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/1456769-large-file-slow-running

回答： Jan 2021-9-20

latmin=-90; %chooce your location
latmax=90;
longmin=-180; a=1;
longmax=180;
data=zeros(90,90);
 for b=1:length(x(:,1))
if  (x(b,8)>=longmin && x(b,8)<longmax && x(b,7)>=latmin && x(b,7)<latmax);
 for col=1:13
 data(a,col)=x(b,col);
 end
 a=a+1;
 end
 end

theres a file called x.mat (601760x13). What might be the reason it runs too slow

1 个评论
显示 -1更早的评论隐藏 -1更早的评论

KSSV 2021-9-20

What you are trying to achieve? It can be achived without loop. You need to give an explanation of your problem. What does x.mat have?

请先登录，再进行评论。

请先登录，再回答此问题。

Answer 1

Jan 2021-9-20

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/1456769-large-file-slow-running#answer_791364

在 MATLAB Online 中打开

I guess that the iterative growing of the array data slows down the processing.

Remember that the growing of an array required to create a new array and to copy the foermer contents.

x = [];
for k = 1:1e6
   x(k) = rand;
end

This creates a new array in each iteration. Although the final array needs 8MB only (8 bytes per double), Matlab has to allocate and to copy sum(1:1e6)*8 bytes, which are more than 4 TB ! The solution is easy: Allocate with the final size:

x = zeros(1, 1e6);   % Pre-allocation
for k = 1:1e6
   x(k) = rand;
end

In your case you do not know the needed size in advance. You tried to pre-allocate with data=zeros(90,90), but this is not the final size. You could determin the final size:

latmin  =  -90;
latmax  =   90;
longmin = -180;
longmax =  180;
index   = (x(:,8) >= longmin & x(:, 8) < longmax & x(:, 7) >= latmin && x(:,7) < latmax);
data    = zeros(sum(index), 13);
a       = 1;
for b = 1:size(x, 1)   % nicer and faster than: length(x(:,1))
   if index(b)
      for col = 1:13
          data(a, col) = x(b, col);
      end
      a = a + 1;
   end
end

But it is easier, nicer and faster to perform the copy directly without a loop:

latmin  =  -90;
latmax  =   90;
longmin = -180;
longmax =  180;
index   = (x(:,8) >= longmin & x(:, 8) < longmax & x(:, 7) >= latmin && x(:,7) < latmax);
data    = x(index, 1:13);