Loop through large number of files and access data outside the loop

2 次查看（过去 30 天）
dipak sanap
dipak sanap 2015-11-23
评论： Walter Roberson 2015-11-23
I have to run this code over 50000 files which is around 250 GB so, I am also looking to improve performance.

请先登录，再回答此问题。

回答（2 个）

dipak sanap
dipak sanap 2015-11-23
编辑：Walter Roberson 2015-11-23
numfiles = 50000
for i = 1:numfiles
f = fopen(sprintf('F%d', i), 'r'); %File names are F1, F2 and son on.
X{i} = fscanf(f, '%d %d %f %d',[4, inf]); % Matlab nag about prellocating memory.
X{i} = X{i}';
A{i} = X(:,1:3); %I want to access only three columns of X{i} and make it as A(i) and access it outside loop. i.e make
fclose(f);
end
U = union(A(:,1:2),,'rows'); %Take union of first two colums of all A(i)
for j = 1:numfiles
Ua(j) = setdiff(U(:,1:2), A(:,1:2),'rows');%Again access A(i) from previous loop
Ua_z(j) = [Ua(j) zeros(size(Ua,1),1)]; % Add zero column to Ua(j)
AU(j) = [A(i) ; Ua_z(j)]; % Vertically concatenate A(i) and Ua_z(j)
AU_sorted(j) = sortrows(AU); %Sort rows of AU(j)
end
C = [U(:,1:2), AU_sorted(:,3)]; % Horizontally concatenate U and AU_sorted(j)
Walter Roberson
Walter Roberson 2015-11-23
numfiles = 50000
A = cell{numfiles,1};
for i = 1:numfiles
f = fopen(sprintf('F%d', i), 'r'); %File names are F1, F2 and son on.
A{i} = fscanf(f, '%d %d %f %*d',[3, inf]) .';
fclose(f);
end
You do not use column 4, so tell fscanf that it exists but that no value is to be returned for it. With this done you do not need the temporary variable X.
  3 个评论
显示 1更早的评论
Walter Roberson
Walter Roberson 2015-11-23
In your existing code, you were creating X temporarily and using A outside the loop. Now you say that you got rid of A and want to use X outside the loop. Your existing code does not use X outside the loop, only A.
Walter Roberson
Walter Roberson 2015-11-23
U = union( cell2mat(cellfun(@(M) M(:,1:2), A(:), 'Uniform', 0)), 'rows');
However, I would suggest
U = unique( cell2mat(cellfun(@(M) unique(M(:,1:2), 'rows'), A(:), 'Uniform', 0)), 'rows');
However, if you know that those rows are unique within each file, then you might as well use the first command. If those rows are not unique within each file then you can save a lot of memory by taking the unique values by file before merging them all together.

请先登录，再进行评论。

类别

在 Help Center 和 File Exchange 中查找有关 Data Type Conversion 的更多信息

标签

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by