Loop through large number of files and access data outside the loop

2015 11 23
2 个回答
更新时间：2015 11 23
3 次查看（30 天）

0 个投票

I have to run this code over 50000 files which is around 250 GB so, I am also looking to improve performance.

请先登录，再回答此问题。

回答（2 个）

dipak sanap
dipak sanap 2015-11-23
编辑：Walter Roberson 2015-11-23

0 个投票

numfiles = 50000
for i = 1:numfiles
f = fopen(sprintf('F%d', i), 'r'); %File names are F1, F2 and son on.
X{i} = fscanf(f, '%d %d %f %d',[4, inf]); % Matlab nag about prellocating memory.
X{i} = X{i}';
A{i} = X(:,1:3); %I want to access only three columns of X{i} and make it as A(i) and access it outside loop. i.e make
fclose(f);
end
U = union(A(:,1:2),,'rows'); %Take union of first two colums of all A(i)
for j = 1:numfiles
Ua(j) = setdiff(U(:,1:2), A(:,1:2),'rows');%Again access A(i) from previous loop
Ua_z(j) = [Ua(j) zeros(size(Ua,1),1)]; % Add zero column to Ua(j)
AU(j) = [A(i) ; Ua_z(j)]; % Vertically concatenate A(i) and Ua_z(j)
AU_sorted(j) = sortrows(AU); %Sort rows of AU(j)
end
C = [U(:,1:2), AU_sorted(:,3)]; % Horizontally concatenate U and AU_sorted(j)
Walter Roberson
Walter Roberson 2015-11-23

0 个投票

numfiles = 50000
A = cell{numfiles,1};
for i = 1:numfiles
f = fopen(sprintf('F%d', i), 'r'); %File names are F1, F2 and son on.
A{i} = fscanf(f, '%d %d %f %*d',[3, inf]) .';
fclose(f);
end
You do not use column 4, so tell fscanf that it exists but that no value is to be returned for it. With this done you do not need the temporary variable X.

3 个评论
显示 1更早的评论 隐藏 1更早的评论

dipak sanap
dipak sanap 2015-11-23
Hey, thanks for a quick reply. As you suggested I got rid of A(i) temporary variable. Now I want to access X(i)s outside the loop to take union ?
Walter Roberson
Walter Roberson 2015-11-23
In your existing code, you were creating X temporarily and using A outside the loop. Now you say that you got rid of A and want to use X outside the loop. Your existing code does not use X outside the loop, only A.
Walter Roberson
Walter Roberson 2015-11-23
U = union( cell2mat(cellfun(@(M) M(:,1:2), A(:), 'Uniform', 0)), 'rows');
However, I would suggest
U = unique( cell2mat(cellfun(@(M) unique(M(:,1:2), 'rows'), A(:), 'Uniform', 0)), 'rows');
However, if you know that those rows are unique within each file, then you might as well use the first command. If those rows are not unique within each file then you can save a lot of memory by taking the unique values by file before merging them all together.

请先登录，再进行评论。

类别

在 帮助中心 和 File Exchange 中查找有关 Performance and Memory 的更多信息

标签

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by