How to Concatenate struct arrays efficiently

15 次查看（过去 30 天）
Kenneth Yesh
Kenneth Yesh 2018-5-24
评论： Kenneth Yesh 2018-5-25
I use parallel processing to run roughly 35,000 jobs in parallel. They output a .mat file and I want to concatenate them into one big file.
The process takes too long to run. (I am waiting for it to run while writing this) Is there a more optimal way to concatenate struct arrays so this doesn't take hours to run?
I'm guess the the limiting factor is reallocating "finalData".
Note: outData is a struct array of n elements
inputs = dir([intputDir,filesep,'*.mat']);
finalData = [];
if ~exist([outputDir],'dir')
mkdir([outputDir]);
end
l=length(inputs);
for i = 1:l
load([inputs(i).folder,filesep,inputs(i).name]);
finalData = [finalData,outData];
if (mod(i,100) == 0)
fprintf('Finished .Mat File # %d of %d',i,l);
end
end
save([outputDir,filesep,'allOutputs'],'finalData');
struct2xls(finalData,[outputDir,filesep,'Timing_Analysis_Data.xlsx']);
  2 个评论
James Tursa
James Tursa 2018-5-24
编辑：James Tursa 2018-5-24
Load into variables inside the loop, and then concatenate all of the variables after the loop. Rough outline:
c = cell(l,1);
for i = 1:l
c{i} = load([inputs(i).folder,filesep,inputs(i).name]);
if (mod(i,100) == 0)
fprintf('Finished .Mat File # %d of %d',i,l);
end
end
% then concatenate all of the c{:} here
Do all of your individual struct arrays have the same fields?
Kenneth Yesh
Kenneth Yesh 2018-5-25
Yes, they all have the same fields so they concatenate without any issue. I just thought it was taking way to long to run.

请先登录，再进行评论。

请先登录，再回答此问题。

采纳的回答

Stephen23
Stephen23 2018-5-25
编辑：Stephen23 2018-5-25
Your very slow code is due to that your variable finalData changes size on every loop iteration, forcing MATLAB to move it in memory. Read this to know more:
https://www.mathworks.com/help/matlab/matlab_prog/preallocating-arrays.html
It is easy to import that data into one preallocated structure, something like this:
D = dir(fullfile(inputDir,'*.mat'));
N = 1:numel(D);
S = struct('outData',cell(N,1));
for k = 1:N
F = fullfile(inputDir,D(k).name);
S(k) = load(F,'outData');
end
Data = [S.outData];
save('newfile.mat','Data')
Note it is recommended to use fullfile rather than fiddling around with file separators and string concatenation.

更多回答（0 个）

类别

在 Help Center 和 File Exchange 中查找有关 Mathematics 的更多信息

标签

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by