How can i save a large structure array without performace issues?

8 次查看（过去 30 天）
Stefan Wakolbinger
回答： Stefan Wakolbinger 2020-9-23
Hello,
we are importing data with an .Net-dll into Matlab. The data will be saved in a structure like this.
for measure_id = 1:length(messungen)
...
for signal = 1:signal_names.Length
signal_name = signal_names(signal);
...
for RPC_Nr = 1:step.Length
for shot_idx ...
M(measure_id).ETCurve(ETC_Nr).(sprintf('%s', char(signal_name)))(shot_idx).data = DATA
end
M(measure_id).RPCurve(RPC_Nr).(sprintf('%s_t', char(signal_name))) = DATA
...
end
end
...
if measure_id < length(messungen)
save('data.mat', 'M', '-v7.3')
end
end
There are 6 different structure types for the "first level", like M(1).ETCurve, M(1).RPCurve, M(1).TP, ...
With this structure it is easy to acces the needed data like M(1).RPCurve(2).Q or M(1).ETCurve(2).U(100).data and so on.
One measurement has about 0,5 to 1,5 GB of data, when saved to a mat-file. After the 4th measurement saving and loading the workspace will take some time and after the 6th measurement it will take more than 30 minutes...
Has anyone an idea, wyh this will get so slow and can i improve the performance.
Tanks
  5 个评论
显示 3更早的评论
per isakson
per isakson 2020-9-8
To me it looks like you overwrite the variable named M in every iteration.
If each variable value is less than 2GB version, v7, is a faster alternative than v7.3
Stefan Wakolbinger
Stefan Wakolbinger 2020-9-23
Thank you per isakson.
I changed the whole import stuff to the variable M. Before saving i create a variable M1, M2, M3, ... instead of overwriting the list of M(:).
This is much faster and has a linear behavior with the number of measurements.
...
M.RPCurve(RPC_Nr).(sprintf('%s_t', char(signal_name))) = DATA
...
varName = sprintf('M%d', measure_idx);
eval([varName ' = M;'])
tic
if exist(workspace, 'file') == 2
save(workspace, varName, '-append')
else
save(workspace, varName, '-v7.3')
end
toc

请先登录，再进行评论。

请先登录，再回答此问题。

采纳的回答

Stefan Wakolbinger
Stefan Wakolbinger 2020-9-23
I changed the whole import stuff to the variable M. Before saving i create a variable M1, M2, M3, ... and append this variable to the workspace, instead of overwriting the list of M(:).
This is much faster.
% import stuff
...
M.RPCurve(RPC_Nr).(sprintf('%s_t', char(signal_name))) = DATA
...
% save measurement
varName = sprintf('M%d', measure_idx);
eval([varName ' = M;'])
if exist(workspace, 'file') == 2
save(workspace, varName, '-append')
else
save(workspace, varName, '-v7.3')
end

更多回答（0 个）

类别

在 Help Center 和 File Exchange 中查找有关 Loops and Conditional Statements 的更多信息

标签

产品


版本

R2018b

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by