Save data during matlab loop

Question

0 个投票

Hello community!

Time for another question :)

My current project is a simulation run by matlab. So mainly it is a call of a "SimulateStep" function in which a lot of different things happen.

One of the things is the call of a huge mex function which gives me some internal data. So, during each call I have a relativly big struct that has a lot of data in it.

And my question is now: What is the most efficient way to save this data during each step and save it with the propriate time? Also, it should be easy to access and read it afterwards.

My current attemp is looking like that:

function saveData(this, time)
    % input data    
    namelist = fieldnames(this.structToSave);
    for it = 1: length(namelist)
        name = char(namelist(it));
        if ~isfield(this.timeSeries, name)
          this.timeSeries.(name) = timeseries;
        end
        this.timeSeries.(name) = addsample(this.timeSeries.(name), 'Data', this.structToSave.(name), 'Time', time);
    end
end

It works pretty good. At the end of the simulation I save the "timeSeries" struct to a workspace variable and that's it.

But this attemp is way to slow. If I have a list of ~30-40 signals to save you can heavily feel the simulation getting slower.

Does someone has any better alternative to that? Or is it only my programing that sucks here? ;)

Thanks a lot!

2 个评论
显示无隐藏无

Nitin Kapgate 2020-12-16

Is all the data in the structure of numeric type?

Simon Meyer 2020-12-16

编辑：Simon Meyer 2020-12-17

Yes, all the data is numeric. Mainly double, some integer.

There might be also boolean, but it's no problem to convert that to an integer.

But! The element of the struct can be a vector containing doubles.

请先登录，再进行评论。

请先登录，再回答此问题。

Follow Question

Answer 1

Walter Roberson 2020-12-17

编辑：Walter Roberson 2020-12-18

在 MATLAB Online 中打开

0 个投票

Performance during simulation is important, but performance after the simulation is not as important. So save quickly and process later

function saveData(this, time)
    if isfield(this, timeSeries)
        this.ntimeSeries = this.ntimeSeries + 1;
    else
        this.ntimeSeries = 1;
        this.timeSeries = cell(10000,2);       %pre-allocate
    end
    this.timeSeries(this.ntimeSeries,:) = {time, this.structToSave};
end

The 10000 pre-allocation is a total guess about how many simulation steps you need. It is best to over-estimate: the overhead for unused rows is only 16 bytes per row, whereas the overhead for growing the cell can be high. (The code can be tweaked for lower overhead for growth.)

After the execution, this.ntimeSeries will record the number of entries actually used, and this.timeSeries will be an N x 2 cell array in which the first column is time information and the second is the struct you wanted to save.

In post-processing you can run though and do the saving.

To do the post-processing saving, I would suggest that you fieldnames() of the second column (the structs) and unique(), and then for each unique field name, run through all of the cell rows that saved that field, accumulate the time information (first cell) in the big vector, accumulate the contents of the vectors (making sure to concatenate along an appropriate dimension) into a 2D array for that variable, and addsample() all of it at one go -- only one addsample() for each different variable.

Of course if you are always saving exactly the same list of variables each time, the logic would be easier.

2 个评论
显示无隐藏无

Simon Meyer 2020-12-17

编辑：Simon Meyer 2020-12-17

Hi, thanks for that answer!

The structure itself doesn't change during simulation, but it will change in between runs (the internal data is extendable pretty easily). I don't know how this will help to make the logic much easier, can you give any hints? Just the preprocessing, correct?

I'm giving myself a try for the postprocessing and will be back later! :)

Simon Meyer 2020-12-17

编辑：Simon Meyer 2020-12-17

在 MATLAB Online 中打开

Hello Walter,

thanks for your help, it works perfectly!

Here is my post-processing:

function my_timeseries = postProcessData(this)      
  time_vec = this.timeSeries(:,1);
  input_vec = this.timeSeries(:,2);
  internal_vec = this.timeSeries(:,3);
  output_vec = this.timeSeries(:,4);
  idx = cellfun(@isempty, time_vec);
  time_vec = time_vec(~idx);
  input_vec = input_vec(~idx);
  internal_vec = internal_vec(~idx);
  output_vec = output_vec(~idx);
  names = fieldnames(input_vec{1,1});
  for i = 1:numel(names)
    name = char(names(i));
    data = cell2mat(cellfun(@(s) s.(name), input_vec, 'UniformOutput',false));
    my_timeseries.input.(name) = timeseries(data, time_vec);
  end
  names = fieldnames(internal_vec{1,1});
  for i = 1:numel(names)
    name = char(names(i));
    data = cell2mat(cellfun(@(s) s.(name), internal_vec, 'UniformOutput',false));
    my_timeseries.internal.(name) = timeseries(data, time_vec);
  end
  names = fieldnames(output_vec{1,1});
  for i = 1:numel(names)
    name = char(names(i));
    data = cell2mat(cellfun(@(s) s.(name), output_vec, 'UniformOutput',false));
    my_timeseries.output.(name) = timeseries(data, time_vec);
  end
end

As you can see I saved multiple structs in the same line - I just made an easier example in the question above. Also, I modified the saving a little bit. I already have a counter in the simulation call - so I don't need the own counter. But this counter isn't available in the post processing anymore - that's why I'm filtering all data with the time information.

The output is exactly the same like in my previous attemp, but much faster.

During simulation the saving is not noticable and the post processing is - like you said - no problem because it does not matter. I havent measured, but I guess it is the range of only few seconds.

Thanks a again and have a merry christmas!

请先登录，再进行评论。

Answer 2

Nitin Kapgate 2020-12-16

0 个投票

As the data is numeric/boolean, I would suggest that you to write the data at every step or periodically to a file (say a CSV file with columns for different signals and rows for every step). That would help in increasing the RAM available for simulation and improve the simulation speed. At the simulation progresses, the growing size of logged data can consume huge amount of RAM and slow down simulations.

4 个评论
显示 2更早的评论隐藏 2更早的评论

Simon Meyer 2020-12-17

I wanted to give you an answer as well.

We tried to do the csv-file attemp too - but it was also way to slow. It doesn't matter how the data is given (raw, serialized, etc.) and which method is used to write (dlmwrite, fprint, etc.), it took way to long to do it every cycle.

But thanks anyway for that idea! :)

Nitin Kapgate 2020-12-18

I am glad that the issue is resolved.

请先登录，再进行评论。

Save data during matlab loop

2 个评论
显示无隐藏无

采纳的回答

2 个评论
显示无隐藏无

更多回答（1 个）

4 个评论
显示 2更早的评论隐藏 2更早的评论

类别

产品

版本

标签

Community Treasure Hunt

Save data during matlab loop

2 个评论 显示 无 隐藏 无

采纳的回答

2 个评论 显示 无 隐藏 无

更多回答（1 个）

4 个评论 显示 2更早的评论 隐藏 2更早的评论

类别

产品

版本

标签

另请参阅

Community Treasure Hunt

2 个评论
显示无隐藏无

2 个评论
显示无隐藏无

4 个评论
显示 2更早的评论隐藏 2更早的评论