How do I efficiently stitch together logged data from several v4 MAT files into a huge structure with timeseries data?

4 次查看（过去 30 天）
Michael
Michael 2018-6-7
编辑： Michael 2018-6-11
I am running a large Simulink model for a long time and need to save the data of a huge multi-level hierarchical bus. The To File block has some limitations, so I've written my own similar block. It logs data to v4 MAT files because it is a very easy (and fast) way to write a simple header and then stream an arbitrary amount of data to a file. However, like all MAT file versions prior to v7.3 files, the size is limited. I need to save roughly 12GB of data so I save it in chunks. So each .mat file first gets a variable with a character array that specifies the full list of bus leaf nodes in a comma-separated list so that the bus hierarchy can later be rebuilt into a structure where each leaf node is a timeseries. Then each .mat file gets a single MxN 2-D array where M is the number of leaf nodes (plus one for simulation time) and N is a number of data points that I can safely stuff in each .mat file without hitting the max file size or max elements per array. When each .mat file is "full", then I close it (writing the value of N to the header since it wasn't known initially) and start working on a new file.
The above process works nicely for the logging itself but there are a few very slow and memory expensive operations to post-process the data. The first is stitching the data together. Right now, I just go through the files one at a time and chunk them together like this:
data_varTMP = [data_varTMP data_var__];
where data_var__ is the MxN array in each file. Once I have the entire huge array stitched together then I build the hierarchical structure variable making each leaf node a timeseries using this:
varNameCellArray__ = textscan(commaSepSigNames_var__,'%s','delimiter',',');
varNameCellArray__ = varNameCellArray__{1};
simtime__ = data_var__(1,:);
for idxVar__=2:numel(varNameCellArray__)
tmpAvarName__=varNameCellArray__{idxVar__};
eval([fileName_var__ '.' tmpAvarName__ ' = timeseries(data_var__(' num2str(idxVar__) ',:)'', simtime__,''Name'', tmpAvarName__ );']);
end
This too takes a long time. The final step, which also takes a long time, is to save my final structure variable as a v7.3 MAT file.
I expect that there are better ways to handle some of these slow steps but I am not sure the best approach. I am thinking the matfile function might help, but it talks about only v7.3 files being efficient and not being able to work with fields within structures, I'm not sure how I could leverage it best.

请先登录，再回答此问题。

回答（0 个）

类别

在 Help Center 和 File Exchange 中查找有关 Sources 的更多信息

标签

产品


版本

R2018a

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by