Append new values from a streaming time series to a saved .mat file?
5 次查看(过去 30 天)
显示 更早的评论
Hi.
I have data continuously coming in to matlab every few seconds that I want to save to a file with as little delay as possible so that other computers can quickly access the data.
The way I have it set up right now the data gets stored in a matrix and saves it to the computer and then the loop starts over and the matrix gets one value bigger and saves it all over again and so on and so on. This works fine until I start reaching a size of a few thousand values, naturally it then starts to slow down since its writing bigger and bigger files to the hard drive. And after a while the lags are so great that the approach becomes useless. I’m assuming the save command overwrites all the old data every loop without acknowledging the duplicates and that’s what’s taking so long. Is there any way to get around this?
How is logging data with matlab usually done, is it possible to make the save command only focus on adding the last arriving value to an existing .mat file without overwriting the whole thing every iteration?
0 个评论
采纳的回答
Jan
2015-5-26
The problem does not only concern the writing to the hard disk, but creating a large array which grows iteratively consumes a lot of resources also:
v = [];
for k = 1:1e6
v(k) = 1;
end
This does not only write 1e6 ones to the memory. In the first step [1] is created, in the 2nd step [0,0] is created at first, than [1] is copied to the first element, the original array is freed and the next 1 is inserted as last element. Finally you do not reserve and write the memory for 1e6 elements, but for sum(1:1e6) elements: > 250 GB!
A pre-allocation solves the problem:
v = zeros(1, 1e6);
for k = 1:1e6
v(k) = 1;
end
Now there is no re-allocation and data copy in each iteration. And similar problems concerns writing to the disk.
The best solution is either to create a memory mapped file or to open a binary file and append the new data:
fid = fopen('DataFile.dat', 'w');
if fid == -1, error('Cannot create file.'); end
for k = 1:1000
newData = rand(1, 1000);
fwrite(fid, newData, 'double');
end
fclose(fid);
4 个评论
want2know
2015-12-18
Hi PetterS, may I know how did solve your problem eventually? I followed this thread and have the same issue...
更多回答(0 个)
另请参阅
类别
在 Help Center 和 File Exchange 中查找有关 Whos 的更多信息
产品
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!