Fastest way to save and retrieve floating point vectors
2 次查看(过去 30 天)
显示 更早的评论
I am writing a code that generates a row vector of size 1x600 (containing floating point numbers) in each iteration. There are about 100,000 such iterations performed in the entire code. Also, I have to read these saved vectors (row wise) in a different code at a later point of time. I have been using 'dlmwrite' so far for appending the vectors row wise to a .csv file, and the function 'load' for retrieving this data. I found that saving and retrieving data in this way is consuming a lot of time. I wanted to know if there is a better way of saving and retrieving these vectors.
I have read some posts in the forum suggesting to save the data in a binary file/.mat file as some alternatives. But I am not sure which would be the best for my case. Could someone please suggest me a faster way to do the above exercise?
0 个评论
采纳的回答
Walter Roberson
2021-3-11
Simplest way, that will take very low overhead for your program (such as might be needed for real-time work):
fid = fopen(OutputFileName, 'w'); %not 'wt'
for ...
fwrite(fid, YourBufferOfDoubles);
end
fclose(fid)
However, you have about 480 megabytes worth of raw data, and that is enough that unless you are writing to SSD, that the file overhead is going to add up.
Because of that, you might want to consider methods that compress data as you go. You would buffer some data and call compression routines sometimes, and write out the compressed data. One way to do that would be https://docs.oracle.com/javase/7/docs/api/java/util/zip/GZIPOutputStream.html
A third way... just store all the values in memory as you produce them (having pre-allocated the array), and afterwards save() them to a .mat file. .mat does compression.
更多回答(0 个)
另请参阅
类别
在 Help Center 和 File Exchange 中查找有关 Data Import and Export 的更多信息
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!