Fastest way to save and retrieve floating point vectors

2 次查看(过去 30 天)
I am writing a code that generates a row vector of size 1x600 (containing floating point numbers) in each iteration. There are about 100,000 such iterations performed in the entire code. Also, I have to read these saved vectors (row wise) in a different code at a later point of time. I have been using 'dlmwrite' so far for appending the vectors row wise to a .csv file, and the function 'load' for retrieving this data. I found that saving and retrieving data in this way is consuming a lot of time. I wanted to know if there is a better way of saving and retrieving these vectors.
I have read some posts in the forum suggesting to save the data in a binary file/.mat file as some alternatives. But I am not sure which would be the best for my case. Could someone please suggest me a faster way to do the above exercise?

采纳的回答

Walter Roberson
Walter Roberson 2021-3-11
Simplest way, that will take very low overhead for your program (such as might be needed for real-time work):
fid = fopen(OutputFileName, 'w'); %not 'wt'
for ...
fwrite(fid, YourBufferOfDoubles);
end
fclose(fid)
However, you have about 480 megabytes worth of raw data, and that is enough that unless you are writing to SSD, that the file overhead is going to add up.
Because of that, you might want to consider methods that compress data as you go. You would buffer some data and call compression routines sometimes, and write out the compressed data. One way to do that would be https://docs.oracle.com/javase/7/docs/api/java/util/zip/GZIPOutputStream.html
A third way... just store all the values in memory as you produce them (having pre-allocated the array), and afterwards save() them to a .mat file. .mat does compression.
  2 个评论
SKP
SKP 2021-3-11
Thank you very much for this prompt answer. I shall try out these methods and get back to you.
SKP
SKP 2021-3-15
The suggested methods are very time efficient and occupy lesser memory compared to the ones I was previously using. Thank you.

请先登录,再进行评论。

更多回答(0 个)

类别

Help CenterFile Exchange 中查找有关 Data Import and Export 的更多信息

产品


版本

R2020b

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by