Append data to matfile using parallel method

1 次查看（过去 30 天）
Yu Li
Yu Li 2018-9-12
评论： Walter Roberson 2018-9-13
Hi:
I have a lots of data that needs to be saved into a test.mat file, below are my test code:
x=rand(10000,1);
save('test.mat','x');
for i=1:1:100
eval(['va_',num2str(i),'=rand(10000,1);'])
eval(['save(','''','test.mat','''',',','''','va_',num2str(i),'''',',','''','-append','''',')'])
end
the problem is that, this is a test code, in my real situation:
1. the number of variables is very large in my situation (up to va_10000).
2. the size of data of each 'va_i' is very large (up to size of 2e6*1).
in this way, although I have upgrade my drive into 960EVO ssd, the saving time is still significantly large.
is there anyway to improve the code into parallel saving? so that I could save the computational cost?
Thanks!
Yu
  6 个评论
显示 4更早的评论
Walter Roberson
Walter Roberson 2018-9-13
Is the size and data type of each variable the same?
Is the data likely to be compressible?
Yu Li
Yu Li 2018-9-13
Data size are not the same, data are not compressible. I have to save them in the same matfile for future reading.

请先登录，再进行评论。

请先登录，再回答此问题。

回答（2 个）

Steven Lord
Steven Lord 2018-9-13
Consider writing each variable to a different file in such a way that when you want to use them later on you can construct a datastore using that collection of files and make a tall array from the datastore.
  1 个评论
Yu Li
Yu Li 2018-9-13
My memory is enough for these data, I need to append these data in a given matfile for future reading.

请先登录，再进行评论。

Walter Roberson
Walter Roberson 2018-9-13
You cannot write to a mat file in parallel. If writing in parallel to a mat file is a requirement then your problem cannot be solved.
If computation of the items is expensive, then do the computation in parallel, writing to different mat files (though potentially one per parallel core rather than one per variable.) Afterwards, merge the files together in a serial loop.
With the data not being compressible, either write in binary or else use the -7.3 option to not compress the output.
  2 个评论
Yu Li
Yu Li 2018-9-13
If save these files with their seperate name using parallel, and merge them, the merging process requires load these data into memory and re-save them. Which may increase the overall saving time.
Or do you have any other ways to merge these data?
Walter Roberson
Walter Roberson 2018-9-13
Overall saving time might not increase under the assumption that calculation of the array is expensive. If the average rate of graduation is less than the time required to save one variable then parfor for the calculation and merging afterwards can potentially save time.
Another approach in the case where calculations are expensive is to use a pollable data queue to calculate results in parallel and send them back to the client process to do the saving.
If the average rate of graduation is faster than the time to save one variable then you are probably bandwidth limited in writing to the ssd, and increasing the number of simultaneous writers will not increase the bandwidth.

请先登录，再进行评论。

类别

在 Help Center 和 File Exchange 中查找有关 Timetables 的更多信息

标签

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by