Saving data in background

10 次查看(过去 30 天)
Zack Fifer
Zack Fifer 2018-5-17
评论: Zack Fifer 2018-5-21
I am trying to save a large amount of data (2.5GB in a large uint8 array, as well as a much smaller 3MB struct full of a few different types including doubles and char arrays). Due to time limitations, I would rather save this data in the background while other parts of the code are running. I am using the parallel toolbox.
I first tried batch('simple_save_script') [this is literally an .m file with
save('full_path_str','v_large_array', 'strct','-v3.7')
in it..
The error that I get back reads:
Error using batch (line 179)
An unexpected error occurred accessing properties: "CaptureDiary" "CreateDateTime" "CreateTime" "DependentFiles" "Diary" "Error"
"ErrorIdentifier" "ErrorMessage" "FinishDateTime" "FinishTime" "Function" "InputArguments" "DiagnosticWarnings" "Name"
"NumOutputArguments" "OutputArguments" "StartDateTime" "StartTime" "StateEnum" "Worker"
Caused by:
Error using parallel.internal.cluster.FileSerializer>iSaveMat (line 281)
Data too large to be saved.
Well darn it.
I then switched to parfeval, since this ought to do exactly as I want. My code is:
P = parfeval(@save, 0, 'full_path', 'v_large_array', 'strct','-v7.3');
When I output P, I get the following error:
Error: Variable 'v_large_array' not found.
Okay, perhaps this is the size thing again... However, when I try the same code excluding 'v_large_array', I get the same error with 'strct'. In fact, I get the same error no matter what I try to pass into the parfeval function.
To see if it was the syntax that I wasn't understanding, I try the following code with feval:
feval(@save,'full_path', 'v_large_array', 'strct','-v7.3');
This runs as expected, and the data is saved.
So, what the heck can I do to save this data in the background? I have been trying to get this to work for several days now, and I have run out of ideas as this point...
Thank you in advance.

回答(2 个)

Edric Ellis
Edric Ellis 2018-5-18
I fear that handing the data off to another process simply to save it will not actually give you any benefit. The reason for that is you have to send the 2.5GB of data across to the worker, which is a separate process. Sending data to a worker is in some ways equivalent to saving it to a file, and then loading it again on the worker - except that instead of a file, the "saved" data is kept in memory. Anyway. The problem you're having with parfeval is that you are sending to the worker only the name of the array that you would like to be saved, and not the data. This is because the save command itself expects to find the named variable in the calling workspace. To work around this, you need to create a tiny wrapper function. Something like this:
function saveData(fname, data, varargin)
save(fname, 'data', varargin{:});
end
which then you can use like so:
parfeval(@saveData, 0, 'full_path', largeArray, '-v7.3')
  5 个评论
Walter Roberson
Walter Roberson 2018-5-20
parfeval(@save_funct, 0, v_large_array, strct)
Zack Fifer
Zack Fifer 2018-5-21
Sorry, just trying to increase visibility before Monday morning.
The '-v7.3' flag was a typo when writing this thread. This flag was not included in the actual parfeval call.

请先登录,再进行评论。


Walter Roberson
Walter Roberson 2018-5-20
Look in the File Exchange for Shared Matrix.

类别

Help CenterFile Exchange 中查找有关 Parallel for-Loops (parfor) 的更多信息

产品


版本

R2017b

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by