MATLAB Answers

How do I use SAVE with a PARFOR loop using Parallel Computing Toolbox?

461 次查看（过去 30 天）
显示 更早的评论
MathWorks Support Team
MathWorks Support Team ，2014-6-18
  • 0
回答： Muh Alam ，2022-3-1
I would like to save some variables to MAT files from inside a PARFOR loop. However I get an error:
??? Error using ==> parallel_function at 598
Error in ==> parallel_function>make_general_channel/channel_general at 894
Transparency violation error.
See Parallel Computing Toolbox documentation about Transparency.

请先登录，再回答此问题。

采纳的回答

MathWorks Support Team
编辑：MathWorks Support Team 2020-2-24
Transparency is violated by the SAVE command because in general MATLAB cannot determine which variables from the workspace will be saved to a file.
The solution is to move the call to SAVE to a separate function and to call that function from inside the PARFOR loop. Pass any variables that are to be saved as arguments to the function. That way MATLAB can determine which ones will be saved and transparency is not violated.
For example:
Save the following as "parsave.m":
function parsave(fname, x,y)
save(fname, 'x', 'y')
end
Then run it with:
parfor ii = 1:4
x = rand(10,10);
y = ones(1,3);
parsave(sprintf('output%d.mat', ii), x, y);
end
Similarly, for loading several files with enumerated file names, we can use the following code.
function [output] = par_load(i)
eval(['load file_' num2str(i) '.mat']);
output = time;
end
Which can be run with:
parfor i=1:4
data = par_load(i);
a(i) = 10 + data(1);
end
  5 个评论
显示 4更早的评论

请先登录，再进行评论。

更多回答（7 个）

Daniel
Daniel 2016-3-14
A nice alternative to implement your own save function is using the built-in matfile function:
parfor ii = 1:4
m=matfile(sprintf('output%d.mat', ii),'writable',true)
x = rand(10,10);
y = ones(1,3);
m.x=x;
m.y=y;
end
Here the parser has no issue understanding which variables are used.
Tihomir Kostadinov
Tihomir Kostadinov 2014-8-22
编辑：Tihomir Kostadinov 2014-8-22
I have enhanced the function that allows saving within a parfor loop as follows. This should allow giving it a variable number of arguments and the function recognizes their names. Let me know if there are problems with this implementation.
function save_for_parfor(fname,numvars,varargin)
for i = 1:numvars
eval([inputname(i+2),' = varargin{i};']);
end
save('-mat',fname,inputname(3));
for i = 2:numvars
save('-mat',fname,inputname(i+2),'-append');
end
This function would be called as follows:
>> x = 3; y = 4; z = 5; g = 'tester_parfor';
>> save_for_parfor('tester.mat',4,x,y,z,g)
  1 个评论
Walter Roberson
Walter Roberson 2016-3-13
Don't use eval(). Store the variables in a struct and use save with the -struct option. No -append is needed.

请先登录，再进行评论。

Qifeng Chen
Qifeng Chen 2014-11-30
This is a better implementation based on Tihomir. The code here does not require the input of "numvars".
function parsave(fname,varargin)
numvars=numel(varargin);
for i=1:numvars
eval([inputname(i+1),'=varargin{i};']);
end
save('-mat',fname,inputname(2));
for i = 2:numvars
save('-mat',fname,inputname(i+1),'-append');
end
end
  5 个评论
显示 4更早的评论

请先登录，再进行评论。

Alex Shaw
Alex Shaw 2015-12-9
编辑：Alex Shaw 2015-12-9
I encountered this issue, and used the function proposed by Qifeng Chen.
This worked very well in 2015a.
However, since upgrading to 2015b my code reports a transparancy violation again. The release notes state that transparancy is now enforced more strictly, however all variables that I attempt to save either appear as inputs in expression in the parfor loop, or are the results of calculation in the parfor loop so I think this may be a bug.
Further to this, the affected function seems to be inputname(). The simplest (and probably fastest) way to work round this is to add all variables to a single structure variable, and then save that with a fixed name:
function parsave_simple(fname,C )
save('-mat',fname,'C');
end
however, this would have been a lot of surgery in my case, plus other code expects the variables not to be in structure format. So the 'minimal surgery' option in my case is simply to pass the names of variables into the function via a cell array:
function parsave_named(fname____,vnames___,varargin)
numvars___=numel(varargin);
for ii___=1:numvars___
eval([vnames___{ii___},'=varargin{ii___};']);
end
save('-mat',fname____,vnames___{1});
for ii___ = 2:numvars___
save('-mat',fname____,vnames___{ii___},'-append');
end
end
This method seems a bit messy, there is a performance overhead to calling 'eval' and some very hard to trace bugs will arise if names do not match the actual variables. Also note the use of strange var names to ensure eval does not overwrite any internal logic! But on the other hand, it only needs one extra line in the calling loop to populate the names variable.
  2 个评论
显示 1更早的评论
echo zhang
echo zhang 2019-4-11
This works for me, and a working example below:
parfor ii = 1:n
x=1;
y=2;
z=3;
var_name = ["x","y","z"];
parsave('whatever.mat',var_name,x,y,z);
end

请先登录，再进行评论。

Anand
Anand 2016-3-13
编辑：Anand 2016-3-13
Another addition to various versions of doing this. This one does not use a structure. It, IMO, combines the best features of all answers above.
function parfor_save(varargin)
fname=varargin{1};
for i=2:nargin
eval([inputname(i),'=varargin{i};']);
if i==2
save('-mat',fname,inputname(i));
else
save('-mat',fname,inputname(i),'-append');
end
end
  1 个评论
Walter Roberson
Walter Roberson 2016-3-13
Don't use eval(). Store the variables in a struct and use save with the -struct option. No -append is needed.

请先登录，再进行评论。

CrossEntropy
CrossEntropy 2019-12-3
The solution that works for me, on R2018a, is to define a function which includes the entire for loop, and then use save inside that function. Anything else I tried among the solutions in here id not work for me.
Muh Alam
Muh Alam 2022-3-1
This answer is similar to previous answers and comments but does not use eval to avoid any possible troubles and weird outputs and simplify the code.
function par_save(file_name, varargin)
%-----------inputs-----------:
%file_name; name of .matfile to be saved
%varargin{}: variables needed to be saved to the matfile named in file_name.
ct ={};%to be converted to structure as struct(field1,value1,...,fieldN,valueN)
fi=1;% initial field index
vi=2;% = value index
for i=1:numel(varargin)
ct{fi} = inputname(i+1);%+1 since the 1st input is filename
ct{vi} =varargin{i};
fi=fi+2;
vi=vi+2;
end
s = struct(ct{:}) ;% put inputs in a structure with fields that has same name as the inputs
save(file_name,'-v7.3', '-struct','s','-nocompression')
end
  • you might use default flags at save function but if you know that you are going to save large variables (>2GB) you need to set the version to v7.3 otherwise matlab will not save the variable (it will issue warning).
  • nocompression flag is optional as well but I found that it is making save faster for saving very large matrices and that might also faster if you use matfile function to partially load variables you saved (also requires v7.3 ).

类别

标签

产品


版本

R2010a

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by