MATLAB Answers

0

How do I use SAVE with a PARFOR loop using Parallel Computing Toolbox?

I would like to save some variables to MAT files from inside a PARFOR loop. However I get an error:
??? Error using ==> parallel_function at 598
Error in ==> parallel_function>make_general_channel/channel_general at 894
Transparency violation error.
See Parallel Computing Toolbox documentation about Transparency.

Tags

Products


Release

R2010a

6 Answers

Answer by MathWorks Support Team on 18 Jun 2014
 Accepted Answer

Transparency is violated by the SAVE command because in general MATLAB cannot determine which variables from the workspace will be saved to a file.
The solution is to move the call to SAVE to a separate function and to call that function from inside the PARFOR loop. Pass any variables that are to be saved as arguments to the function. That way MATLAB can determine which ones will be saved and transparency is not violated.
For example:
Save the following as "parsave.m":
function parsave(fname, x,y)
save(fname, 'x', 'y')
end
Then run it with:
parfor ii = 1:4
x = rand(10,10);
y = ones(1,3);
parsave(sprintf('output%d.mat', ii), x, y);
end

  4 Comments

Show 1 older comment
I am now using R2016b and the workaround is working again
This parsave method works in 2018a. How about loading different files under different parfor loop iterations, any similar method?

Sign in to comment.


Answer by Daniel
on 14 Mar 2016

A nice alternative to implement your own save function is using the built-in matfile function:
parfor ii = 1:4
m=matfile(sprintf('output%d.mat', ii),'writable',true)
x = rand(10,10);
y = ones(1,3);
m.x=x;
m.y=y;
end
Here the parser has no issue understanding which variables are used.

  2 Comments

Neither of the top-2 answers seems to work for 2015b
This works perfectly for my on Matlab2018a.

Sign in to comment.


Answer by Tihomir Kostadinov on 22 Aug 2014
Edited by Tihomir Kostadinov on 22 Aug 2014

I have enhanced the function that allows saving within a parfor loop as follows. This should allow giving it a variable number of arguments and the function recognizes their names. Let me know if there are problems with this implementation.
function save_for_parfor(fname,numvars,varargin)
for i = 1:numvars
eval([inputname(i+2),' = varargin{i};']);
end
save('-mat',fname,inputname(3));
for i = 2:numvars
save('-mat',fname,inputname(i+2),'-append');
end
This function would be called as follows:
>> x = 3; y = 4; z = 5; g = 'tester_parfor';
>> save_for_parfor('tester.mat',4,x,y,z,g)

  1 Comment

Don't use eval(). Store the variables in a struct and use save with the -struct option. No -append is needed.

Sign in to comment.


Answer by Qifeng Chen on 30 Nov 2014

This is a better implementation based on Tihomir. The code here does not require the input of "numvars".
function parsave(fname,varargin)
numvars=numel(varargin);
for i=1:numvars
eval([inputname(i+1),'=varargin{i};']);
end
save('-mat',fname,inputname(2));
for i = 2:numvars
save('-mat',fname,inputname(i+1),'-append');
end
end

  5 Comments

Erik's solution still give me a transparency violation error in Line 6 (savevar...)
This still gives an error in Matlab R2016a: the variable I'm trying to save cannot be classified. Same with Alex Shaw's version below. Any ideas? Here is my code:
function test
var_to_save = NaN(10,1);
parfor ii = 1:10
var_to_save(ii,:) = rand;
parsave('test.mat',var_to_save)
end
end
function parsave(varargin)
savefile = varargin{1}; % first input argument
for i = 2:nargin
savevar.(inputname(i)) = varargin{i}; % other input arguments
end
save(savefile,'-struct','savevar')
end
Please contact MathWorks Technical Support.

Sign in to comment.


Answer by Alex Shaw on 9 Dec 2015
Edited by Alex Shaw on 9 Dec 2015

I encountered this issue, and used the function proposed by Qifeng Chen.
This worked very well in 2015a.
However, since upgrading to 2015b my code reports a transparancy violation again. The release notes state that transparancy is now enforced more strictly, however all variables that I attempt to save either appear as inputs in expression in the parfor loop, or are the results of calculation in the parfor loop so I think this may be a bug.
Further to this, the affected function seems to be inputname(). The simplest (and probably fastest) way to work round this is to add all variables to a single structure variable, and then save that with a fixed name:
function parsave_simple(fname,C )
save('-mat',fname,'C');
end
however, this would have been a lot of surgery in my case, plus other code expects the variables not to be in structure format. So the 'minimal surgery' option in my case is simply to pass the names of variables into the function via a cell array:
function parsave_named(fname____,vnames___,varargin)
numvars___=numel(varargin);
for ii___=1:numvars___
eval([vnames___{ii___},'=varargin{ii___};']);
end
save('-mat',fname____,vnames___{1});
for ii___ = 2:numvars___
save('-mat',fname____,vnames___{ii___},'-append');
end
end
This method seems a bit messy, there is a performance overhead to calling 'eval' and some very hard to trace bugs will arise if names do not match the actual variables. Also note the use of strange var names to ensure eval does not overwrite any internal logic! But on the other hand, it only needs one extra line in the calling loop to populate the names variable.

  2 Comments

You can use the -struct option when you save the structure.
This works for me, and a working example below:
parfor ii = 1:n
x=1;
y=2;
z=3;
var_name = ["x","y","z"];
parsave('whatever.mat',var_name,x,y,z);
end

Sign in to comment.


Answer by Anand
on 13 Mar 2016
Edited by Anand
on 13 Mar 2016

Another addition to various versions of doing this. This one does not use a structure. It, IMO, combines the best features of all answers above.
function parfor_save(varargin)
fname=varargin{1};
for i=2:nargin
eval([inputname(i),'=varargin{i};']);
if i==2
save('-mat',fname,inputname(i));
else
save('-mat',fname,inputname(i),'-append');
end
end

  1 Comment

Don't use eval(). Store the variables in a struct and use save with the -struct option. No -append is needed.

Sign in to comment.