How do I use SAVE with a PARFOR loop using Parallel Computing Toolbox?

Asked by MathWorks Support Team on 18 Jun 2014

Latest activity Commented on by echo zhang on 11 Apr 2019

Accepted Answer by MathWorks Support Team

396 views (last 30 days)

I would like to save some variables to MAT files from inside a PARFOR loop. However I get an error:

??? Error using ==> parallel_function at 598

Error in ==> parallel_function>make_general_channel/channel_general at 894

Transparency violation error.

See Parallel Computing Toolbox documentation about Transparency.

Products

Parallel Computing Toolbox

Release

R2010a

6 Answers

Link

Answer by MathWorks Support Team on 18 Jun 2014

Accepted Answer

Transparency is violated by the SAVE command because in general MATLAB cannot determine which variables from the workspace will be saved to a file.

The solution is to move the call to SAVE to a separate function and to call that function from inside the PARFOR loop. Pass any variables that are to be saved as arguments to the function. That way MATLAB can determine which ones will be saved and transparency is not violated.

For example:

Save the following as "parsave.m":

function parsave(fname, x,y)
save(fname, 'x', 'y')
end

Then run it with:

parfor ii = 1:4
x = rand(10,10);
y = ones(1,3);
parsave(sprintf('output%d.mat', ii), x, y);
end

4 Comments

Show 1 older comment

Walter Roberson on 30 Apr 2016

Notice Alex Shaw's discussion, below

Hazem on 13 Dec 2016

I am now using R2016b and the workaround is working again

Dave Lee on 22 Dec 2018

This parsave method works in 2018a. How about loading different files under different parfor loop iterations, any similar method?

Link

Answer by Daniel on 14 Mar 2016

A nice alternative to implement your own save function is using the built-in matfile function:

parfor ii = 1:4
m=matfile(sprintf('output%d.mat', ii),'writable',true)
x = rand(10,10);
y = ones(1,3);
m.x=x;
m.y=y;
end

Here the parser has no issue understanding which variables are used.

2 Comments

Shan Dou on 7 Jun 2017

Neither of the top-2 answers seems to work for 2015b

Duc Truong on 22 Jun 2018

This works perfectly for my on Matlab2018a.

Link

Answer by Tihomir Kostadinov on 22 Aug 2014

Edited by Tihomir Kostadinov on 22 Aug 2014

I have enhanced the function that allows saving within a parfor loop as follows. This should allow giving it a variable number of arguments and the function recognizes their names. Let me know if there are problems with this implementation.

function save_for_parfor(fname,numvars,varargin)
for i = 1:numvars
   eval([inputname(i+2),' = varargin{i};']);  
end
save('-mat',fname,inputname(3));
for i = 2:numvars    
    save('-mat',fname,inputname(i+2),'-append');
end

This function would be called as follows:

>> x = 3; y = 4; z = 5; g = 'tester_parfor';
>> save_for_parfor('tester.mat',4,x,y,z,g)

1 Comment

Walter Roberson on 13 Mar 2016

Don't use eval(). Store the variables in a struct and use save with the -struct option. No -append is needed.

Link

Answer by Qifeng Chen on 30 Nov 2014

This is a better implementation based on Tihomir. The code here does not require the input of "numvars".

function parsave(fname,varargin)
  numvars=numel(varargin);
  for i=1:numvars
     eval([inputname(i+1),'=varargin{i};']);  
  end
  save('-mat',fname,inputname(2));
  for i = 2:numvars    
    save('-mat',fname,inputname(i+1),'-append');
  end
end

5 Comments

Show 2 older comments

Linus Schumacher on 23 May 2017

Erik's solution still give me a transparency violation error in Line 6 (savevar...)

Myrtle42 on 16 Jun 2017

This still gives an error in Matlab R2016a: the variable I'm trying to save cannot be classified. Same with Alex Shaw's version below. Any ideas? Here is my code:

function test
var_to_save = NaN(10,1);
parfor ii = 1:10
    var_to_save(ii,:) = rand;
    parsave('test.mat',var_to_save)
end
end
function parsave(varargin)
savefile = varargin{1}; % first input argument
for i = 2:nargin
    savevar.(inputname(i)) = varargin{i}; % other input arguments
end
save(savefile,'-struct','savevar')
end

Esha Bhargava on 2 Aug 2017

Please contact MathWorks Technical Support.

Link

Answer by Alex Shaw on 9 Dec 2015

Edited by Alex Shaw on 9 Dec 2015

I encountered this issue, and used the function proposed by Qifeng Chen.

This worked very well in 2015a.

However, since upgrading to 2015b my code reports a transparancy violation again. The release notes state that transparancy is now enforced more strictly, however all variables that I attempt to save either appear as inputs in expression in the parfor loop, or are the results of calculation in the parfor loop so I think this may be a bug.

Further to this, the affected function seems to be inputname(). The simplest (and probably fastest) way to work round this is to add all variables to a single structure variable, and then save that with a fixed name:

function parsave_simple(fname,C )
    save('-mat',fname,'C');
end

however, this would have been a lot of surgery in my case, plus other code expects the variables not to be in structure format. So the 'minimal surgery' option in my case is simply to pass the names of variables into the function via a cell array:

function parsave_named(fname____,vnames___,varargin)
  numvars___=numel(varargin);
  for ii___=1:numvars___
       eval([vnames___{ii___},'=varargin{ii___};']);  
  end
  save('-mat',fname____,vnames___{1});
  for ii___ = 2:numvars___    
    save('-mat',fname____,vnames___{ii___},'-append');
  end
end

This method seems a bit messy, there is a performance overhead to calling 'eval' and some very hard to trace bugs will arise if names do not match the actual variables. Also note the use of strange var names to ensure eval does not overwrite any internal logic! But on the other hand, it only needs one extra line in the calling loop to populate the names variable.

2 Comments

Walter Roberson on 13 Mar 2016

You can use the -struct option when you save the structure.

echo zhang on 11 Apr 2019

This works for me, and a working example below:

parfor ii = 1:n
    x=1;
    y=2;
    z=3;
    var_name = ["x","y","z"];
    parsave('whatever.mat',var_name,x,y,z);
end

Link

Answer by Anand on 13 Mar 2016

Edited by Anand on 13 Mar 2016

Another addition to various versions of doing this. This one does not use a structure. It, IMO, combines the best features of all answers above.

function parfor_save(varargin)
    fname=varargin{1};    
    for i=2:nargin
       eval([inputname(i),'=varargin{i};']);  
       if i==2
            save('-mat',fname,inputname(i));
       else
           save('-mat',fname,inputname(i),'-append');
       end        
    end

1 Comment

Walter Roberson on 13 Mar 2016

Don't use eval(). Store the variables in a struct and use save with the -struct option. No -append is needed.

Products

Parallel Computing Toolbox

Release

R2010a

You are now following this question