How do I use SAVE with a PARFOR loop using Parallel Computing Toolbox?
461 次查看(过去 30 天)
显示 更早的评论
I would like to save some variables to MAT files from inside a PARFOR loop. However I get an error:
??? Error using ==> parallel_function at 598
Error in ==> parallel_function>make_general_channel/channel_general at 894
Transparency violation error.
See Parallel Computing Toolbox documentation about Transparency.
采纳的回答
MathWorks Support Team
2020-2-24
编辑:MathWorks Support Team
2020-2-24
Transparency is violated by the SAVE command because in general MATLAB cannot determine which variables from the workspace will be saved to a file.
The solution is to move the call to SAVE to a separate function and to call that function from inside the PARFOR loop. Pass any variables that are to be saved as arguments to the function. That way MATLAB can determine which ones will be saved and transparency is not violated.
For example:
Save the following as "parsave.m":
function parsave(fname, x,y)
save(fname, 'x', 'y')
end
Then run it with:
parfor ii = 1:4
x = rand(10,10);
y = ones(1,3);
parsave(sprintf('output%d.mat', ii), x, y);
end
Similarly, for loading several files with enumerated file names, we can use the following code.
function [output] = par_load(i)
eval(['load file_' num2str(i) '.mat']);
output = time;
end
Which can be run with:
parfor i=1:4
data = par_load(i);
a(i) = 10 + data(1);
end
更多回答(7 个)
Daniel
2016-3-14
A nice alternative to implement your own save function is using the built-in matfile function:
parfor ii = 1:4
m=matfile(sprintf('output%d.mat', ii),'writable',true)
x = rand(10,10);
y = ones(1,3);
m.x=x;
m.y=y;
end
Here the parser has no issue understanding which variables are used.
Tihomir Kostadinov
2014-8-22
编辑:Tihomir Kostadinov
2014-8-22
I have enhanced the function that allows saving within a parfor loop as follows. This should allow giving it a variable number of arguments and the function recognizes their names. Let me know if there are problems with this implementation.
function save_for_parfor(fname,numvars,varargin)
for i = 1:numvars
eval([inputname(i+2),' = varargin{i};']);
end
save('-mat',fname,inputname(3));
for i = 2:numvars
save('-mat',fname,inputname(i+2),'-append');
end
This function would be called as follows:
>> x = 3; y = 4; z = 5; g = 'tester_parfor';
>> save_for_parfor('tester.mat',4,x,y,z,g)
1 个评论
Walter Roberson
2016-3-13
Don't use eval(). Store the variables in a struct and use save with the -struct option. No -append is needed.
Qifeng Chen
2014-11-30
This is a better implementation based on Tihomir. The code here does not require the input of "numvars".
function parsave(fname,varargin)
numvars=numel(varargin);
for i=1:numvars
eval([inputname(i+1),'=varargin{i};']);
end
save('-mat',fname,inputname(2));
for i = 2:numvars
save('-mat',fname,inputname(i+1),'-append');
end
end
5 个评论
Alex Shaw
2015-12-9
编辑:Alex Shaw
2015-12-9
I encountered this issue, and used the function proposed by Qifeng Chen.
This worked very well in 2015a.
However, since upgrading to 2015b my code reports a transparancy violation again. The release notes state that transparancy is now enforced more strictly, however all variables that I attempt to save either appear as inputs in expression in the parfor loop, or are the results of calculation in the parfor loop so I think this may be a bug.
Further to this, the affected function seems to be inputname(). The simplest (and probably fastest) way to work round this is to add all variables to a single structure variable, and then save that with a fixed name:
function parsave_simple(fname,C )
save('-mat',fname,'C');
end
however, this would have been a lot of surgery in my case, plus other code expects the variables not to be in structure format. So the 'minimal surgery' option in my case is simply to pass the names of variables into the function via a cell array:
function parsave_named(fname____,vnames___,varargin)
numvars___=numel(varargin);
for ii___=1:numvars___
eval([vnames___{ii___},'=varargin{ii___};']);
end
save('-mat',fname____,vnames___{1});
for ii___ = 2:numvars___
save('-mat',fname____,vnames___{ii___},'-append');
end
end
This method seems a bit messy, there is a performance overhead to calling 'eval' and some very hard to trace bugs will arise if names do not match the actual variables. Also note the use of strange var names to ensure eval does not overwrite any internal logic! But on the other hand, it only needs one extra line in the calling loop to populate the names variable.
2 个评论
echo zhang
2019-4-11
This works for me, and a working example below:
parfor ii = 1:n
x=1;
y=2;
z=3;
var_name = ["x","y","z"];
parsave('whatever.mat',var_name,x,y,z);
end
Anand
2016-3-13
编辑:Anand
2016-3-13
Another addition to various versions of doing this. This one does not use a structure. It, IMO, combines the best features of all answers above.
function parfor_save(varargin)
fname=varargin{1};
for i=2:nargin
eval([inputname(i),'=varargin{i};']);
if i==2
save('-mat',fname,inputname(i));
else
save('-mat',fname,inputname(i),'-append');
end
end
1 个评论
Walter Roberson
2016-3-13
Don't use eval(). Store the variables in a struct and use save with the -struct option. No -append is needed.
CrossEntropy
2019-12-3
The solution that works for me, on R2018a, is to define a function which includes the entire for loop, and then use save inside that function. Anything else I tried among the solutions in here id not work for me.
Muh Alam
2022-3-1
This answer is similar to previous answers and comments but does not use eval to avoid any possible troubles and weird outputs and simplify the code.
function par_save(file_name, varargin)
%-----------inputs-----------:
%file_name; name of .matfile to be saved
%varargin{}: variables needed to be saved to the matfile named in file_name.
ct ={};%to be converted to structure as struct(field1,value1,...,fieldN,valueN)
fi=1;% initial field index
vi=2;% = value index
for i=1:numel(varargin)
ct{fi} = inputname(i+1);%+1 since the 1st input is filename
ct{vi} =varargin{i};
fi=fi+2;
vi=vi+2;
end
s = struct(ct{:}) ;% put inputs in a structure with fields that has same name as the inputs
save(file_name,'-v7.3', '-struct','s','-nocompression')
end
- you might use default flags at save function but if you know that you are going to save large variables (>2GB) you need to set the version to v7.3 otherwise matlab will not save the variable (it will issue warning).
- nocompression flag is optional as well but I found that it is making save faster for saving very large matrices and that might also faster if you use matfile function to partially load variables you saved (also requires v7.3 ).
0 个评论
另请参阅
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!