Identifying lab id within a parfor

39 次查看（过去 30 天）
Javier
Javier 2012-2-25
评论： HiWave 2020-9-25
Hello all,
I have an problem that I would like to run in parallel using a parfor. Within the parfor I have to call an external program that runs some calculations using system('external_command result_file_name'). This external program saves the result into a text file that I have to read after execution.
In order for the different iterations not overwrite the results of each other, I would like to know what is the labid running each iteration such that I can compose the file name such as file_name_labid.
I have tried the function labindex but it always returns 1 within a parfor. Is there any other way I can get the labid?
Many thanks,

请先登录，再回答此问题。

采纳的回答

Edric Ellis
Edric Ellis 2012-2-27
As mentioned elsewhere, LABINDEX will not work for this purpose. You could either use
t = getCurrentTask(); t.ID
to find out which task you're running. Each worker has a unique task ID which remains the same for the duration of the MATLABPOOL.
Or, my personal preferred solution would be to use TEMPNAME to generate a unique filename to write the output to.
  3 个评论
显示 1更早的评论
Damdae
Damdae 2019-7-23
It would be better if labindex returned the value of get(getCurrentTask(),'ID') when it's in a parfor loop.
HiWave
HiWave 2020-9-25
I'm facing the same problem...how did you solve this?

请先登录，再进行评论。

更多回答（6 个）

Jiro Doke
Jiro Doke 2012-2-26
Why not save the file based on the iteration number? Wouldn't you still overwrite the results if you were using the labindex because different iterations could be running on the same worker.
As Oleg mentioned, if you want something to happen based on the labindex, use spmd instead. You would break up your loop into n groups, where n is the number of workers. Something like this:
spmd
for id = (1:100)+(labindex-1)*100
% do some computation
save(sprintf('file_name_%d', labindex));
end
end
EDIT 1: Combine SPMD and PARFOR to write to different directories
spmd
mkdir(sprintf('worker%d', labindex));
cd(sprintf('worker%d', labindex));
end
parfor id = 1:1000
fid = fopen('myresults.txt', 'a+');
fprintf(fid, '%d\n', id);
fclose(fid);
end
spmd
cd ..
end
EDIT 2:
function [] = main_function()
function [y] = obj(x)
% write parameters x to an input file
system('external_program input_file output_file');
% read output file
y = read_output();
end
spmd
folder = sprintf('worker%d', labindex);
mkdir(folder);
cd(folder);
end
% now I can use a parfor
result = [];
c = linspace(10,100,10);
parfor k=1:10
result(k) = obj( c(k) );
end
plot(c,result);
% or I can call fmincon
problem.x0 = x0;
problem.objective = @obj;
[x,fval] = fmincon(problem);
spmd
cd ..
end
end
Creating folders and cd-ing to the appropriate folder (using spmd) ensures that each worker resides in different locations. Then file writing and reading will happen in different locations.
Oleg Komarov
Oleg Komarov 2012-2-25
Trying a simple parfor never uses more than 1 worker:
matlabpool(2)
parfor ii = 1:10000
if numlabs > 1
disp(labindex)
end
end
matlabpool close
But the following snippet forces the execution of labindex on all workers:
matlabpool(2)
spmd
labindex
end
matlabpool close
Somebody who can enlighten us about this behaviour?
  1 个评论
Edric Ellis
Edric Ellis 2012-2-27
LABINDEX always returns the value 1 when not inside an SPMD block (or the body of a parallel job). You can check e.g. "t = getCurrentTask(); t.ID" to see that different tasks work on different PARFOR loop iterations. LABINDEX and NUMLABS are designed to show you with whom you can communicate using LABSEND and LABRECEIVE. Inside a PARFOR loop, that's no-one.

请先登录，再进行评论。

Javier
Javier 2012-2-25
Look at this other thread:
http://www.mathworks.com/matlabcentral/answers/5529-mysterious-behavior-in-parfor-i-know-sounds-basic-but
It seems that since communication is not allowed in parfor loops, numlabs and labindex are set to 1 in each worker process.
So this seems to be the expected behavior, but it does not solve my problem.
Any idea on how to identify each worker process?
Javier
Javier 2012-2-26
Thanks for the suggestion.
Indeed I could use the iteration number in the for-loop, but the reason I wanted to do it with the labindex is that I expect to use the same code later with a optimization function such as fmincon. From what I tried, it seems that fmincon behaves like parfor in this regard.
In this case, I won't have access to any iteration number, and I cannot use spmd. I just call fmincon with the objective function being the function that calls the external program and reads the result, so the only way I can think of distinguish between the different workers is if I can have an identification number such as labindex.
I don't mind to overwrite the result after each iteration with the same worker, because once I read the result I save the parameters into a vector in matlab. I just don't want that a different worker overwrites the result before I read it.
Any other idea?
  1 个评论
Jiro Doke
Jiro Doke 2012-2-26
With fmincon, you probably won't use parfor anyway, so I'm not sure I understand when you say you want to use the same code.
Another way I would do it is to use SPMD and PARFOR together. You can use SPMD to change the folder that each worker is in. Then you can write to a file, and the files will be unique for each worker because they're in different folders. See my edit to my answer.

请先登录，再进行评论。

Javier
Javier 2012-2-26
Maybe I didn't make myself clear, sorry. Somewhat simlified he code looks something like this. I think I don't miss anything important now
function [] = main_function()
function [y] = obj(x)
% write parameters x to an input file that will be passed to an external program
system('external_program input_file output_file');
% read output file
y = read_output();
end
% now I can use a parfor
result = [];
c = linspace(10,100,10);
parfor k=1:10
result(k) = obj( c(k) );
end
plot(c,result);
% or I can call fmincon
problem.x0 = x0;
problem.objective = @obj;
[x,fval] = fmincon(problem);
end
The code works fine when I use a regular for, or when I use fmincon with a single lab. I'm just worried that it can fail when using multiple labs.
Javier
Javier 2012-3-1
Thanks everyone for the replies.
Jiro Doke:
I did a simple test with your method, but it did not work. Maybe I did not understood you or something is missing. This is what I did:
function [] = test()
spmd
folder=sprintf('worker%d',labindex);
mkdir(folder);
cd(folder);
end
parfor k=1:10
disp(pwd)
end
spmd
cd ..
end
The problem is that once the worker changes the folder, they are not able to find the source code and throw an error.
Edric Ellis:
That's exactly what I needed! Thanks!

类别

在 Help Center 和 File Exchange 中查找有关 Parallel for-Loops (parfor) 的更多信息

标签

产品

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by