Synchronization for parallel code
9 次查看(过去 30 天)
显示 更早的评论
Hi,
I have a simple task to accomplish using the parallel tool box and distributed computing server. I need to execute a program (unix() call) on multiple workers. The workers run the same program but the program takes in a data file and writes a data file. Each worker's running program takes a different data file in a directory which all workers have access to. There are more data files than workers, so I would like a worker to choose another data file to run after executing the program on a previous data file. I want all workers to do this until all data files are exhausted. After reading the doc, it seems spmd is the way to go.
I have a cell array of file names and I would like to create an array of booleans of the same dimension to use for synchronization. When a worker accesses a file to work on, it should set the boolean at the array index of the index of the file that it's working on. When each worker finishes it's program run, it indexes through the booleans to find a file that has not been run and sets the boolean for that file index and then runs that one.
Is there any way for the workers to see a common array and also modify this same array?
Right now I'm doing it in a parfor loop that is dimensioned for maximum number of workers. This works fine, but I have a lot more files than workers. Therefore I have to wait for the parfor loop to finish before manually repeating the process for the remaining files. The program takes variable hours to complete on each worker so the parfor loop is bottlenecked by the longest run.
Is there a way to accomplish this using tasks as well? It seems like a simple thing but I have not found any examples of how to do this.
Thank you for reading.
0 个评论
采纳的回答
Jill Reese
2012-10-17
It sounds like you have independent tasks with more work to do than workers to do it. You also would like the tasks to be properly load-balanced. With these goals, parfor is the right tool. Note that parfor is a different construct than spmd in the Parallel Computing Toolbox.
Since you already have a cell array of file names and all the workers have access to the directory they are in, parfor can easily help you to load data from the files and use the data.
For example, if a different A variable is saved in each file, this code will allow 2 workers to process 5 files in an independent fashion without stomping on each other (no boolean array required).
matlabpool open local 2
inFiles = {'A.mat','B.mat', 'C.mat','D.mat','E.mat'};
outData = cell(1,length(inFiles));
parfor i=1:length(inFiles)
S = load(inFiles{i}, 'A');
outData{i} = S.A+3;
end
At the end of the computation, each worker has stored the result of its computation in a cell array with the same number of elements as the number of files processed.
Unfortunately, you cannot call save from within a parfor loop. You didn't mention how much data you need to save at each iteration. It may be that you can store the results as shown above until the parfor loop completes and then have a post-processing save step that executes outside of a parfor loop.
3 个评论
Jill Reese
2012-10-18
The parfor command automatically disperses the work (the total number of inFiles in your example) amongst the available workers. There is no guarantee that worker 1 will be assigned iteration 1. In fact, there is no guarantee that workers will execute their assigned values of i in any particular order. This is one reason why all iterations must be completely independent.
Once a worker completes the first iteration that it was assigned, it moves on to the next iteration that it was assigned to execute. This continues until all the work is complete. Does this help?
更多回答(0 个)
另请参阅
类别
在 Help Center 和 File Exchange 中查找有关 Parallel for-Loops (parfor) 的更多信息
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!