I have a file 1x200 struct how can I distribute it to the worker (32 worker)

Question

Student for ever 2018-1-19

0
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/377912-i-have-a-file-1x200-struct-how-can-i-distribute-it-to-the-worker-32-worker

评论： Ahmed Abdalazeez 2018-1-22

Dear all, I have a file 1x200 struct, I would like to send it to the cluster (parallel computation), how can I distribute it to the worker (32 worker). I can make it one by one, but it is really take very long time.

Thanks in advance.

Best regards, ahmed

2 个评论
显示无隐藏无

Walter Roberson 2018-1-19

Is it a file (text or binary) or is it just a struct()? Do the workers need access to all of it or only to the portion they are acting on?

Student for ever 2018-1-19

Hello Walter, it just a struct(). I think only to the portion they are acting on. I am sorry I am not familiar with this cluster stuff. Thank you

请先登录，再进行评论。

请先登录，再回答此问题。

Answer 1

Walter Roberson 2018-1-19

1
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/377912-i-have-a-file-1x200-struct-how-can-i-distribute-it-to-the-worker-32-worker#answer_300910

Just create the struct normally before the parfor() loop, and inside the parfor() loop, index the struct by the parfor variable. MATLAB will automatically distribute only the needed portion to each worker.

3 个评论
显示 1更早的评论隐藏 1更早的评论

Walter Roberson 2018-1-22

在 MATLAB Online 中打开

data = struct('test', num2cell(rand(10,200),1) );

Now data is a 1 x 200 struct with one field named test, with each entry data(K).test being a 10 x 1 vector. The details of the above are not important: I just needed to create a 1 x 200 struct to show parfor.

Now:

parpool(32)
sums = zeros(1,length(data));
parfor K = 1 : length(data)
   sums(K) = sum( data(K).test );
end

In this example, parfor does not need to copy all of data to every worker: it only copies the portion needed by the worker. Notice that no special code was needed for this: parfor handles it automatically.

This is an example of a "sliced variable", https://www.mathworks.com/help/distcomp/sliced-variable.html

If the use of the elements of the data structure was more complicated, then parfor could decide that the easiest thing to do is to copy all of data to every worker. In that situation, data would be referred to as a "broadcast variable", https://www.mathworks.com/help/distcomp/broadcast-variable.html

Ahmed Abdalazeez 2018-1-22