How can implement parallel pipeline using 3 workers in one computer ?

Question

Ammar 2017-8-7

0
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/351806-how-can-implement-parallel-pipeline-using-3-workers-in-one-computer

评论： Walter Roberson 2017-8-23

Dear everyone, Please I have a question:

I would like to used 3 workers (cores) to implement 3 functions in a parallel as pipeline, i.e. the first worker create the data, do some process(fun-add), and sends the results (x) to the first buffer/queue/matrix. Then, the second worker receives the data from the first buffer, do some process (fun-mult), and send the results (y) to the second buffer. The third worker receive the data, do some process (fun-double) and print the output(z). In the following attach the picture show the situation.

My question is that: How can I implement these three functions in parallel pipeline ?

Please, any idea how to solve this problem?

Kind regards Ammar

0 个评论
显示 -2更早的评论隐藏 -2更早的评论

请先登录，再进行评论。

请先登录，再回答此问题。

Answer 1

Walter Roberson 2017-8-19

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/351806-how-can-implement-parallel-pipeline-using-3-workers-in-one-computer#answer_278556

You could use spmd and labSend the data from one worker to another.

4 个评论
显示 2更早的评论隐藏 2更早的评论

Ammar 2017-8-23

Dear Walter Roberson, I would like to thank you again for this answer. Please, could I ask, how can we save the temporal data in a queue/buffer/matrix ? Then, the next core just make access to the queue to take the data not directly receive from the previous worker. And you can see in the figure the both workers make access to the queue at the same time simultaneously. Thanks - Ammar.

Walter Roberson 2017-8-23

In order to implement that by using memory instead of a file, you would have to use something like the routine named sharedmatrix from the File Exchange; however, I see a post that claims that routine might not work starting with R2017a. Also, you would have to change the source code for the sharedmatrix routine a little if you are using MS Windows.

There is no built-in routine for handling queues that are based on files. There are a few different approaches. https://www.mathworks.com/help/matlab/import_export/share-memory-between-applications.html shows one of them. That example is not reliable as a Queue, though, as it is not designed to allow there to be several entries in the queue.

It is a bit tricky to get file-based queues working properly; it is common to encounter "race conditions" unless you have operating system support for atomic operations, such as semaphore objects .

请先登录，再进行评论。