Data compression in Parallel Computing approach

Question

Pierre Harouimi 2018-12-3

0
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/433510-data-compression-in-parallel-computing-approach

回答： Walter Roberson 2018-12-8

I have a program which uses parallel computing approach with communication between workers (spmd block and distributed arrays). The code is ok, but I figured that the time-consuming was especially due to the transfer of data (and not the calculations) between labs.

My question is: is there a way to compress data for communication between workers, like Huffman approach or other?

Thanks

Pierre

0 个评论
显示 -2更早的评论隐藏 -2更早的评论

请先登录，再进行评论。

请先登录，再回答此问题。

Answer 1

Ran Chen 2018-12-7

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/433510-data-compression-in-parallel-computing-approach#answer_351138

Hi,

Currently, it is a limit for parallel computing. You need to balance the overhead. If the communication time is far more than computation time, you may consider reducing the thread number. You can implement some compression algorithm can apply it to your code, it depends on what data type you have.

0 个评论
显示 -2更早的评论隐藏 -2更早的评论

请先登录，再进行评论。

Answer 2

Walter Roberson 2018-12-8

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/433510-data-compression-in-parallel-computing-approach#answer_351152

there is no built in method .

The Distributed Computing interface is built on MPI which I think is being used for spmd as well but I am less sure about parfor (which goes through java interfaces that I would have to dig into)

There is specifically provision to supply your own MPI for Distributed Computing toolbox.

People suggest that it might not be worth doing except on slow interconnects. However there are research papers listed below and I see at least one more as well

https://stackoverflow.com/questions/10849983/mpi-is-there-mpi-libraries-capable-of-message-compression

There is github repository for implementation of one of the papers mentioned .