linear least squares/mldivide for large matrices in parallel?

2 次查看（过去 30 天）
Arvind
Arvind 2015-4-8
评论： Sean de Wolski 2016-7-27
I have a really large system to solve using linear least squares. The A matrix can have 2-3 million rows and 2000-3000 columns. The B matrix has same row size but with a single column.
I have access to a supercomputer, and I want to run the x = A\B (or) mldivide(A,B) command in parallel, since I can easily run out of RAM even on workstations with lots of memory.
Any ideas? I am able to run EIG and SVD without any issues in parallel, since I assume it is automatically parallelized by MATLAB. What about linear least squares? Suggestions outside of MATLAB are also welcome. Thanks.

请先登录，再回答此问题。

采纳的回答

Edric Ellis
Edric Ellis 2015-4-8
编辑：Edric Ellis 2015-4-8
If you have access to a cluster of machines, you could use distributed arrays to solve the large system in parallel using the multiple memories. You'll need MATLAB Distributed Computing Server worker licenses on the cluster, and Parallel Computing Toolbox on the client machine. Something like this:
parpool();
A = distributed.rand(20000,2000);
b = sum(A, 2);
x = A\b;
  3 个评论
显示 1更早的评论
David
David 2016-7-27
Okay so how do I then load up my distributed matrix? It seems that distributed arrays don't support non scalar right hand assignment... do I really need to loop through the entire 200,000 x 20,000 array in my case one by one? Won't this be really slow compared with the vectorized matrix manipulations I was using before?
I tried stuffing the matrix and then distributing it, but its over my single thread RAM limit at this point.
Sean de Wolski
Sean de Wolski 2016-7-27
David, please ask this in a new question - the answer will end up being to use codistributed arrays inside of spmd.

请先登录，再进行评论。

更多回答（1 个）

Mahdiyar
Mahdiyar 2015-4-8
Hi Arvind
Parallel computing helps you to use more amount of CPU to run your simulation in a shorter time. As well as I know, when you have memory problem, it does not help you.
What I can suggest you is that you can implement the "x=A\B" by your own code.
I mean that write the m-file to calculate this x = A\B. The only difference is that you have to save your data and delete another one when you do not need it to avoid Memory problem.
For example, to calculate the A\B, you need to calculate A^(-1). Thus, first, JUST load matrix A and calculate A^(-1) and then save that matrix as a matrix and delete matrix A (be cause you do not need it anymore).
I hope it helps you.
Regards,

类别

在 Help Center 和 File Exchange 中查找有关 Distributed Arrays 的更多信息

标签

产品

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by