I want to store a big matrix (a 720GB variable )

Question

seokwoo 2011-4-14

1
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/5474-i-want-to-store-a-big-matrix-a-720gb-variable

I want to store a big matrix. It needs about 720GB memory.

The big matrix is used to calculate eigs fuction (to find eigenvector) as a parameter.

Is it possible to get big matrix (720GB memory or even more) by using distributed computer environment?

I have ten-computers which has 72GB memory, so I want to save the big matrix divided to 10-computers.

Then do eigs function -> result = eigs('big matrix')

So the big matrix could be recognized as one variable even if it is stored in divided computers.

Is this possible?

1 个评论
显示 -1更早的评论隐藏 -1更早的评论

Oleg Komarov 2011-4-14

That's a HUGE matrix! How did you get that?

请先登录，再进行评论。

请先登录，再回答此问题。

Answer 1

Sarah Wait Zaranek 2011-4-14

1
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/5474-i-want-to-store-a-big-matrix-a-720gb-variable#answer_7708

I can think of two ways to do this:

1. If you have a sparse matrix, then I would use the sparse data type in MATLAB and probably use a 64bit machine/OS. This will not hold the non-zero values, greatly lessening your memory footprint. d = eig(S) accepts a sparse symmetric matrix S. Otherwise, use eigs in place of eig.

documentation on sparse

2. Using distributed arrays (across multiple machines). This will do what you suggest in your question, it will create a matrix whose elements are distributed across the memory of multiple machines but can still be treated as one big matrix. To do this you need to have the MATLAB Distributed Computing Server product installed on your cluster. Here is a demo using backslash - but eig should follow the same idea since it is supported for distributed arrays.

demo link

2 个评论
显示无隐藏无

Walter Roberson 2011-4-14

Unless something changed in 2011a, it is not possible to use eigs() to calculate all the eigenvalues of a sparse matrix: it errors out and says to use eig() instead.

Also, eigs is *very* slow if you are calculating more than a handful of eigenvalues

Sarah Wait Zaranek 2011-4-14