out of memory

Question

Muhammad 2012-1-17

0
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/26294-out-of-memory

I am trying to find covariance matrix of a matrix of size 3 by 65536, and getting "out of memory" error. I want to avoid this error please suggest some way to get rid of this error.

Thanks

0 个评论
显示 -2更早的评论隐藏 -2更早的评论

请先登录，再进行评论。

请先登录，再回答此问题。

Answer 1

Anton Semechko 2012-1-18

1
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/26294-out-of-memory#answer_34494

在 MATLAB Online 中打开

Here is an example:

N=6E4; % number of variables
M=10; % number of samples
% Simulate M samples and store them into one matrix
X=zeros(N,M);
for i=1:M
    X(:,i)=[4*randn(3E4,1);randn(3E4,1)];
end
% Compute the mean and center
Xave=mean(X,2);
X=bsxfun(@minus,X,Xave);

Now your covariance matrix would be defined as C=(X*X')/(M-1). Because N>>>M, this covariance matrix will only have (M-1) non-zero eigenvalues. Since we want to avoid computing C explicitly, we can get the required eigen-decomposition with the use of SVD:

[P,D,~] = svd(X/sqrt(M-1),0);
P(:,M)=[];
D=diag(D);
D(M)=[];

where P is N-by-(M-1) matrix containing the eigenvectors of C and D is the list of corresponding eigenvalues.

6 个评论
显示 4更早的评论隐藏 4更早的评论

Muhammad 2012-1-18

Thanks alot Anton Semechko

Xintao 2013-8-27

If I do it in the following way, then the results should be the same? That is: P=P1 and D=D1?

C=(X*X')/(M-1);

[P1,D1,~] = svd(C);

请先登录，再进行评论。

Answer 2

Anton Semechko 2012-1-17

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/26294-out-of-memory#answer_34358

I am assuming what you really meant to say is that you have 65536 data points in R3 and you are trying to compute the covariance matrix of size 65536. Unless your machine has at the minimum (65536^2)*8*3/2^20 =98304 GB of memory you will not be able to do that.

There are ways around it, however. It all depends on why you need the covariance matrix to begin with. If for instance you are going to perform orthogonal decomposition (i.e. find the eigenvalues and eigenvectors), then this operation can be done with the help of SVD which does not require explicit calculation of the covariance matrix.

3 个评论
显示 1更早的评论隐藏 1更早的评论

Muhammad 2012-1-17

Thanks Anton Semechko for answering

Basically I am trying to find PCA, for that I first have to find covariance matrix. I want to know what should I do to avoid "out of memory" error, by increasing the size of virtual memory or RAM or something else, please guide me in this regard. Thanks

Walter Roberson 2012-1-17

There are routines in the stats toolbox, but I do not know if they have ways of reducing memory; see http://www.mathworks.com/help/toolbox/stats/brkgqnt.html#f75476

But yes, make sure you use a 64 bit version of MATLAB, push up your virtual memory to at least 100 gigabyte (256 Gb would be safer), and let it go. It probably won't take more than 4 years.

请先登录，再进行评论。