memory pca vs pcacov

Question

Hans van der Horn 2020-11-2

0
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/634044-memory-pca-vs-pcacov

评论： Hans van der Horn 2020-11-2

Dear all,

I'm running a pca on a large matrix (33*500,000) and with pcacov I get a memory error, but pca gives me no trouble. Could anyone explain this to me? Is the matrix somehow being reduced before computing the covariance matrix in pca?

Thanks!

Best

Hans

0 个评论
显示 -2更早的评论隐藏 -2更早的评论

请先登录，再进行评论。

请先登录，再回答此问题。

Answer 1

Hiro Yoshino 2020-11-2

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/634044-memory-pca-vs-pcacov#answer_531894

In PCA, your matrix (p x q) will be once converted into the variance-covariance matrix (q x q).

This would reqiure huge memory comsumption. Meanwhile, pcacov accepts a variance-covariance matrix as an input and, therefore the argument (input) should be a square matrix though.

As for big data anaysis, you may want to use tall array - this can be a solution.

tall array link

0 个评论
显示 -2更早的评论隐藏 -2更早的评论

请先登录，再进行评论。

Answer 2

Hans van der Horn 2020-11-2

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/634044-memory-pca-vs-pcacov#answer_531914

Dear Hiro,

Thanks for your answer. If I compute cov(my matrix) to get the input for pcacov I run out of memory. However, if I run the pca function on my matrix. Why does the pca.m not get into memory problems?

Best

Hans

2 个评论
显示无隐藏无

Hans van der Horn 2020-11-2

*sorry, something was missing 'However, when I run the pca function on my matrix, it goes well. Why does pca.m not get into memory problems?'

Thanks

Hiro Yoshino 2020-11-2

I do not believe pcacov works with your matrix in the first place since the shape of your matrix is unacceptable. I do not know what the error will be like.

请先登录，再进行评论。

Answer 3

Hans van der Horn 2020-11-2

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/634044-memory-pca-vs-pcacov#answer_531919

Dear Hiro,

The problem is that by constructing the covariance matrix Matlab runs out of memory. Why does the pca.m function not give memory problems (when I of course enter the original matrix)?

thanks!

2 个评论
显示无隐藏无

Hiro Yoshino 2020-11-2

I got your point now!!

OK, actually to avoid memory problem, pca normally takes a different approach to calculate eigen vectors - SVD. This is not a direct method and produces some by-product. This is a well-known fact - you may find the this in your text book too, I'm sure.

Also, the documentation for PCA reads it uses SVD.

Hans van der Horn 2020-11-2