memory pca vs pcacov

3 次查看（过去 30 天）
Hans van der Horn
Hans van der Horn 2020-11-2
评论： Hans van der Horn 2020-11-2
Dear all,
I'm running a pca on a large matrix (33*500,000) and with pcacov I get a memory error, but pca gives me no trouble. Could anyone explain this to me? Is the matrix somehow being reduced before computing the covariance matrix in pca?
Thanks!
Best
Hans

请先登录，再回答此问题。

回答（3 个）

Hiro Yoshino
Hiro Yoshino 2020-11-2
In PCA, your matrix (p x q) will be once converted into the variance-covariance matrix (q x q).
This would reqiure huge memory comsumption. Meanwhile, pcacov accepts a variance-covariance matrix as an input and, therefore the argument (input) should be a square matrix though.
As for big data anaysis, you may want to use tall array - this can be a solution.
tall array link
Hans van der Horn
Hans van der Horn 2020-11-2
Dear Hiro,
Thanks for your answer. If I compute cov(my matrix) to get the input for pcacov I run out of memory. However, if I run the pca function on my matrix. Why does the pca.m not get into memory problems?
Best
Hans
  2 个评论
Hans van der Horn
Hans van der Horn 2020-11-2
*sorry, something was missing 'However, when I run the pca function on my matrix, it goes well. Why does pca.m not get into memory problems?'
Thanks
Hiro Yoshino
Hiro Yoshino 2020-11-2
I do not believe pcacov works with your matrix in the first place since the shape of your matrix is unacceptable. I do not know what the error will be like.

请先登录，再进行评论。

Hans van der Horn
Hans van der Horn 2020-11-2
Dear Hiro,
The problem is that by constructing the covariance matrix Matlab runs out of memory. Why does the pca.m function not give memory problems (when I of course enter the original matrix)?
thanks!
  2 个评论
Hiro Yoshino
Hiro Yoshino 2020-11-2
I got your point now!!
OK, actually to avoid memory problem, pca normally takes a different approach to calculate eigen vectors - SVD. This is not a direct method and produces some by-product. This is a well-known fact - you may find the this in your text book too, I'm sure.
Also, the documentation for PCA reads it uses SVD.
Hans van der Horn
Hans van der Horn 2020-11-2
thanks! It's clear to me now:) I will also read the documentation for PCA (I thought pcacov also used svd..).

请先登录，再进行评论。

类别

在 Help Center 和 File Exchange 中查找有关 Dimensionality Reduction and Feature Extraction 的更多信息

标签

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by