PCA or LSA on a very large dataset

Question

Eugene Kogan 2012-8-30

0
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/47020-pca-or-lsa-on-a-very-large-dataset

回答： arushi 2024-7-30

I am trying to run LSA or PCA on a very large dataset, 50k docs by 300k terms, to reduce the dimensionality of the words.

My system runs out of memory and grinds to a halt. I am using this code on the TFIDF matrix to compute the LSA, and it gets stuck on SVD:

   % SVD decomposition of tf-idf matrix
   [ U S V ] = svd(tfidfmatrix);
   % Generate new rank reduced matrix of rank k
   Sk = S(1:K,1:K);
   Uk = U(:,1:K);
   output = inv(Sk)*Uk';

and using PRINCOMP for PCA. In both cases, it seems I have too many terms for my system to handle.

Is there a more efficient way to do dimensionality reduction for two words? The end goal is to visualize the documents in 2d or 3d where they are grouped by similarity to each other.

0 个评论
显示 -2更早的评论隐藏 -2更早的评论

请先登录，再进行评论。

请先登录，再回答此问题。

Answer 1

arushi 2024-7-30

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/47020-pca-or-lsa-on-a-very-large-dataset#answer_1492311

Hi Eugene,

When dealing with very large datasets, such as your 50k documents by 300k terms matrix, traditional methods like full Singular Value Decomposition (SVD) and Principal Component Analysis (PCA) can be computationally expensive and memory-intensive. Instead, you can use more efficient methods designed for large-scale data.Efficient Methods for Dimensionality Reduction

1. Truncated SVD (also known as Latent Semantic Analysis - LSA):