PCA or LSA on a very large dataset

2 次查看（过去 30 天）
Eugene Kogan
Eugene Kogan 2012-8-30
回答： arushi 2024-7-30
I am trying to run LSA or PCA on a very large dataset, 50k docs by 300k terms, to reduce the dimensionality of the words.
My system runs out of memory and grinds to a halt. I am using this code on the TFIDF matrix to compute the LSA, and it gets stuck on SVD:
% SVD decomposition of tf-idf matrix
[ U S V ] = svd(tfidfmatrix);
% Generate new rank reduced matrix of rank k
Sk = S(1:K,1:K);
Uk = U(:,1:K);
output = inv(Sk)*Uk';
and using PRINCOMP for PCA. In both cases, it seems I have too many terms for my system to handle.
Is there a more efficient way to do dimensionality reduction for two words? The end goal is to visualize the documents in 2d or 3d where they are grouped by similarity to each other.

请先登录，再回答此问题。

回答（1 个）

arushi
arushi 2024-7-30
Hi Eugene,
When dealing with very large datasets, such as your 50k documents by 300k terms matrix, traditional methods like full Singular Value Decomposition (SVD) and Principal Component Analysis (PCA) can be computationally expensive and memory-intensive. Instead, you can use more efficient methods designed for large-scale data.Efficient Methods for Dimensionality Reduction
1. Truncated SVD (also known as Latent Semantic Analysis - LSA):
  • Instead of computing the full SVD, you can compute only the top K singular values and vectors using methods like svds in MATLAB.
2. Incremental PCA:
  • Incremental PCA is designed to handle large datasets by processing data in chunks.
Hope this helps.

类别

在 Help Center 和 File Exchange 中查找有关 Dimensionality Reduction and Feature Extraction 的更多信息

标签

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by