How can I read Or Use txt file for Next Step?
1 次查看(过去 30 天)
显示 更早的评论
Hi
I had Big Data(with 1 million Rows!) and i have to save it on txt file because Matlab couldn't perform it and happend Errore like: "OUT OF MEMORY"
So, i saved it in txt file(output).for Example a part of output is Attached(a_freq.txt).
Now, I need to Use New output for Next Step . my next step 's code is( X=output ):
X=??? ; %means I don't know!
param = 'Logarithmic' ;
[Y w] = tfidf2(X,param ) ;
and the Related Functione is :(also it's attached)
function [Y w] = tfidf2( X , param)
switch param
case 'Boolean'
% FUNCTION applies TF-IDF weighting to word count vector matrix.
%
% [Y w] = tfidf2( X );
%
% INPUT :
% X - word count vectors (one column = one document)
%
% OUTPUT :
% Y - TF-IDF weighted document-term matrix
% w - IDF weights (useful to process other documents)
%
% get inverse document frequencies
w = idf( X );
% TF * IDF
Y = tf1( X ) .* repmat( w, 1, size(X,2) );
case 'Logarithmic'
% get inverse document frequencies
w = idf( X );
% TF * IDF
Y = tf2( X ) .* repmat( w, 1, size(X,2) );
end
end
function Y = tf1( X )
% SUBFUNCTION computes word frequencies(Boolean)
Y = X ./ repmat( sum(X,1), size(X,1), 1 );
Y( isnan(Y) ) = 0;
end
function Y = tf2( X )
% SUBFUNCTION computes word frequencies(Logarithmic)
% Y = log(1+X) ;
Y = X ./ repmat( sum(X,1), size(X,1), 1 );
Y( isnan(Y) ) = 0;
end
function I = idf(X)
% SUBFUNCTION computes inverse document frequencies
% % count the number of words in each document
% counto the number of documents the term is repestead in
nz = sum( ( X > 0 ), 2 );
% compute idf for each document
I = log( size(X,2) ./ (nz(:) + 1) );
end
So, How can I Read txt file for New Step ???it's very important For me.
6 个评论
Guillaume
2014-11-28
With regards to your new question, I've no idea what you're asking. It sounds like it's about file reading but the code you've posted does not include any such thing.
Can you clarify what you want and remove stuff that is not relevant?
回答(3 个)
Guillaume
2014-11-28
The memory used by the content of your file, when read as double is only 2 MB (and only 250 KB as uint8). How can that be too big for matlab?
x = csvread('a_freq.txt');
should work. If you're really constrained by memory that you can't read a 2MB array, then most likely the rest of your code won't be able to cope either.
However, if a file is too big to read all at once, you can use low level functions to parse it yourself:
fid = fopen('a_freq.txt', 'rt');
tline = fgetl(fid); %read a line at time
while ~isempty(tline)
xrow = str2num(tline); %convert one line of text into a row vector of numbers
%...
%do something with that row
%...
tline = fgetl(tline); %read next line
end
fclose(fid);
4 个评论
Guillaume
2014-11-28
Another option, would be to use a sparse matrix when you generate the data, since it looks like your matrix is full of 0s.
Image Analyst
2014-11-28
2 个评论
Image Analyst
2014-11-29
Honestly I haven't used it personally but I know that this is it's main purpose. It's for cases where you have massive amount of data in a file and not enough RAM memory. This is it's whole reason for existing. Sorry, I don't know the details of how to use it though.
另请参阅
类别
在 Help Center 和 File Exchange 中查找有关 Data Type Conversion 的更多信息
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!