How can I read Or Use txt file for Next Step?

Question

Isay 2014-11-27

0
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/164571-how-can-i-read-or-use-txt-file-for-next-step

评论： Image Analyst 2014-11-29

Hi

I had Big Data(with 1 million Rows!) and i have to save it on txt file because Matlab couldn't perform it and happend Errore like: "OUT OF MEMORY"

So, i saved it in txt file(output).for Example a part of output is Attached(a_freq.txt).

Now, I need to Use New output for Next Step . my next step 's code is( X=output ):

X=??? ; %means I don't know!
param = 'Logarithmic'  ;
[Y w] = tfidf2(X,param ) ;

and the Related Functione is :(also it's attached)

function [Y w] = tfidf2( X , param)
switch param
    case 'Boolean'
          % FUNCTION applies TF-IDF weighting to word count vector matrix.
          %
          %   [Y w] = tfidf2( X );
          %
          % INPUT :
          %   X        - word count vectors (one column = one document)
          %
          % OUTPUT :
          %   Y        - TF-IDF weighted document-term matrix
          %   w        - IDF weights (useful to process other documents)
          %
          % get inverse document frequencies
          w = idf( X );
          % TF * IDF
          Y = tf1( X ) .* repmat( w, 1, size(X,2) );
      case 'Logarithmic'
          % get inverse document frequencies
          w = idf( X );
          % TF * IDF
          Y = tf2( X ) .* repmat( w, 1, size(X,2) );
end
end
function Y = tf1( X )
% SUBFUNCTION computes word frequencies(Boolean)
Y = X ./ repmat( sum(X,1), size(X,1), 1 );
Y( isnan(Y) ) = 0;
end
function Y = tf2( X )
% SUBFUNCTION computes word frequencies(Logarithmic)
% Y = log(1+X) ;
Y = X ./ repmat( sum(X,1), size(X,1), 1 );
Y( isnan(Y) ) = 0;
end
function I = idf(X)
% SUBFUNCTION computes inverse document frequencies
% % count the number of words in each document
% counto the number of documents the term is repestead in
nz = sum( ( X > 0 ), 2 );
% compute idf for each document
I = log( size(X,2) ./ (nz(:) + 1) );
end

So, How can I Read txt file for New Step ???it's very important For me.

6 个评论
显示 4更早的评论隐藏 4更早的评论

Guillaume 2014-11-28

With regards to your new question, I've no idea what you're asking. It sounds like it's about file reading but the code you've posted does not include any such thing.

Can you clarify what you want and remove stuff that is not relevant?

Isay 2014-11-28

在 MATLAB Online 中打开

please look at the txt file(that is attached).It's my OUTPUT.

a_freq.txt is X , ok ?

Now, please look at the other Function , "function [Y w] = tfidf2( X , param)"

how can I read X in Matlab ???(X is too big and i can't load it in matlab)

请先登录，再进行评论。

请先登录，再回答此问题。

Answer 1

Guillaume 2014-11-28

1
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/164571-how-can-i-read-or-use-txt-file-for-next-step#answer_160533

在 MATLAB Online 中打开

The memory used by the content of your file, when read as double is only 2 MB (and only 250 KB as uint8). How can that be too big for matlab?

x = csvread('a_freq.txt');

should work. If you're really constrained by memory that you can't read a 2MB array, then most likely the rest of your code won't be able to cope either.

However, if a file is too big to read all at once, you can use low level functions to parse it yourself:

fid = fopen('a_freq.txt', 'rt');
tline = fgetl(fid); %read a line at time
while ~isempty(tline)
    xrow = str2num(tline); %convert one line of text into a row vector of numbers
    %...
    %do something with that row
    %...
    tline = fgetl(tline); %read next line
end
fclose(fid);

4 个评论
显示 2更早的评论隐藏 2更早的评论

Guillaume 2014-11-28

Another option, would be to use a sparse matrix when you generate the data, since it looks like your matrix is full of 0s.

Isay 2014-11-28

sparse method didn't help me,out of memory happend again.

请先登录，再进行评论。

Answer 2

Thorsten 2014-11-28

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/164571-how-can-i-read-or-use-txt-file-for-next-step#answer_160531

在 MATLAB Online 中打开

X = dlmread('a_freq.txt');
[Y w] = tfidf2(X, 'Logarithmic');

1 个评论
显示 -1更早的评论隐藏 -1更早的评论

Isay 2014-11-28

Thanks,but I said 'a_freq.txt' is a part of Real output!! :(

The real output is 9 GB !!

i can't load it in MATLAB with dlmread

请先登录，再进行评论。

Answer 3

Image Analyst 2014-11-28

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/164571-how-can-i-read-or-use-txt-file-for-next-step#answer_160560

Try memmapfile():

http://www.mathworks.com/matlabcentral/answers/?search_submit=answers&query=memmapfile&term=memmapfile

2 个评论
显示无隐藏无

Isay 2014-11-28

编辑：Isay 2014-11-28

在 MATLAB Online 中打开

thanks,but is it for just Binary method? http://blogs.mathworks.com/loren/2013/07/09/using-memmapfile-to-navigate-through-big-data-binary-files/

I try to use it, but i think first , i should create memmapfile ? and then read it ??? or I can use a_freq.txt ?

anyway, i use my output txt file(a_freq.txt)

my code:

X = memmapfile('a_freq.txt')'

and message was :

Filename: '...\a_freq.txt'
    Writable: false
      Offset: 0
      Format: 'uint8'
      Repeat: Inf
        Data: 544269x1 uint8 array

(why Data is 544269*1 ?? a_freq.txt is 49*5549)

so , can I use "X.Data" for other function? is it ture? but it's faild..

can you say an example ?

Image Analyst 2014-11-29

Honestly I haven't used it personally but I know that this is it's main purpose. It's for cases where you have massive amount of data in a file and not enough RAM memory. This is it's whole reason for existing. Sorry, I don't know the details of how to use it though.

请先登录，再进行评论。