A faster way to read a large text file

Question

Azza Ahmed 2012-7-2

0
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/42528-a-faster-way-to-read-a-large-text-file

Hi,

I am reading a large text file ~3GB in chunks. Each time I open the file,I read the header lines, then read a chunk "of lets say 1000 lines", then I close the file before I process my data. The second time I want to read another chunk, I re-open my file and go through the file with "fgetl" line by line till I reach the line where I stopped previously before I grab a new chunk of data. I was wondering if there is a faster way to grab chunks of lines instead going through the beginning of the file line by line in each and every time.

Any help with this will be highly appreciated.

Best wishes

AA

0 个评论
显示 -2更早的评论隐藏 -2更早的评论

请先登录，再进行评论。

请先登录，再回答此问题。

Answer 1

Jonathan Sullivan 2012-7-2

2
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/42528-a-faster-way-to-read-a-large-text-file#answer_52323

编辑：Jonathan Sullivan 2012-7-2

在 MATLAB Online 中打开

Before you exit, store what line you are at, then, when you reopen it, seek to that line.

Right before you close:

file_position = ftell(fid);

Then after you open it:

fseek(fid,file_position,'bof');

0 个评论
显示 -2更早的评论隐藏 -2更早的评论

请先登录，再进行评论。

Answer 2

Azza Ahmed 2012-7-2

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/42528-a-faster-way-to-read-a-large-text-file#answer_52339

编辑：Walter Roberson 2012-7-2

在 MATLAB Online 中打开

Hi Jonathan,

Many thanks for getting back to me. I have tried to do something similar earlier by I had difficulties on where to have the fseek in my code. Here is the code to read the file:

StartLine=11;
NumberOflinesToGet=1000;
fid1 = fopen(filename);
if fid1 == -1
    disp(' ')
    disp('Operation was unsuccessful - Check the filename')
    y = [];
    return
end
for k=1:StartLine-1      %this reads the header lines   
    fgetl(fid1);
end
k=1;
while ~feof(fid1) && k<=NumberOflinesToGet  %this reads my data of 1000 lines
    d = fgetl(fid1);
    a = str2num(d);
    a = a(ColumnsToGet);
    y(k,:)=a;
    k=k+1;
end
if feof(fid1)
   file_position=0;
else
   file_position=ftell(fid1);     
end
fclose(fid1);

So would you kindly show me where to have the fseek? I tried to have it immediately after end of the first "if" loop but that seemed to have made things complicated.

Many thanks

AA