Fastest way to import single column?

Question

Brian 2014-1-13

0
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/112242-fastest-way-to-import-single-column

评论： Kelly Kearney 2014-1-14

I am attempting to import just a single column (a textual date field) from a large file with many columns and many rows. When I use textscan to import only a single column, it still takes a very long time to import my column (file is approx 700 MB). This leads me to believe that it's reading the entire file and bringing back the column that I am specifying. Can someone offer up a faster solution to import a single column from a large text file? The text file can have any number of rows and any number of columns.

Currently I am doing the following -

%Import Date field only to check latest date in file
    frmt = '%*s %s %*[^\n]';
    dates = textscan(fid,frmt,'delimiter',',','Headerlines',1);
    fclose(fid);
    dates = dates{1};

Thanks a lot! Brian

1 个评论
显示 -1更早的评论隐藏 -1更早的评论

José-Luis 2014-1-13

There's no going around reading most of the file for these kinds of problems unless you do some kind of preprocessing or generate only the data you need.

请先登录，再进行评论。

请先登录，再回答此问题。

Answer 1

David Sanchez 2014-1-13

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/112242-fastest-way-to-import-single-column#answer_120739

在 MATLAB Online 中打开

From matlab's documentation:

Using a text editor, create a file scan1.dat that contains data in the following form:

09/12/2005 Level1 12.34 45 1.23e10 inf Nan Yes 5.1+3i
10/12/2005 Level2 23.54 60 9e19 -inf  0.001 No 2.2-.5i
11/12/2005 Level3 34.90 12 2e5   10  100   No 3.1+.1i

Read the first column of the file into a cell array, skipping the rest of the line:

fid = fopen('scan1.dat');
dates = textscan(fid, '%s %*[^\n]');
fclose(fid);

2 个评论
显示无隐藏无

Brian 2014-1-13

David, thanks for the reply. If you look at my code you can see that I am already doing exactly what you suggested. I am wondering if there is a faster method than the one I'm currently using.

David Sanchez 2014-1-13

Reading from a file usually is a time consuming task. Reading from such a big file ( 700 MB ) will take a long time no matter how you try to do it. The method you are using might be the best solution for the task.

请先登录，再进行评论。

Answer 2

Kelly Kearney 2014-1-13

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/112242-fastest-way-to-import-single-column#answer_120745

It's not a Matlab solution, but I have found that pre-processing in Perl (split columns and spit out a one-column intermediate file) and then using load to bring it into Matlab is faster than textscan.

4 个评论
显示 2更早的评论隐藏 2更早的评论

Brian 2014-1-14

That seems simple enough I may give it a try but am hesitant to put into production because our system is supported by multiple users. While I may take the time to understand it, others may not. Is there any way to do something similar completely within Matlab's coding architecture?

Kelly Kearney 2014-1-14

I don't think so... I've always found textscan to be the fastest Matlab-only way of reading in an ascii file. As others have said, there's really no way around reading the whole file and then extracting what you need. The perl program does the exact same thing: read in a full line, break it up based on commas, keep just one column... perl just happens to be very good and very fast when it comes to churning through large amounts of text.

请先登录，再进行评论。