Large data file with mixed character strings and numerical formats

2 次查看(过去 30 天)
I have a large data text file with 2,000,000 rows and 100 columns. Some columns have numerical values and some are character strings with variable length. I don't need all the data at once but I need to be able to import different columns (character columns and numerical columns) for my analysis. How should I do that?
The issue is with the size of the file rather than the mixed formats. MATLAB datastore function only reads 20,000 rows at a time and I don't know if converting the data into a SQL datatable would help.
Thanks in advance

采纳的回答

Stephen23
Stephen23 2017-2-2
编辑:Stephen23 2017-2-2
Do not use fgetl or fgets: on such a large file as this would be very slow. Use textscan, exactly as the MATLAB documentation recommends:
The third textscan input lets you specify a block size, which sets a limit to how many lines to read. So to read your required data into MATLAB without reading all of the data at once, you need to do the following in a loop (pseudocode, see link above):
out = {};
k = 0 ;
while ~feof(fileID)
k = k+1;
C = textscan(fileID,formatSpec,N);
out{end+1} = the columns you need
end
This reads each block, you extract and store the columns that you need, and then the rest of the data is discarded. In this way, all of the data is read into MATLAB, just not simultaneously! This is really quite fast :)
  2 个评论
Stephen23
Stephen23 2017-2-2
编辑:Stephen23 2017-2-2
@KSSV: textscan could read just one line, but why is that relevant to the question or even to my answer? Reading one line at-a-time would be incredibly inefficient compared to textscan reading blocks of data. That is why my answer reads blocks of data at once, and gets the requested columns without importing the entire file at once into MATLAB memory.

请先登录,再进行评论。

更多回答(1 个)

Aaditya Kalsi
Aaditya Kalsi 2017-2-2
You could use datastore to select the columns and read only those columns in.
ds = datastore('filepath',...);
ds.SelectedVariables = {'Var1', ...};
tbl = readall(ds);

类别

Help CenterFile Exchange 中查找有关 Large Files and Big Data 的更多信息

标签

产品

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by