Large data file with mixed character strings and numerical formats
1 次查看(过去 30 天)
显示 更早的评论
I have a large data text file with 2,000,000 rows and 100 columns. Some columns have numerical values and some are character strings with variable length. I don't need all the data at once but I need to be able to import different columns (character columns and numerical columns) for my analysis. How should I do that?
The issue is with the size of the file rather than the mixed formats. MATLAB datastore function only reads 20,000 rows at a time and I don't know if converting the data into a SQL datatable would help.
Thanks in advance
0 个评论
采纳的回答
Stephen23
2017-2-2
编辑:Stephen23
2017-2-2
Do not use fgetl or fgets: on such a large file as this would be very slow. Use textscan, exactly as the MATLAB documentation recommends:
The third textscan input lets you specify a block size, which sets a limit to how many lines to read. So to read your required data into MATLAB without reading all of the data at once, you need to do the following in a loop (pseudocode, see link above):
out = {};
k = 0 ;
while ~feof(fileID)
k = k+1;
C = textscan(fileID,formatSpec,N);
out{end+1} = the columns you need
end
This reads each block, you extract and store the columns that you need, and then the rest of the data is discarded. In this way, all of the data is read into MATLAB, just not simultaneously! This is really quite fast :)
2 个评论
Stephen23
2017-2-2
编辑:Stephen23
2017-2-2
@KSSV: textscan could read just one line, but why is that relevant to the question or even to my answer? Reading one line at-a-time would be incredibly inefficient compared to textscan reading blocks of data. That is why my answer reads blocks of data at once, and gets the requested columns without importing the entire file at once into MATLAB memory.
更多回答(1 个)
Aaditya Kalsi
2017-2-2
You could use datastore to select the columns and read only those columns in.
ds = datastore('filepath',...);
ds.SelectedVariables = {'Var1', ...};
tbl = readall(ds);
0 个评论
另请参阅
类别
在 Help Center 和 File Exchange 中查找有关 Data Import and Export 的更多信息
产品
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!