How to import big data CSV files

3 次查看（过去 30 天）
Abhishek Singh
Abhishek Singh 2019-5-25
评论： Abhishek Singh 2019-5-30
Hi,
Similar questions have been already asked but I wanted to know if there is an alternative to importing. I have a csv file with 3.5 million rows and 56 columns. At present while importing, I have to select the range I need. For instance I am selecting only 36 columns and barring few rows almost all. I tried textscan() but unable to achieve what I want which is without importing my code should select those rows and columns needed and also my main aim is to save time of importing and since textscan() is pretty fast I was using that. My column values are numerics.
Please let me know if I need to update with something here. I have a 8GB RAM so I think it should be easily feasible.
Thanks in advance!text

请先登录，再回答此问题。

采纳的回答

Walter Roberson
Walter Roberson 2019-5-25
编辑：per isakson 2019-5-25
These days, often the most convenient way is to use detectImportOptions, and set the SelectedVariableNames property of that to choose particular columns, and the readtable() -- or as of R2019a, readmatrix() if you are pure numeric.
textscan() is also a possibility. The easiest way to use that might be something like,
numcol = 56;
wanted_cols = [5 17:33 44:47]; %adjust to suit
fmt_cell = repmat('%*s', 1, numcol);
fmt_cell(wanted_cols) = {'%f'};
fmt = [fmt_cell{:}];
data = cell2mat( textscan(fid, fmt, 'Delimiter', ',', 'CollectOutput', 1) );
  17 个评论
显示 15更早的评论
Walter Roberson
Walter Roberson 2019-5-30
Shrug. You can get a wrong and unusable input quickly, or you can get a correct and usable input more slowly.
The extract you showed is not a csv file. csv files are, by definition, Comma Separated Values, never whitespace separated. RFC4180 specifically says that spaces are to be considered part of a field, not a separator.
It is not difficult to extract from a file only the lines that start with numbers. It is not all that much more difficult to divide a file into blocks of numbers, so that non-number marks the end of a block and the blocks are to be imported separately. But as soon as you start wanting to extract particular information from the non-number sections and associating it with a block of numbers, the task becomes more complicated.
Abhishek Singh
Abhishek Singh 2019-5-30
I am sorry. Importing in matlab showed it in columns and since it was a csv format file I thought it must be like that. But I do understand your point. I think I will stick to the first plan of using textscan() since it was also efficient.

请先登录，再进行评论。

更多回答（0 个）

类别

在 Help Center 和 File Exchange 中查找有关 Text Files 的更多信息

标签

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by