Reading large .csv data with 12 million rows

Question

Ganesh Naik 2021-11-10

0
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/1583049-reading-large-csv-data-with-12-million-rows

评论： Mathieu NOE 2021-12-16

Hi all, I have .CSV file with 14 columns and around 12 million rows. I would like to read the data and use it for further analysis. When I use "readtable" command it reads only first two columns with all the information but fails to read other column data. Is there anyway I can read the entire dataset once?

Thanks in advance.

3 个评论
显示 1更早的评论隐藏 1更早的评论

Ganesh Naik 2021-12-16

Hi Mathieu thanks for your message. I have solved the problem using the following method:

1) I have used "Split CSV" program to split the original CSV files into 12 sub-CSV files (each one million rows)

https://tipsmake.com/5-best-free-csv-splitting-tools-csv-file-splitting-software

2) Read each CSV data as:

data1 = readtable('Data-1.csv');

data2 = readtable('Data-2.csv');

.

data12 = readtable('Data-12.csv');

3) Combine the final file as:

Final_data = [data1; data2;data3; data4;data5; data6;data7; data8;data9; data10;data11; data12];

This created a large table for me (on the fly) at workspace and each time loading the above files dont take much memory. It worked for me, but I believe there maybe better alternative methods.

Mathieu NOE 2021-12-16

ok glad you havefound a workaround ! :)

请先登录，再进行评论。

请先登录，再回答此问题。

Answer 1

Sivani Pentapati 2021-12-1

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/1583049-reading-large-csv-data-with-12-million-rows#answer_844724

Hi Ganesh,

You can try using readmatrix in place of readtable. Other workaround is to convert the 'csv' files into 'mat' files and save them with '-v7.3' option. Please refer to this answer for more information.

1 个评论
显示 -1更早的评论隐藏 -1更早的评论

Ganesh Naik 2021-12-16

Dear Sivani, I have solved the problem using SplitCSV method, reading each file and combining the data (I have explained my method above). It worked for me.

请先登录，再进行评论。