Loading very large CSV files (~20GB)
显示 更早的评论
I have some CSV files that I need to import into the MATLAB (preferably in a .mat format). I already tried to break down these files using csvread function into 100 cluster pieces, but it's very slow and it doesn't go further than 60 steps even though my computer machine is fairly new. I need to extract only numeric values and this values are separated by comma. I will appreciate it if you can help me to get through this.
Thanks,
Sajad
2 个评论
per isakson
2016-3-17
- It might help to split the file into smaller files before importing to Matlab. If on Windows, see GSplit is a free reliable file splitter that lets you split your large files
- How much RAM is in your system.
- What OS?
- Upload a small piece of the file to your question
Sadjad Fakouri Baygi
2016-3-17
采纳的回答
Robert
2016-3-17
You should look into datastore and mapreduce. They were introduced in R2014b and are intended for handling large data sets. The datastore object allows you to read the data in chunks, skip columns, and store the results in a table. It's behavior is somewhat similar to fread or fscanf with a size input; however, the datastore's use of a table allows you to assign different data types to each column. I use this for data that includes Boolean status bits along with doubles so that I don't have to store the Booleans as doubles in my data array.
mapreduce is specifically designed for operating on data sets that don't fit in memory. Rather than attempt to explain it I will simply suggest you check out the documentation.
docsearch Getting started with mapreduce
更多回答（0 个）
类别
在 帮助中心 和 File Exchange 中查找有关 Text Files 的更多信息
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!
Translated by 
