can MATLA read very large CSVs (>150GB)?

2 次查看（过去 30 天）
sani
sani 2022-11-23
评论： Cris LaPierre 2022-11-23
Hi all,
I need to read and analyze six very large CSV files. In each file, I'm interested in only two columns. I already used opt to try reducing the amount of data, but my PC is constantly collapsing (since it reached the max memory of 128GB and the allocated memory as well).
My question is, do you know another approach to handle such large CSV files?
Eventually, I want to split these files, but I need to read them first.
Is that right?
Thanks :)

请先登录，再回答此问题。

回答（2 个）

Steven Lord
Steven Lord 2022-11-23
Use the tools in MATLAB for processing Large Files and Big Data.
Cris LaPierre
Cris LaPierre 2022-11-23
This sounds like what is called Big Data to me. MATLAB does have functionality for working with this data, A good starting place may be the Big Data with MATLAB page. You may be particularly intersted in the functionality for Tall Arrays.
Here's a good example you may be able to follow to get started.
  2 个评论
sani
sani 2022-11-23
Thsnk you for your answer!
can I convert a Tall array to a table later? (my script already working with tables, should I do any modifications?)
Cris LaPierre
Cris LaPierre 2022-11-23
MATLAB Tables were designed to work with Tall Arrays. See the 'Extended Capabilities' section of the coresponding doc pages

请先登录，再进行评论。

类别

在 Help Center 和 File Exchange 中查找有关 Data Type Conversion 的更多信息

标签

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by