HELP running large code.. having storage issues

Julian Jesso

2019 2 20

0 个回答

8 次查看（30 天）

0 个投票

I am attempting to sort through csv data points in order to filter a certain data type out to put into a new csv. The large csv contains 55 columns and approximately 43 million rows. I am running into storage issues from Matlab and my computer, even though I have a 500GB adidtional storage drive that I am working through. It takes about 4 1/2 hours to run this code, but I always seem to have an error when I check back. It is not a java heap error, but just states "out of memory. Type HELP MEMORY" and I get a popup box that says low on storage and shuts matlab down. This is haulting my project and I am not sure where to go from here. Any recommendations are appreciated...

clc;clear;
myfile = fopen('file_2009.csv');
whichSite = input('Which site? '); 
year = 2009;
recs{10e4,1} = []; 
ctr  = 1;
while ~feof(myfile)
linestring = fgetl(myfile); 
supercell=strsplit(linestring, ',','CollapseDelimiters',false);
if str2double(supercell{3}(2:5)) == whichSite
    recs{ctr,1} = supercell;
    ctr = ctr+1;
end
end
 
fclose(myfile);
mycsvfile = sprintf('Site%d_%d.csv',whichSite, year);
recstable = cell2table(recs);
writetable(recstable,mycsvfile);
recs = [];
recstable = []; 

1 个评论
显示 -1更早的评论隐藏 -1更早的评论

Adam 2019-2-20

Surely you can just read and write your data in sensible-sized chunks instead of trying to store 10^4 cells in memory at once? It seems to just be a sequential operation so I would assume you can just append results to those previously saved.

请先登录，再进行评论。

请先登录，再回答此问题。

Follow Question