Importing Column Vectors Sequentially
1 次查看(过去 30 天)
显示 更早的评论
Hi, I'm trying to import data from a large tab delimited CSV with headers. (0.8 GB)
I don't want to import everything, just a number of specific columns. I.e. I would like to create unique column vectors for:
1. Cells D19:D234568
then
2. Cells F19:F234568
and so on.
Currently I'm doing this one-by-one as, even with 12GB ram I'm running out of memory.
There must be a simple way of doing this quickly, no? Once the vectors are loaded and saved as .mat they load up in seconds.
Cheers,
Alasdair
3 个评论
per isakson
2016-8-7
"tab delimited CSV with headers. (0.8 GB) .... even with 12GB ram I'm running out of memory"   I find it hard to believe that a 0.8GB text-file should cause an out of memory error.
Is it numerical, text or mixed data?
Did you try something like this?
frm = '%*s%*s%*s*f%*s%f%*[^\n]';
cac = textscan( fid, frm, (234568-18), 'Headerlines',18, 'Delimiter',\t')
采纳的回答
dpb
2016-8-7
编辑:dpb
2016-8-7
"... in ... import app those are the cells ... tab delimited..."
In that case, use textscan and a format string set up to read the desired columns. This isn't particularly difficult to automate depending on the columns wanted...
cols={'D','F'}; % the list of wanted columns
fmt=[]; % empty string to build format string into
for i=1:length(cols) % over the number of columns to read
fmt=[fmt repmat('%*f',1,cols{i}-'A') '%f']; % skip N-1, read 1
end
fmt=[fmt '%*[^\n']; % and then skip rest of line
fid=fopen('filename','r');
data=cell2mat(textscan(fid,fmt,'delimiter','\t', ...
'headerlines', 18, ...
'collectoutput',1)); % and read the file
fid=fopen(fid);
There's a section Large Text Files linked to at the doc for textscan that describes how to read a file in blocks if this still errors out on memory altho if the import tool can do it, the above likely will work as I'd venture it's what it does as a first try, anyway...
0 个评论
更多回答(0 个)
另请参阅
类别
在 Help Center 和 File Exchange 中查找有关 Large Files and Big Data 的更多信息
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!