Using readtable (or similar) to import multiple ranges of data
21 次查看(过去 30 天)
显示 更早的评论
Hi team,
I need to import some data from spreadsheets (.csv), my issue is, is thast the data is scattered throughout a single column vector, I have the row indexes for the data I need, but I am not sure how to go about implementing this using readtable or similar.
I have about 117,000 .csv files that i need data from, the data is all in the same place, so a simple for loop would do this, I have attempted to do this, but the time taken for this to process is extremely large ( I understand I have a lot of .csv files to go through).
is there anyway to increase the finishing time i.e., using an alternative to a for loop?
% range = 35 values not in sequential order
for i=3:(height(filedir))
file = convertCharsToStrings(filedir(i).name);
S{i-2} = readtable(file,'Range',range,'ReadVariableNames',false);
end
SCADA = [];
for i = 1:width(S)
arr = table2array(S{i});
SCADA = [SCADA; arr];
end
save SCADA.mat SCADA
this current form of the code imports ALL data in the column, some 300 odd ..
any help would be amazing!
0 个评论
回答(1 个)
J. Alex Lee
2022-10-24
it's hard to say without seeing the csv file, but it sounds like the data is not really tabular, and you may be better off using "readmatrix" or if there's too much overhead, some lower level function. See if below works
% probably best to use readmatrix and avoid your post processing
filedir = dir("folder/*.csv"); % use filter to only return .csv files
% then you don't have to play games with indexing
for i = numel(filedir):-1:1
file = convertCharsToStrings(filedir(i).name);
S{i} = readmatrix(file,'Range',range,'ReadVariableNames',false);
end
SCADA = vertcat(S{:});
By the way, another thing that may be taking a lot of time is growing your array SCADA continually (especially 117,000 times)
Try either of these to see if it brings down your time significantly if you want to stick with readtable
% intuitively to me, this should be faster
T = vertcat(S{:});
SCADA = table2array(T);
or
for i = numel(S):-1:
M{i} = table2array(S{i});
end
SCADA = vertcat(M{:});
0 个评论
另请参阅
类别
在 Help Center 和 File Exchange 中查找有关 Spreadsheets 的更多信息
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!