How do I process big datasets ??
显示 更早的评论
I have data of on file per day , that goes for each day of the month and then each month of the year. I now process daily files separately for daily results. How can I do this for complete analysis of full one year file. My file is like this YYYYMMDD.loc ; this means YYYY- Years, MM- months, DD-day. What if I want to take data of particular month and analyse ??
采纳的回答
Hi
Do you now have each day in a separate file, or everything in one file?
Personally, if you want to evaluate different sections of time, I would keep each day in a separate file. Then, you can always just read the files you need.
E.g. Let's say you want to read the months of June and July of years 2013-2015:
(I am assuming that you want to read the same days and months for each year here)
years = 2013:2015;
months = 6:7;
days = [ 1 1 ; 30 31 ]; % first and last day to read, one month per column
N_years = length(years);
N_months = length(months);
N_days = sum(diff(days)+1); % Note: 1 is added to EACH month
N_files = N_years*N_months*N_days;
filenames = cell(N_files,1);
current_index = 1; % makes indexing easier
for i = 1:N_years
for j = 1:N_months
next_index = current_index + days(2,j)-days(1,j);
filenames(current_index:next_index,1) = ...
arrayfun(@(x)sprintf('%i%02i%02i.loc',years(i),months(j),x), days(1,j):days(2,j), 'UniformOutput',0);
current_index = next_index+1;
end
end
Then, you can load all the data you need
for i = 1:N_files
% load data, process, combine.... whatever you need
end
Cheers
2 个评论
Add
2016-4-5
Walter Roberson
2016-4-5
As Ced says,
Then, you can load all the data you need
for i = 1:N_files
% load data, process, combine.... whatever you need
end
更多回答（0 个）
类别
在 帮助中心 和 File Exchange 中查找有关 Dates and Time 的更多信息
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!
Translated by 
