How do I process big datasets ??

3 次查看(过去 30 天)
I have data of on file per day , that goes for each day of the month and then each month of the year. I now process daily files separately for daily results. How can I do this for complete analysis of full one year file. My file is like this YYYYMMDD.loc ; this means YYYY- Years, MM- months, DD-day. What if I want to take data of particular month and analyse ??

采纳的回答

Ced
Ced 2016-4-5
编辑:Ced 2016-4-5
Hi
Do you now have each day in a separate file, or everything in one file?
Personally, if you want to evaluate different sections of time, I would keep each day in a separate file. Then, you can always just read the files you need.
E.g. Let's say you want to read the months of June and July of years 2013-2015:
(I am assuming that you want to read the same days and months for each year here)
years = 2013:2015;
months = 6:7;
days = [ 1 1 ; 30 31 ]; % first and last day to read, one month per column
N_years = length(years);
N_months = length(months);
N_days = sum(diff(days)+1); % Note: 1 is added to EACH month
N_files = N_years*N_months*N_days;
filenames = cell(N_files,1);
current_index = 1; % makes indexing easier
for i = 1:N_years
for j = 1:N_months
next_index = current_index + days(2,j)-days(1,j);
filenames(current_index:next_index,1) = ...
arrayfun(@(x)sprintf('%i%02i%02i.loc',years(i),months(j),x), days(1,j):days(2,j), 'UniformOutput',0);
current_index = next_index+1;
end
end
Then, you can load all the data you need
for i = 1:N_files
% load data, process, combine.... whatever you need
end
Cheers
  2 个评论
Add
Add 2016-4-5
Where does it load the data ?
Walter Roberson
Walter Roberson 2016-4-5
As Ced says,
Then, you can load all the data you need
for i = 1:N_files
% load data, process, combine.... whatever you need
end

请先登录,再进行评论。

更多回答(0 个)

类别

Help CenterFile Exchange 中查找有关 Data Type Identification 的更多信息

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by