Create FOR loop to process and create datasets from all files found

Question

bugguts99 2011-5-11

0
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/7190-create-for-loop-to-process-and-create-datasets-from-all-files-found

Forgetting the need to append all data (which will be my ulimate goal), I need to process all files found to create x number of individual datasets. How do I run the following script with a FOR loop to create these datasets??

% sets base directory and lists all files stored within its subfolders
base_dir = 'Farm34Maize/';
[status, list] = system('dir /B /S TOA5_2436.flux*.dat');
result = textscan(list, '%s', 'delimiter', '\n');
filelist = result{1};
%%Read files
% Run FOR loop to read all .dat files
for i = 1:numel(filelist);
fid = fopen(filelist{i}, 'r');
%%Extract column headers
row = fgetl(fid{i}); % Skip header row
row = fgetl(fid{i}); % Extract second row that represents columns headers
cols = textscan(row, '%q', 'Delimiter', ','); % Identify column headers
cols = strtrim(cols{1}).';
%%Extract data and timestamp
row = fgetl(fid{i}); % Skip third row
row = fgetl(fid{i}); % Skip fourth row 
c = length(cols);
format = ['%q', repmat('%f', 1, c-1)];
data = textscan(fid, format, 'Delimiter', ',', 'TreatAsEmpty', {'"NAN"', '"INF"'}, 'CollectOutput', 1, 'headerlines', 1);
%%Split data and timestamp
data = [datenum(data{1}) data{2}]; % In order to work correctly, input file date formats must be 'yyyy-mm-dd HH:MM:SS' or 'yyyy/mm/dd HH:MM:SS PM' or 'y
fclose(fid{i});
end

Thanks in advance.

2 个评论
显示无隐藏无

Wieger Duursema 2011-6-9

Hi Bugguts99, I am wondering how you did manage to save the data within the FOR LOOP so that they do not overwrite. So, how did you do that? Thanks, Wieger

bugguts99 2012-6-8

1 year later....I ended up writing and appending to a delimited text file :

dlmwrite('20110420.txt', data, 'delimiter', ',', '-append');

请先登录，再进行评论。

请先登录，再回答此问题。

Answer 1

Walter Roberson 2011-5-11

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/7190-create-for-loop-to-process-and-create-datasets-from-all-files-found#answer_9856

在 MATLAB Online 中打开

Don't use system() to get the file names, use dir()

fileinfo = dir('TOA5_2436.flux*.dat');
for K = 1 : numel(fileinfo);
  thisfile = fileinfo(K).name;
  fid = fopen(thisfile,'rt');
  % Extract column headers
  row = fgetl(fid); % Skip header row
  row = fgetl(fid); % Extract second row that represents columns headers
  %and so on
  fclose(fid)
 end

Your code was trying to index fid, which was not a cell array in your code. That would have crashed your code. The change to dir() is just better programming.

3 个评论
显示 1更早的评论隐藏 1更早的评论

bugguts99 2011-5-11

Hi Walter, I have solved my 2nd query by writing each iteration to txt file (or the like)....so thanks anyhow. But if do have some suggestions re: dir() vs. system(dir), I would be very interested to hear these?

Walter Roberson 2011-5-11

Your code never uses base_dir . The system() call is *not* going to make use of that variable.

%find what is in the base directory

basedirinfo = dir(base_dir);

%throw away the info on everything that is not a directory

basedirinfo = basedirinfo([basedirinfo.isdir]);

%throw away directories "." and ".."

basedirinfo = basedirinfo(~ismember({basedirinfo.name},{'.','..'}));

%everything left is a subdirectory. Loop over all of them

data = {}; %to hold the data

for didx = 1 : length(basedirinfo)

thisdir = basedirinfo(didx).name;

subspec = sprintf('%s\%s', thisdir, 'TOA5_2436.flux*.dat');

fileinfo = dir(subspec);

numfiles = length(fileinfo);

%extend the data cell array to hold more entries

oldlen = size(data,1);

data(oldlen+numfiles) = {};

%now process the files

for fidx = 1 : numfiles

thisfile = sprintf('%s\%s', thisdir, fileinfo(fidx).name);

fid = fopen(thisfile,'rt');

% extract column headers

row = fgetl(fid); %Skip header row

..... %copy appropriate code here

thisdata = textscan(fid, format, 'Delimiter', ',', 'TreatAsEmpty', {'"NAN"', '"INF"'}, 'CollectOutput', 1, 'headerlines', 1);

thisdata = [datenum(thisdata{1}) thisdata{2}]; % In order to work correctly, input file date formats must be 'yyyy-mm-dd HH:MM:SS' or 'yyyy/mm/dd HH:MM:SS PM' or 'y

data{oldlen+fidx} = thisdata;

fclose(fid);

end %of file loop within directory

end %of directory loop

%now data{1}, data{2}, and so on are the accumulated data sets

请先登录，再进行评论。