Alternatives for dir command
显示 更早的评论
Hello dear masterminds!
I have a working code for collecting strings from several txt.files (appr. 53000 files) in a network folder. I have one problem though. It is quite time consuming (17-19 seconds) using the dir command to identify the filenames. Is there a better way of doing this? I will attach a part of the code and some of the filenames.
Thank you!
addpath \\winfs\data\prod\maprod\arkivering\TAF;
DS = StartDTstr;
DE = StopDTstr;
TI = duration(timein,'InputFormat','hh:mm');
TO = duration(timeout,'InputFormat','hh:mm');
P = '\\winfs\data\prod\maprod\arkivering\TAF';
S = dir(fullfile(P,'*.txt'));
D = regexp({S.name},'\d{10}','match','once');
T = datetime(D, 'InputFormat','yyMMddHHmm');
X = isbetween(T,DS,DE);
tod = timeofday(T);
Y = tod>=TI & tod<=TO;
Z = S(X&Y);
Z.name;
numfiles = length(Z(:,1)); %create empty cell
for h=1:numfiles
filename=Z(h).name;
fileID=fopen(filename); %open filename to create fileID
Data{h}=textscan(fileID,'%s','delimiter','=','headerlines',1);
fclose(fileID); %close fileID
end
rmpath \\winfs\data\prod\maprod\arkivering\TAF;
Utcell = cell(1,length(Data)); %output data of all TAF
6 个评论
Linus Dock
2021-10-10
"Any ideas why this happens?"
You need to use an absolute/relative filename, for example:
fnm = fullfile(P,Z(h).name);
fileID = fopen(fnm);
This is exactly what Jan's answer shows too.
Linus Dock
2021-10-10
Stephen23
2021-10-11
"Is it the network and the size of that folder that's the issue then?"
If the files are being accessed over a network then that is likely to be a bottleneck.
But only measuring the properties of your storage+network+local machine would answer that question.
Linus Dock
2021-10-11
采纳的回答
addpath \\winfs\data\prod\maprod\arkivering\TAF;
Why do you append this folder to Matlab's path? Are the files stored in this folder or some required Matlab functions? If this folder does not contain Matlab functions, there is no reason to append it to the path. Simply omit the addpath/rmpath commands. See below.
What is the purpose of this:
DS = StartDTstr;
DE = StopDTstr;
TI = duration(timein,'InputFormat','hh:mm');
TO = duration(timeout,'InputFormat','hh:mm');
Are this variables or functions?
What exactly takes 17-19 seconds?
Is Data pre-allocated? If the final size is known, allocate it instead of letting the array grow iteratively:
numfiles = size(Z, 1); % Better than: length(Z(:,1));
Data = cell(1, numfiles);
for h=1:numfiles
filename = fullfile(P, Z(h).name); % Absolute path
fileID = fopen(filename);
Data{h} = textscan(fileID, '%s', 'delimiter', '=', 'headerlines', 1);
fclose(fileID);
end
It is more efficient to use the absolute path name because with fopen('file.txt') Matlab's complete path is searched.
Reading 53'000 files over a potentially slow network connection takes time. How large are the files? Maybe the runtime is limited by the network connection. Then there is no way to accelerate it magically.
4 个评论
Linus Dock
2021-10-10
Linus Dock
2021-10-10
This is from the profiler when using Run and Time:
Stephen23
2021-10-10
"Is this a wrong way of doing it"
Yes, that is the wrong way of doing it.
"Could you give me an example?"
Jan's answer already shows you an example of how to use an absolute filename.
Linus Dock
2021-10-10
更多回答（0 个）
类别
在 帮助中心 和 File Exchange 中查找有关 Search Path 的更多信息
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!
Translated by 
