reading multiple text file from two different folders using nested for loop

6 次查看（过去 30 天）
Salma fathi
Salma fathi 2022-3-31
编辑： Konrad 2022-3-31
i am using the following lines to read multiple text files and do some comparison between them. Baiscally the first loop will read text files of certain format (we have 30 files of this type). the second loop will read text files of another format (we have 33000 of them) and we do some comparison between them.
I have my code working and it is fine but it is extremely slow because of the nested loop, and its important to mention that i dont really need these loop for anything else, only for reading the files from folders.
So is there a more effecient way of doing this.
for n = 1 :nfiles2
fprintf('.......... File - %d of %d\n',n, nfiles2)
fullFileName2 = fullnames2{n};
t2 = readtable(fullFileName2); %t2 stores the ionosonde profiles
for m = 1 :10
fprintf('.......... insideFile - %d of %d\n',m, 10)
fullFileName1 = fullnames1{m};
t1 = readtable(fullFileName1); %t1 stores the ISR profiles

请先登录，再回答此问题。

回答（1 个）

Konrad
Konrad 2022-3-31
Ran in:
Hi Salma
It depends on what exactly you want to do with the data in the text files, but reading the file line by line seems to be considerably faster compared to readtable()
tic;
for k = 1:100
t = readtable('test.csv');
end
toc
Elapsed time is 1.449934 seconds.
tic;
for k = 1:100
fid = fopen('test.csv');
clear tl;
tl{1} = fgetl(fid);
while ischar(tl{end})
tl{end+1} = fgetl(fid);
end
fclose(fid);
end
toc
Elapsed time is 0.633903 seconds.
Of course readtable() converts numbers to double while fgetl() returns char and if you have to do type conversion the speed gain might disappear. But I think its worth trying.
Best, Konrad
  2 个评论
Salma fathi
Salma fathi 2022-3-31
thank you for your reply, I have tried what you suggested, it is much more faster than the way I used to do it.
but I get the content of the textfiles in the following form, each row of the file is stored in a cell
while we want it to be in the following form
Would there be a way to do that?
Konrad
Konrad 2022-3-31
编辑：Konrad 2022-3-31
Ran in:
Yes, thats what I ment with type-conversion ;)
I assume your data is comma delimited.
get the header in the first row:
fid = fopen('test.csv');
tline = fgetl(fid);
header = strsplit(tline,','); % <- the variable names as cellstring
If the data is all numeric (with variable names in the first row) you can do the following:
tic;
data = csvread('test.csv',1,0);
toc
Elapsed time is 4.558796 seconds.
Else: you can read the text data, store it in a cellstring and convert the numeric part (you will have to know which columns you need). This takes much longer due to the type conversion (and the growing array, which could be avoided by preallocating), but could be made faster if you only need a small fraction of the data for your comparison. But I don't know what you want to compare, so here's how you get everything from the test.csv:
tic;
dataCellstr = cell(0);
selectedCols = 1:3;
tline = fgetl(fid);
while ischar(tline)
cols = strsplit(tline,',');
dataCellstr(end+1,:) = cols(selectedCols);
tline = fgetl(fid);
end
data2 = str2double(dataCellstr);
fclose(fid);
toc
Elapsed time is 15.281463 seconds.

请先登录，再进行评论。

类别

在 Help Center 和 File Exchange 中查找有关 Data Import and Export 的更多信息

标签

产品


版本

R2021b

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by