What is the fastest and smartest way to import and manage/plot many text files in matlab?
39 次查看(过去 30 天)
显示 更早的评论
Hi guys! I've processed my data in Fortran and I produced 55 .txt files (as shown in the following image). These files contains asteroids position stored in three columns (x,y,z coordinates); see an example file atteched below.
I'm looking for a smart and quick procedure to import them in matlab by using few code lines. Then I want to plot my data and maybe perform further computations. My idea is to store the files in a data structure subdivided by the number of asteroids (55 in my case), then create a loop that runs through all the files in my folder and maybe be able to stop at the last file by automatically determining the index value of said loop. In a few words I would like to create an automated procedure to import many files in a single variable that I can then manage for further calculations or plots in matlab. I found a code that seems to fit my request:
filenames = dir('D:\OneDrive\MSc_Thesis\Projects\NEOs_orbits\OutputFiles\Orbit_asteroid_*.txt');
number_of_files = numel(filenames); col_values = [];
for ii = 1:number_of_files
all_values = load(filenames(ii).name); %Error using load Unable to read file 'Orbit_asteroid_01.txt'. No such file or directory.
col_values = [col_values; all_values(:,1)];
end
But I get the error printed as comment. Can you help me, please?
0 个评论
采纳的回答
Walter Roberson
2022-2-9
编辑:Walter Roberson
2022-2-9
filename = 'https://www.mathworks.com/matlabcentral/answers/uploaded_files/888225/Orbit_asteroid_02.txt';
%"warm up" -- this will likely read the file into memory so we can remove
%the disk speed component
for K = 1 : 3; urlread(filename); end
%done warm-up -- mostly ignore the above, it is disk dominated
tic;
allvalues = readmatrix(filename);
toc
tic;
allvalues = table2array(readtable(filename));
toc
tic;
S = urlread(filename);
allvalues = cell2mat(textscan(S, '%f %f %f', 'HeaderLines', 3));
toc
tic;
L = regexp(urlread(filename), '\r?\n', 'split');
L(1:3) = [];
allvalues = cell2mat(cellfun(@(S) sscanf(S, '%f %f %f', [1 inf]), L, 'uniform', 0));
toc
tic;
L = regexp(urlread(filename), '\r?\n', 'split');
L(1:3) = [];
S = strjoin(L, '\n');
allvalues = sscanf(S, '%f %f %f', [3 inf]).';
toc
tic;
L = regexprep(urlread(filename), '^.*\n.*\n.*\n', '', 'once', 'dotexceptnewline');
allvalues = sscanf(L, '%f %f %f', [3 inf]).';
toc
tic;
L = regexprep(urlread(filename), '^.*\n.*\n.*\n', '', 'once', 'dotexceptnewline');
allvalues = cell2mat(textscan(L, '%f %f %f'));
toc
tic;
L = regexprep(urlread(filename), '^.*\n.*\n.*\n', '', 'once', 'dotexceptnewline');
allvalues = str2num(L);
toc
tic;
L = regexprep(urlread(filename), '^.*\n.*\n.*\n', '', 'once', 'dotexceptnewline');
T = tempname();
fid = fopen(T, 'w');
fwrite(fid, L);
fclose(fid);
allvalues = load(T);
toc
I had a bug in the earlier version of some of the conversions, so these conclusions are revised:
Fastest: read the file as text, use text processing to remove the header, textscan() the result. Oddly this was notably faster than textscan() of the full data with 'headerliens'
Second fastest: various text processing methods are close enough to each other over multiple runs that I cannot judge between them. That is, using this test facility, the times varied enough that the answer is not clear. Running on your own system would likely produce a different result.
Of particular note: read the file as a string, removing the header, writing the file out, and using load(), is still about 5 times faster than using readtable()
In your real system, it might turn out that using fopen() to open the file, and then use textscan(), might work out as the fastest. It is a bit difficult to benchmark for the case where the file is not already in filesystem cache, without risking the possibility that as you run the several tests, that the second and following tests might be taking advantage of the file system already having cached the file.
3 个评论
Walter Roberson
2022-2-9
struct is certainly do-able.
projectdir = 'D:\OneDrive\MSc_Thesis\Projects\NEOs_orbits\OutputFiles';
dinfo = dir( fullfile(projectdir, 'Orbit_asteroid_*.txt') );
filenames = fullfile( {dinfo.folder}, {dinfo.name});
number_of_files = numel(filenames);
Astroid(number_of_files) = struct('x', [], 'y', [], 'z'); %pre-allocates
for ii = 1:number_of_files
thisfilename = filenames{ii};
all_values = CodeToLoadTheFile(thisfilename);
Astroid(ii).x = all_values(:,1);
Astroid(ii).y = all_values(:,2);
Astroid(ii).z = all_values(:,3);
end
but cell array might have higher performance, perhaps
projectdir = 'D:\OneDrive\MSc_Thesis\Projects\NEOs_orbits\OutputFiles';
dinfo = dir( fullfile(projectdir, 'Orbit_asteroid_*.txt') );
filenames = fullfile( {dinfo.folder}, {dinfo.name});
number_of_files = numel(filenames);
Astroid = cell(number_of_files, 1);
for ii = 1:number_of_files
thisfilename = filenames{ii};
all_values = CodeToLoadTheFile(thisfilename);
Astroid{ii} = all_values;
end
which would
plot3(Asteroid{ii}(:,1), Asteroid{ii}(:,2), Asteroid({ii}(:,3))
更多回答(2 个)
David Hill
2022-2-8
As long as you are in the folder you want, you just need the file name.
for k=1:55
f=num2str(k);
if length(f)==1
f=strcat('0',f);
end
r(:,(k-1)*3+1:3*k)=readmatrix(strcat('Orbit_asteroid_',f));
end
%then plot what you want
plot3(r(:,4),r(:,5),r(:,6));%2nd asteroid
0 个评论
Highphi
2022-2-8
you can't use 'load' here, you can only load MAT-files or certain ASCII files
best way to import these files is by using:
all_values = readtable(filenames(ii).name);
all_values = table2array(all_values);
Good luck
1 个评论
Walter Roberson
2022-2-17
Yes and no.
You are correct that the original format of the file is something that load() would not process. And in cases where clarity and convenience is important (which is a lot of the time, really), using one of the other possibilities instead of load() would be a better choice.
But as I showed near the end of https://www.mathworks.com/matlabcentral/answers/1645930-what-is-the-fastest-and-smartest-way-to-import-and-manage-plot-many-text-files-in-matlab#answer_891795 it is possible to read the file, strip off the header, write the the revised content, and load() the revised content, and still end up with a processing time that is competitive with nearly all the more convenient approaches. At least until the file starts taking a major fraction of your memory, or until the hard-drive speed starts to dominate.
另请参阅
类别
在 Help Center 和 File Exchange 中查找有关 Database Toolbox 的更多信息
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!