How to call data from files named with dynamic index

Question

H Mueller 2020-11-17

0
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/651013-how-to-call-data-from-files-named-with-dynamic-index

编辑： Stephen23 2020-11-20

I know that a similar question has been turned down many times as we don't EVER want to use dynamic vaiable names in programming. The fact is that I'm trying to make my life easier when plotting measurement results. The files are named after the sample as in sample001.csv or SampleName_<Date>.txt (and I cannot change the export or naming format). All I want to do is:

% for each of these text files, plot a set column as x and another as y
figure
hold on
% k is a high number and the number of files to import for plotting
for n = 1:k
    plot(sample<n>(:,14),sample<n>(:,17))
end

I can think of two ways to do this:

Break the Rules of Matlab and somehow implement the dynamic variable name anyway (since the files per default imprt with the file name as variable name). I've tried some things with num2str or putting a char or string variable into plot but couldn't get it to work
Get the files to import to a multidimensional (in my case 3D) array or table from which I can call the data in a conventional method

I presume the second will be the experts' choice, therefore, how can I

select a great number of files for importing (best from a Windows Explorer "open a file" dialog)
forget the header in the files (it would be nice to keep the column names but it makes accessing the rest of the data so much more complicated so I'd rather scrap the header and import the actual data as numeric matrix)
make sure the decimal separator is correct
import them in a way that I can access the data of "sample345.txt" as array(:,:,345) or something like that
get the whole thing to be robust so that it works if the data has different (arbitrary) height (and possibly width) in each file, contains strings as well as numbers or files are missing

Any help much appreciated.

______

PS: I've come up with something similar already but the problem is that it only works for numeric data with no header etc. and overwrites the variable every time. Sorry for the syntax, I'm not a programmer

for k=(1:3:72)
	textFileName = ['sample_' num2str(k) '.txt'];
	textData = readmatrix(textFileName);
	sz = size(textData);
	x = sz(1);
	dehnung = textData(:,4);
	spannung = textData(:,5);
	displacement = textData(:,10);
	length = textData(1,17);
	z=displacement(1);
	for i = 1:x
		displacement(i) = displacement(i) - z;
		dehnung(i) = displacement(i)/(length(1)*1000)*100;
	end
	
	plot(dehnung,spannung)
	hold on
end

1 个评论
显示 -1更早的评论隐藏 -1更早的评论

Peter Perkins 2020-11-19

What about (3) read each file into a field of a scalar struct named for the file?

请先登录，再进行评论。

请先登录，再回答此问题。

Answer 1

Stephen23 2020-11-19

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/651013-how-to-call-data-from-files-named-with-dynamic-index#answer_550008

编辑：Stephen23 2020-11-19

"select a great number of files for importing (best from a Windows Explorer "open a file" dialog)"

If you really have a "great number" of files your user is going to dislike this way of selecting them. Most likely simply selecting the folder and using dir with a filename filter would be simpler and less fiddly to use. But each to their own...

[C,P] = uigetfile(..,'MultiSelect','on');
N = numel(C);
D = cell(1,N);
for k = 1:N
    F = fullfile(P,C{k});
    M = .. whatever file importing function you want to use
    .. whatever processing you want to do
    D{k} = M;
end

You can access the filenames and filedata using basic indexing, e.g. for the third file:

C{3} % filename
D{3} % imported data

https://www.mathworks.com/help/matlab/matlab_prog/access-data-in-a-cell-array.html

A simple alternative is to import the data into the same non-scalar structure as dir returns, e.g.:

P = 'absolute/relative path to the folder where the files are saved';
S = dir(fullfile(P,'sample_*.txt'));
for k = 1:Nnumel(S)
    F = fullfile(P,S(k).name);
    M = .. whatever file importing function you want to use
    .. whatever processing you want to do
    S(k).data = M;
end

2 个评论
显示无隐藏无

H Mueller 2020-11-19

编辑：H Mueller 2020-11-19

Great, the wavy parentheses did the trick. I find it very confusing when to use which kind. Thank you!

Here is the solution I derived from your remarks. It outputs an array that contains the data and a string array with the corresponding sample names. Calculations, plotting etc. can be done at the same time.

% Select files to import
[files,path] = uigetfile('D:\filepath\*.*','MultiSelect', 'on');
% Separate file name from extension (optional, could do with "files")
i = 1;
for fn = files
[pathx,namex,extx] = fileparts(append(path,"\",fn));
sample(i) = cellstr(namex);
i = i+1;
end
% Import data to struct
for n = 1:length(files)
    data{n} = readtable([path,files{n}]);
end
% Do things with the data (in my case plot)
for k = 1:length(files)
    figure
    hold on
    plot(data{k}{:,19},data{k}{:,28:end})
end
clearvars -except sample data 

Stephen23 2020-11-20

编辑：Stephen23 2020-11-20

Do NOT use the variable path as this will shadow the important inbuilt path function.

You don't need to add the filepath just to remove it again with fileparts.

I recommend looping over indices rather than looping over data values, it is usually simpler.

I strongly recommend that you preallocate the cell arrays before the loops.

I strongly recommend that you use fullfile (the correct tool) instead of concatenating strings.

[C,P] = uigetfile('D:\filepath\*.*','MultiSelect', 'on');
N = numel(C);
D = cell(N,1); % preallocate
[~,S] = fileparts(C); % sample
% Import data to cell array:
for k = 1:N
    F = fullfile(P,C{k});
    D{k} = readtable(F);
end

请先登录，再进行评论。

Answer 2

Steven Lord 2020-11-17

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/651013-how-to-call-data-from-files-named-with-dynamic-index#answer_547383

编辑：Steven Lord 2020-11-17

%{
% for each of these text files, plot a set column as x and another as y
figure
hold on
% k is a high number and the number of files to import for plotting
for n = 1:k
    plot(sample<n>(:,14),sample<n>(:,17))
end
%}

Where are you importing the files? Before you create the figure (where the first comment is located) or inside the for loop (omitted to make the example simpler?)

Do you need to store all the data in memory at once?

If you don't need all the data in memory at once, you don't need to store the data in different named variables. In this example I'm reusing the variable y in the loop.

% Sample data
x = 0:360;
% Set up figure
figure
axis([0 360 -1 1]);
hold on
% Y will simulate the data loaded from your files
for k = 1:5
    y = sind(k*x);
    h = plot(x, y);
end
shg

If you do need all the data at once, and each data set should be associated with its filename, consider using a struct or something similar.

x = 0:360;

axis([0 360 -1 1]);

hold on

% Create struct to hold data

allData = struct('multiplier', {}, 'yData', {}, 'handle', {});

for k = 1:5

y = sind(k*x);

h = plot(x, y);

allData(k).multiplier = k;

allData(k).yData = y;

allData(k).handle = h;

end

disp(allData(4))

multiplier: 4 yData: [1×361 double] handle: [1×1 Line]

5 个评论
显示 3更早的评论隐藏 3更早的评论

H Mueller 2020-11-19

编辑：H Mueller 2020-11-19

Doesn't really matter whether I first import all files and then have a bunch of variables to deal with or import it to the struct right away. At some point I need to extract the index and convert it into a 3rd dimension column or struct array field. And it would be nice to have all data in memory to then be able to plot one graph again with modifications without having to touch the rest (i.e. run the whole script again).

I can see that a struct (of which I didn't know it existed) would be an adequate fomat to keep the date. The problem persists that I don't know how to store the data into it.

For simplicity let's say the data is already imported into tables: data1 (27x35), data2 (303x35), and data3 (45x35). In this case the header has already been omitted (i.e. converted into the column names) and the first column is some arbitrary string; the rest is the numerical data I need. If we now want to put that into the struct we stand before the same problem (or, instead of putting it into the struct could do the plotting right away):

% Create struct to hold data
allData = struct('ColumnHeaders', {}, 'RowHandle', {}, 'Data', {});
% transfer data to struct
for K = 1:3
    allData(K).ColumnHeaders = dataK.Properties.VariableNames; %here I need to call dataK, which of course I can't do beacuse it is dynamically named
    allData(K).RowHandle = dataK{:,1};
    allData(K).Data = dataK{:,2:end};
end

I am also not sure if I understand the concept of how a structure works correctly.

Steven Lord 2020-11-19

You may need to use matlab.lang.makeValidName or perform some other processing to create the struct field name if the filenames are not valid MATLAB identifiers.

name = ["sample 001"; "grades (MA101 Fall 2020)"];
matlab.lang.makeValidName(name)
ans = 2×1 string array
    "sample001"
    "grades_MA101Fall2020_"

H Mueller 2020-11-19

编辑：H Mueller 2020-11-19

I don't understand how to feed the data into the struct. Somehow the fields need to be named beforehand and I can't name them after every entry of a string array? Here is what I have so far:

% Select files to import
[files,path] = uigetfile('D:\Path\*.*','MultiSelect', 'on');
% Separate file name from extension (is there a simpler way?)
i = 1;
for fn = files
[pathx,namex,extx] = fileparts(append(path,"\",fn));
file(i) = cellstr(namex);
i = i+1;
end
% Import Data to struct
AllData = struct
for n = (1:length(file))
    import = readtable([path,files{n}]);
    SampleName = file{n};
    ColumnHeaders = import.Properties.VariableNames;
    RowNames = (import{:,1});
    Data = table2array(import(:,2:end));
    AllData.(SampleName) = struct(ColumnHeaders{:}; RowNames{:}, Data(:)); % please fix this line
end

(Why is it an such an absolute mess if I put import into the struct? That would be way easier, but it puts every property associated with the table into a separate field.)

[EDIT]: Stephen proposed a simple way to do precisely that. Problem solved. Thanks for your answers!

请先登录，再进行评论。

How to call data from files named with dynamic index

1 个评论
显示 -1更早的评论隐藏 -1更早的评论

采纳的回答

2 个评论
显示无隐藏无

更多回答（1 个）

5 个评论
显示 3更早的评论隐藏 3更早的评论

另请参阅

类别

标签

产品

版本

Community Treasure Hunt

How to call data from files named with dynamic index

1 个评论 显示 -1更早的评论隐藏 -1更早的评论

采纳的回答

2 个评论 显示 无隐藏 无

更多回答（1 个）

5 个评论 显示 3更早的评论隐藏 3更早的评论

另请参阅

类别

标签

产品

版本

Community Treasure Hunt

WeChat

1 个评论
显示 -1更早的评论隐藏 -1更早的评论

2 个评论
显示无隐藏无

5 个评论
显示 3更早的评论隐藏 3更早的评论