How to efficiently extract 2-D data from multiple text files and save it as a single 3-D variable?

Question

Austin M. Weber 2022-4-6

0
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/1690390-how-to-efficiently-extract-2-d-data-from-multiple-text-files-and-save-it-as-a-single-3-d-variable

评论： Voss 2022-4-8

I have 456 comma-delimited text files. Each file represents one month of climate data, containing 600 rows and 900 columns.

I want to extract the data from each text file and store it in one three-dimensional variable (600x900x456).

The names of the text files are standardized. For example, the first file is called 'data198201.txt' (which means refers to January 1982), the second file is called 'data198002' (which refers to February 1982), and so forth all the way to December 2019.

I have a successful way of extracting the data for individual 12-month periods, for example, Jan-1982 to Dec-1982:

% Vector of months
mon = [{'01'};{'02'};{'03'};{'04'};{'05'};{'06'};{'07'};{'08'};{'09'};{'10'};{'11'};{'12'}];
% Extract the monthly data for 1980
data1982 = zeros(600,900,12); % Preallocate memory
for b = 1:12
	filename = ['data1982',mon{b},'.txt']; 
	data = table2array(readtable(filename));
	data1982(:,:,b) = data;
end

But for efficiency I want to create a loop that goes through all 456 months at once. Here is what I tried:

% Create a variable of years as cells
a = 1982:2019;
y = [];
for n = 1:length(a)
	ycell = {num2str(a(n))}; % save year as a cell
	y = [y ycell]; % final output: [{'1982'},{'1983'},{'1984'},...,{'2019'}]
end
% Extract and store all the data as one variable
alldata = zeros(600,900,456); % pre-allocate memory
for n = 1:length(y)
for b = 1:456
for c = 1:12
	filename = ['data',y{n},mon{c},'.txt'];
	data = table2array(readtable(filename));
	alldata(:,:,b) = data;
end
end
end

I had the loop run for ~20 minutes before I came to the conclusion that I either did something wrong or that there must be a more efficient way to do this. I would greatly appreciate any corrections or tips that I can try to implement.

0 个评论
显示 -2更早的评论隐藏 -2更早的评论

请先登录，再进行评论。

请先登录，再回答此问题。

Answer 1

Voss 2022-4-7

1
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/1690390-how-to-efficiently-extract-2-d-data-from-multiple-text-files-and-save-it-as-a-single-3-d-variable#answer_936600

在 MATLAB Online 中打开

Using sprintf to construct your file names will make that easier. Also, you only need one additional loop (loop over years) to wrap around the loop over months you already had.

a = 1982:2019;
alldata = zeros(600,900,numel(a)*12); % pre-allocate memory
for n = 1:numel(a)
    for c = 1:12
        filename = sprintf('data%4d%02d.txt',a(n),c);
        % does readmatrix work for these files?
        % instead of reading to a table and immediately converting to an array
        data = readmatrix(filename);
%         data = table2array(readtable(filename));
        alldata(:,:,12*(n-1)+c) = data;
    end
end