Tall arrays are not allowed to contain data of type struct
3 次查看(过去 30 天)
显示 更早的评论
I am having trouble with tall arrays in the following workflow,
- Process data in a loop. In each iteration, generate structure containing results and save structure in a *.mat file
- Read all *.mat files into a datastore
- Turn datastore into a tall array. Gives the error "Tall arrays are not allowed to contain data of type struct"
- Turn tall array into a timetable
Here's an example, which I generated by modifying Edric Ellis's answer here. Can you help me eliminate the error?
%% Setup
nRowLimit = 100; % Number of records in each file
nCalculation = 20; % Number of calculation iterations and also mat files
tStart = datetime(2013, 11, 1, 8, 0, 0); % For later use in generating datetime array
%% Prepare to save matlab files
iFileToSave = 0;
dirSave = 'c:\temp';
%%
for iCalc = 1:nCalculation % Loop through iterations of the calculation
clear StrucResults; % Structure to hold results of calculations
% Generate structure containing pretend calculation results
for iRow = 1:nRowLimit
% Generate example datetime to fill a datetime array
StrucResults.date(iRow) = tStart + 1000*iFileToSave + iRow;
% Generate example character to fill a cell array
StrucResults.label{iRow} = char(round(rand(1)*100));
% Generate example value to fill a numerical matrix
StrucResults.white(iRow, :) = rand(1, 5);
end
%% Save structure containing results to small mat file
iFileToSave = iFileToSave + 1;
% Make a distinct file name
fileSave = fullfile(dirSave, sprintf('FileContainingStruct_%05d.mat', iFileToSave));
save(fileSave, 'StrucResults');
end
%% Read the data back in as a tall array
% First create a datastore from all the mat files
fds = fileDatastore(fullfile(dirSave, '*.mat'), ... % Files to load
'ReadFcn', @(x)struct2table(load(x)), ... % Try to convert structure to table to avoid error in next code line
'UniformRead', true);
% Read into a tall array
tallData = tall(fds); % Throws error "Tall arrays are not allowed to contain data of type struct"
% Change to a tall timetable
tallTimeTable = table2timetable(tallData);
It's fine to eliminate the structure in favor of storing the results in individual variables, but I couldn't get that to work either.
0 个评论
采纳的回答
Edric Ellis
2021-4-23
Your code was so nearly working. I made two tiny changes, marked with %###:
%% Setup
nRowLimit = 100; % Number of records in each file
nCalculation = 20; % Number of calculation iterations and also mat files
tStart = datetime(2013, 11, 1, 8, 0, 0); % For later use in generating datetime array
%% Prepare to save matlab files
iFileToSave = 0;
dirSave = fullfile(tempdir(), '809910');
if ~exist(dirSave, 'dir')
mkdir(dirSave);
end
%%
for iCalc = 1:nCalculation % Loop through iterations of the calculation
clear StrucResults; % Structure to hold results of calculations
% Generate structure containing pretend calculation results
%### Ensure vector fields are columns
for iRow = 1:nRowLimit
% Generate example datetime to fill a datetime array
StrucResults.date(iRow, 1) = tStart + 1000*iFileToSave + iRow;
% Generate example character to fill a cell array
StrucResults.label{iRow, 1} = char(round(rand(1)*100));
% Generate example value to fill a numerical matrix
StrucResults.white(iRow, :) = rand(1, 5);
end
%% Save structure containing results to small mat file
iFileToSave = iFileToSave + 1;
% Make a distinct file name
fileSave = fullfile(dirSave, sprintf('FileContainingStruct_%05d.mat', iFileToSave));
%### Use "save -struct" to ensure the fields of StrucResults are saved into the file directly.
save(fileSave, '-struct', 'StrucResults');
end
%% Read the data back in as a tall array
% First create a datastore from all the mat files
fds = fileDatastore(fullfile(dirSave, '*.mat'), ... % Files to load
'ReadFcn', @(x)struct2table(load(x)), ... % Try to convert structure to table to avoid error in next code line
'UniformRead', true);
% Read into a tall array
tallData = tall(fds); % Throws error "Tall arrays are not allowed to contain data of type struct"
% Change to a tall timetable
tallTimeTable = table2timetable(tallData);
% Check the results
head(tallTimeTable)
The first change was simply to ensure that the vectors in the struct were columns.
The second more important change was to use save -struct to get rid of the extra layer of "struct" in the MAT file.
更多回答(0 个)
另请参阅
类别
在 Help Center 和 File Exchange 中查找有关 Large Files and Big Data 的更多信息
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!