I have a script that is working through files, comparing signals in those files, generating 'hits' where those signals are different from each other, then trying to analyze those hits to explain them. The results are written to a table, which looks like it will have ~2000 results per file. I'd like to append each file's results to the table, so that I can generate a more overall report and have better access to all the results afterwards. However, I am running these scripts on about 500 files, so this will be a very large table.
I've tried to read up on using tall objects in Matlab to work with larger files, but all the examples seem to be based on reading in from large input files using a datastore. In my case, the input files aren't large, there are just a lot of them. I thought that tall would be an excellent fit to my application, because during the analysis I only care about the specific file I'm working on, so I don't need access to all the previous results.
Is there a way to initially create a tall table, then just append as needed during the iteration? So I'm creating a table like this:
reportTable = cell2table(procReport(2:end,:));
reportTable.Properties.VariableNames = {'signalName','genesisSignalIndex',...
'motohawkSignalIndex','testNumber','testResults','genesisHitIndex',...
'motohawkHitIndex','genesisAvgInterval','motohawkAvgInterval'};
I pass that out from this function to the main function, and was hoping to append each file's results as they are done. But I know that it will be too large for memory, so I was hoping to use a tall table instead.
