I have some very large datasets with corresponding metadata from some large finite element analyses that I would like to try and put into tall arrays.
I struggle to load these datasets on my computer and often have to go to a high performance computing cluster to just get enough memory to load it. Once I have the data loaded, I usually just do statistics on various blocks of rows of the data. Tall arrays seem like a good solution for this but my data does not align with any of the examples I have seen.
My datasets are currently formatted into a matrix where the rows are specific nodes or element results I've recovered and the columns are the specific loadcases I've run (e.g. static loadcase 1, 2, 3, ...etc or frequency 1, 2, 3 ...etc Hz). I have a corresponding loadcase/frequency/time vector matching the number of columns (500 - 10,000 columns) and a table of metadata corresponding to each row of data (10 - 1E7 rows). The table of metadata include a name of the recovered result (e.g. Element ID 100230 End A Force X or Grid ID 2103 Direction X), the type of result it is (e.g. Acceleration, Force, Moment, Displacement), and the associated units (e.g. G, lbf, in*lbf, in).
I've considered using some of the table metadata and custom metadata but it kind of feels like I'm trying to shove a square peg into a round hole. I've also started to look into time series and time tables but haven't made it very far.
