1. Rough Idea of my Procedure - Juggle Nested Structures according to what Parameter I need to Operate On
I am reading in a huge amount of text files with spatial coordinates and time, then organzing the data into structures with nested fields. There is more than 1 level of nestedness, but for a mininal example I give 2 levels (in fact there are more, but I am attempting to not overwhealm with details):
myStructreadIn(t).circle(c).dat(theta data); Step (1) - read in data to a struct with nested fields
I am then taking the fft on a certain parameter, such as time, and to do that I am creating a new struct, and juggling the fields, time, so that it's on the end as a vector of size [1,numTimesteps], with the vector that was previously at the end:
myNewStructForTimeFft(c).azimuthal(theta).dat( time data ) (2)
and then, in order to take the fft on another parameter, i create a new struct, and again juggle the fields so that the last one is a vector of size [1,numAzimuthalPoints],
YetAnotherNewStructForTimeFft(t).azimuthal(theta).dat( azimuthal ); (3)
and then take its fft. And so on. Lots of juggling like this, lots of data, then doing fft, xcorr, etc.
Since I have quite a lot of data, moving the data around to the different structs is quite expensive operation, even with parfor loops.
2. Question about Improving my Procedure --- Eliminate unnecessary 'field juggling' Possible? Eg via eg structFun...?
My Question is: is there a way to take the fft on the structure in (1) without creating structure in (2) (ie, dont juggle time to the last field)? I was doing some reading on structFun, can that be used?
3. Another Issue I have: Parfor crashes, but Regular For Loops Work Fine Memorywise.
Structures with nested fields make parfor loops drastically increase ram usage, when I juggle the fields as in Steps (1)-(3). I have properly initialized my structs with eg
`myStruct=struct('t', repmat({struct('circle', repmat({ struct('dat',repmat({zeros(3,1079)}, [1,540]))}, [1,numTimesteps])) }, [1,numCrossSections]));`
Ideally, it seems that although parfor loops have some overhead, my current way of juggling subfields (as in 1-3 above) is increasing the ram to my workstation's 192gb (so that matlab crashes) from 30gb if a parfor loop is being used, it barely increases about 32gb if the code is ran in parallel for one such 'field juggle'. I would like to stick with using structures for now if it's not totally the wrong way to approach a lot of data, but using another data format would be fine too, if that would fit the bill (maybe using datastore and arranging in tables---although it's not clear how to use tables with this nested idea).
