Am wondering what will be best approach for parallelizing the code for the problem am working on.
I have a time series data (a trajectory) that is ~800,000 x2 matrix and I do some operations on it that involves matrix exponentiation, diagonalization & multiplications. It is a maximum likelihood (ML) calculation & since it produces 'Inf' for chunks of more than 1000 points (1000x2 matrix), I divide the long trajectory into pieces of 1000 points and do the required calculation on them (& the final ML is total additive of that of the pieces). Besides the time series data, I also have some other parameters. So my objFunc is like:
[MLtotal,MLsplit] = objFunc(params,trajectoryMat)
trajChunks = reshape(trajectoryMat,...
parfor i=1:length(trajChunks)
MLsplit(i) = calculateML(params,trajChunks)
end
MLtotal = sum(MLsplit)
calculateML actually is a fast calculation taking ~0.5 seconds and the serial caculation takes ~ 8 minutes for the whole Matrix (on 8 processor machine).
When I use the parfor with a 100 workers matlabpool, it takes like 8-10 seconds, which is a good scale-up.
Now, I need to do vary the parameters (params) and do the calculation like ~3000 times. (in fact, I want to do a parameter sweep).
paramsMat = [p1Vect; p2Vect;p3Vect ;p4Vect];
for ix = 1:length(paramsMat)
currentparams = paramsMat(ix,:);
[MLtot(ix),MLsplit(ix,:)] = objfun(currentparams,trajectoryData)
end
This takes like 24-30,000 seconds using 100 processors (our cluster size is 144 cores). Though it is scaling linearly, is there a better approach to do it?
It is the same program operating on multiple chunks of data in a way. So I was wondering if using spmd is a better choice ? How to use it ? (I haven't used spmd before)
Do you have any other suggestions for speeding it up.
