That your data is near infrared is not really relevant. :) Anyway, I'll see if I can shed some light on the problem.
I'll make up some random numbers here, as pretend data. Your data might have been more intelligently created, but still the same shape.
Thus nvar is 20, the number of variables in the problem, and ndata is 401, so the size of the current sample in the GA iteration.
nvar = 20;
ndata = 401;
predata = rand(401,20);
actdata = rand(20,1);
If this is for a genetic algorithm, then the presumption is you wish to compare the column vector actdata to EVERY ROW of predata. It would make no sense at all to compare it to predata(:), which simple unrolls the entire array into a single column.
You don't say what release of MATLAB you have. If it is a current or sufficiently recent release, then you can take advantage of anew trick. If not, I'll show you the alternative.
fitnessRMSE = sqrt(sum((predata - actdata').^2,2)/nvar);
The size of fitnessRMSE is a vector 401x1.
size(fitnessRMSE)
ans =
401 1
Given my garbage data, some vectors were closer than others, among the rows of predata.
min(fitnessRMSE)
ans =
0.26331
max(fitnessRMSE)
ans =
0.52635
As I said, the above computation will work in newer releases of MATLAB. If you have an older release, then you can use bsxfun. If you have a seriously old release (maybe 10 years or so at least), then you would need to use repmat.
Here is the solution using bsxfun.
fitnessRMSE = sqrt(sum(bsxfun(@minus,predata,actdata').^2,2)/nvar);
