The function you pass to lsqnonlin must accept a vector of values the same size as your starting point, and must return a vector of function values (possibly a different size.)
Have another look at your fit_simp_optim code: you use A and B and n but those are not defined as variables anywhere. (What happened to C, by the way?)
Redefine your code:
function diff = fit_simp_optim(ABn, X, Y)
% This function is called by lsqnonlin.
% x is a vector which contains the coefficients of the
% equation. X and Y are the option data sets that were
% passed to lsqnonlin.
A = ABC(1); B = ABC(2); n = ABC(3);
diff = (A+B*(X.^n)) - Y;
end
Then code
X = xlsread('QS2T.xls',9,'V638:C1835');
Y = xlsread('QS2T.xls',9,'W638:W1835');
lb = [800;0;0];
ub = [1600;600;10];
StartAt = [800;1537;0.1448];
optABn = lsqnonlin( @(ABn) fit_simp_optim(ABn,X,Y), StartAt, lb, ub);
