This is an integer programming problem, and MATLAB does not have a function for that (although the FEX does have code for solving linear or quadratic mixed integer problems). However, you could cheat. Since your sum is equal to (a^(p+1)-a)/(a-1), where a = exp(1i*x), you could treat p as a real variable, fit it, then round off the answer.
Given the way you formulated the minimum (your f(x) being the square of a function g(x), and minimizing sum(g(x)^2-data^2), you are fitting the modulus of g(x) to the data, where g(x) is your sum of exponentials. I will generate some sample data assuming the integer is 6:
xdata = rand(100,1); n = 6;
a = cos(xdata);
ydata = abs((a.^(n+1)-a)./(a-1)) + 0.1*randn(size(xdata)); %Add some noise
Now define the function to fit
fun = @(p,x) abs((x.^(p+1)-x)./(x-1));
Fit to the data:
p0 = 3;
p = lsqcurvefit(fun,p0,cos(xdata),ydata);
p = round(p)
p =
6
If you want to put some constraints on your parameters, see the fields lb and ub in the structure options in the documentation for lsqcurvefit.
Note: revised to take into account new information.
