You are not able to change the maximum function evaluation because you are not passing the option to fsolve(). Being said that, fsolve() is just the wrong tool for minimizing the objective function. Choose an optimizer such as fmincon() to minimize the objective function. Here is how you can do this
[la, lb, lc] = textread('zt4m.csv','%f, %f, %f');
x = [la, lb, lc] ;
s = size(x);
% la,lb,lc are arm lengths used in dc to get to dm
dm = [0, 5.46, 18.3, 9.45, 13.49, 19.89]'; % the measured value
sd = size(dm);
li = la;
lin = lc;
vec = ones(size(li));
options = optimoptions('fmincon', 'MaxFunctionEvaluations', 10e4, 'MaxIter', 10e4);
[X,Fval] = fmincon(@(x) sum(objFun(li,lin,x(1),x(2),x(3),x(4),dm).^2),[400 400 480 73], [], [], [], [], [], [], [], options)
dc = (li + vec*X(1)).^2 + (lin + vec*X(2)).^2 - 2.*(li.*lin + li.*X(2) + lin.*X(1) + vec.*X(1).*X(2)).*cosd(X(3));
dzc = dc - (X(4) - 0.0001)^2;
dc = sqrt(dzc)
function eps = objFun(li,lin,Di,Din,beta,b,dzm)
vec = ones(size(li));
dc = (li + vec*Di).^2 + (lin + vec*Din).^2 - 2.*(li.*lin + li.*Din + lin.*Di + vec.*Di.*Din).*cosd(beta);
dzc = dc - b^2;
eps = (dzm.^2 - dzc); % i would like to replace this with (dzm - sqrt(dzc)).^2
end
