trust-region reflective algorithm in lsqnonlin and fmincon: same behavior for "unconstrained" optimization"?

Question

SA-W 2023-2-15

1
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/1913280-trust-region-reflective-algorithm-in-lsqnonlin-and-fmincon-same-behavior-for-unconstrained-optimi

编辑： Matt J 2023-2-16

Say I provide constraints to lsqnonlin and fmincon. For both, I will choose trust-region-reflective algorithm

Do both fmincon and lsqnonlin behave identical in case the constraints are not active (unconstrained optimization)?

By identical, I mean is the same variant of trust-region-reflective algorithm implemented in fmincon and lsqnonlin, or are they different?

0 个评论
显示 -2更早的评论隐藏 -2更早的评论

请先登录，再进行评论。

请先登录，再回答此问题。

Answer 1

Matt J 2023-2-15

1
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/1913280-trust-region-reflective-algorithm-in-lsqnonlin-and-fmincon-same-behavior-for-unconstrained-optimi#answer_1172470

编辑：Matt J 2023-2-15

No, they are not the same. They both take steps in a 2D subspace, but for fmincon, the basis vectors for the subspace are determined from the exact Hessian (see equations (3,4) here), whereas for lsqnonlin, they are determined from the Jacobian (see equation (6) here).

4 个评论
显示 2更早的评论隐藏 2更早的评论

Matt J 2023-2-15

编辑：Matt J 2023-2-16

在 MATLAB Online 中打开

How does Matlab find the exact Hessian (for fmincon) if I only supply the gradient along with the objective?

If you only supply the gradient, then fmincon's Hessian calculation will not be "exactly exact". It will compute the Hessian by finite differences. The point though, is that lsqnonlin won't even try to approximate the exact Hessian. It will always rely on the Jacobian of the residuals exclusively, which is not theoretically equivalent to a Hessian calculation except in specific cases like when your residual function is linear.

EDIT: Below is an example showing how different results can be reached by lsqnonlin and by fmincon both with an analytical and finite difference Hessian computation. Note that the final resnorm is in all cases the same.

clc,runIt(false)
 
LSQNONLIN
x = 1×3
    0.4234    2.8138    2.2643
resnorm = 0.2446
 
FMINCON
x = 1×3
   -5.3427   -2.9523   -3.5029
resnorm = 0.2446
 
FMINCON - ONLY GRADIENT
x = 1×3
  735.0770  737.4673  736.9168
resnorm = 0.2446
function runIt(chkGrad)
 A=[0.7989
    0.9302
    0.0047
    0.6500]*[1,1,-2];
 
e=2*ones(1,4)';
x0=[0.7646    3.1550    1.5819];
disp(' ');disp('LSQNONLIN')
opts=optimoptions('lsqnonlin','Algorithm','trust-region-reflective','SpecifyObjec',true,'Display','none','CheckGrad',chkGrad);
[x,resnorm]=lsqnonlin(@resfun,x0,[],[],opts)
disp(' ');disp('FMINCON')
opts=optimoptions('fmincon','Algorithm','trust-region-reflective','SpecifyObjec',true,'Display','none',...
                           'HessianFcn','objective','CheckGrad',chkGrad);
[x,resnorm]=fmincon(@objfun,x0,[],[],[],[],[],[],[],opts)
disp(' ');disp('FMINCON - ONLY GRADIENT')
opts=optimoptions('fmincon','Algorithm','trust-region-reflective','SpecifyObjec',true,'Display','none');
[x,resnorm]=fmincon(@objfun,x0,[],[],[],[],[],[],[],opts)
    function [r,J,z]=resfun(x)
 
        x=x(:);
        z=A*x+1;
        r=z.^e./e;
        J=z.^(e-1).*A;
    end
    function [fval,grad,Hess]=objfun(x)
        [r,J,z]=resfun(x);
        fval=norm(r).^2;
        grad=2*(J.' * r);
        if nargout>2
         Hess=A.'*diag((2.*z.^(2.*e - 1))./e)*A;
        end
        
    end
end

SA-W 2023-2-16

Interestingly, the three solutions differ only by a constant. Would you consider the solution "FMINCON - ONLY GRADIENT" to be the worst of the three solutions?

I obtained very good results with lsqnonlin (trust-region-reflective), but now my problem has both bounds and inequality constraints, which is why I have to switch to fmincon. If both bounds and inequality constraints are present, I can not use trust-region-reflective algorithm anymore. Which of the algorithm in fmincon resembles the trust-region-reflective algorithm most closely?

Matt J 2023-2-16

编辑：Matt J 2023-2-16

Interestingly, the three solutions differ only by a constant.

Because [1;1;1] is in the null space of A.

Would you consider the solution "FMINCON - ONLY GRADIENT" to be the worst of the three solutions?

They all achieve the same resnorm so how could it be worse than any of the others?

Which of the algorithm in fmincon resembles the trust-region-reflective algorithm most closely?

I don't think a comparison can be drawn. Inequality constraints are going to change the algorithm a lot.

请先登录，再进行评论。

trust-region reflective algorithm in lsqnonlin and fmincon: same behavior for "unconstrained" optimization"?

0 个评论
显示 -2更早的评论隐藏 -2更早的评论

采纳的回答

4 个评论
显示 2更早的评论隐藏 2更早的评论

更多回答（0 个）

另请参阅

类别

标签

产品

Community Treasure Hunt

trust-region reflective algorithm in lsqnonlin and fmincon: same behavior for "unconstrained" optimization"?

0 个评论 显示 -2更早的评论隐藏 -2更早的评论

采纳的回答

4 个评论 显示 2更早的评论隐藏 2更早的评论

更多回答（0 个）

另请参阅

类别

标签

产品

Community Treasure Hunt

0 个评论
显示 -2更早的评论隐藏 -2更早的评论

4 个评论
显示 2更早的评论隐藏 2更早的评论