Hessian in mlecov compared to hessian from fmincon

Question

zym 2022-8-27

0
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/1788185-hessian-in-mlecov-compared-to-hessian-from-fmincon

评论： Bruno Luong 2022-8-27

Hi, I am doing custom MLE estimation using fmincon. I want to compute asymptotic covariance matrix of MLE. To do so I could use mlecov or Hessian from fmincon. I was wondering if the hessian is estimated in mlecov in the same way as in fmincon? If not, what are possible advantages of mlecov?

Since I am running a custom problem, mlecov gives me an issue about positive definiteness. I am thinking to opt for the hessian as coming from the fmincon. However, the documentation suggest that the hessian output from fmincon is not precise https://nl.mathworks.com/help/optim/ug/hessian.html . What are other possible options? I identify three:

Use the maximised value of parameters from fmincon and plug it in fminunc to obtain the hessian, which seems to be correctly estimated there. Update: I still find that the Hessian is not-positive definite
Correct the hessian in mlecov to be positive definite (though I am not sure if it is a reliable procedure) Update: In the comment below, Matt J suggests a great function to that.
Drop using fmincon. Since my constraints are only non-negativity of variance, transform parameters and just use fminunc. Update: Hessian from fminunc still comes out to be non positive definite. However, mlecov does the magic. Once I plugin maxima from fmincon in the yet another custom function where parameters are transformed, such the problem is unconstrained, I find that mlecov is able to give me a positive-definite Hessian.

Did any of you happenned to have the same issue?

Thanks for your help. Regards!

2 个评论
显示无隐藏无

Matt J 2022-8-27

编辑：Matt J 2022-8-27

You could also just compute the Hessian yourself, analytically.

zym 2022-8-27

编辑：zym 2022-8-27

Thanks, it is a good point. But my problem is too complicated for the analytical solutions. Includes Kalman filter etc...

请先登录，再进行评论。

请先登录，再回答此问题。

Answer 1

Matt J 2022-8-27

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/1788185-hessian-in-mlecov-compared-to-hessian-from-fmincon#answer_1035455

编辑：Matt J 2022-8-27

I was wondering if the hessian is estimated in mlecov in the same way as in fmincon?

Probably, but even if not, the Hessian inversion will be sensitive to errors from finite difference computation.

You could try nearestSPD to find the nearest positive semi-definite matrix to whatever fmincon gives you,

https://www.mathworks.com/matlabcentral/fileexchange/42885-nearestspd?s_tid=ta_fx_results

Be mindful of a a few things, though,

(1) A positive semi-definite matrix is still not positive definite.

(2) The theory of approximating MLE covariance with Hessians is only valid when no constraints are active at the solution.

4 个评论
显示 2更早的评论隐藏 2更早的评论

Bruno Luong 2022-8-27

编辑：Bruno Luong 2022-8-27

"the Hessian inversion will be sensitive to errors from finite difference computation."

According to this page, https://uk.mathworks.com/help/optim/ug/hessian.html#bsapedt

the finite different only applied for trust region algorithm or when user sets HessianApproximation to 'finite-difference'.

The other two methods use quasi-newton, bfgs approximation of the Hessian. TMW warns that they are not accurate. I believe the bfgs and lbfgs are semi-postitive by design (the lbfgs has rank limited, about 10).

I have no idea about the relationship between mlecov (not my expertise) and the Hessian. I even don't know what is the objective function that links both? It it a log-likelyhood?

If I was you I probably do othe following:

find the parameters that maximize the function, then
compute the Hessian by finite-difference on the gradient, which is the in turn approximate by finite different.

The Hessian obtained this way is not symmetric, so you migh symmetrize it, then eventually perform an eigen-value decomposition, then set all the negative eigen value to 0 to make a non-negative symmetric matrix.

zym 2022-8-27

在 MATLAB Online 中打开

Thanks for this. However, trust region algorithm needs analytical gradient, which I do not have.

To obtain asymptotic covariance matric, mlecov computes Hessian using final differences. Inside the function, they write:

% Compute pure and mixed second order central differences of the
% log-likelihood function.  H is always a square matrix, regardless of
% the shape of params.  nH is the negative hessian, because loglikefun
% returns a negative log-likelihood.

Bruno Luong 2022-8-27

If that is the case, I would trust the Hessian returned by mlecov much more.

The Hessian used and returned by optimlization algorithms are designed for ... optimization, meaning the main purpose is to find a correct descend direction and not too costly. So I would not use it to make any further fine analysis.

请先登录，再进行评论。