Unable to resolve the warning on ill conditioned Jacobian

12 次查看(过去 30 天)
I want to fit my data (1st column: x , 2nd column: y, given in the text file) to a sigmoidal function using the given function file (sigm_fit_base_e.m) The function is the standard matlab function which I modified from base 10 to base e and increased the maxIter to 10000. My initial guess parameters are:
[0 0.2845 9.88 -1] and there are no fixed parameters.
Relevant code lines are:
fPar = sigm_fit_base_e(x,y,[],[0 0.2845 9.88 -1],0);
I get the following warning:
Warning: The Jacobian at the solution is ill-conditioned, and some model parameters may not be estimated well (they are not identifiable). Use caution in making predictions.
> In nlinfit (line 384)
In sigm_fit_base_e (line 130)
I checked all the related answers in the Matlab community and tried to play around by modifiying the initial guess parameters and fixing the 2nd parameter but the warning still persists. Could you please help me fix this warning?
M = importdata('datai2j3.txt');
x = M(:,1);
y = M(:,2);
fPar = sigm_fit_base_e(x,y,[],[0 0.2845 9.88 -1],1)
f = function_handle with value:
@(param,xval)param(1)+(param(2)-param(1))./(1+exp((param(3)-xval)*param(4)))
ans = 1x4
0 0.2845 9.8800 -1.0000
<mw-icon class=""></mw-icon>
<mw-icon class=""></mw-icon>
Warning: The Jacobian at the solution is ill-conditioned, and some model parameters may not be estimated well (they are not identifiable). Use caution in making predictions.
fPar = 1x4
-176.4593 0.2818 11.5553 -5.6354
<mw-icon class=""></mw-icon>
<mw-icon class=""></mw-icon>
function [param,stat]=sigm_fit_base_e(x,y,fixed_params,initial_params,plot_flag)
% Optimization of parameters of the sigmoid function
%
% Syntax:
% [param]=sigm_fit(x,y)
%
% that is the same that
% [param]=sigm_fit(x,y,[],[],[]) % no fixed_params, automatic initial_params
%
% [param]=sigm_fit(x,y,fixed_params) % automatic initial_params
% [param]=sigm_fit(x,y,[],initial_params) % use it when the estimation is poor
% [param]=sigm_fit(x,y,fixed_params,initial_params,plot_flag)
%
% param = [min, max, x50, slope]
%
% if fixed_params=[NaN, NaN , NaN , NaN] % or fixed_params=[]
% optimization of "min", "max", "x50" and "slope" (default)
%
% if fixed_params=[0, 1 , NaN , NaN]
% optimization of x50 and slope of a sigmoid of ranging from 0 to 1
%
%
% Additional information in the second output, STAT
% [param,stat]=sigm_fit(x,y,fixed_params,initial_params,plot_flag)
%
%
% Example:
% %% generate data vectors (x and y)
% fsigm = @(param,xval) param(1)+(param(2)-param(1))./(1+exp((param(3)-xval)*param(4)))
% param=[0 1 5 1]; % "min", "max", "x50", "slope"
% x=0:0.1:10;
% y=fsigm(param,x) + 0.1*randn(size(x));
%
% %% standard parameter estimation
% [estimated_params]=sigm_fit(x,y)
%
% %% parameter estimation with forced 0.5 fixed min
% [estimated_params]=sigm_fit(x,y,[0.5 NaN NaN NaN])
%
% %% parameter estimation without plotting
% [estimated_params]=sigm_fit(x,y,[],[],0)
%
%
% Doubts, bugs: rpavao@gmail.com
% Downloaded from http://www.mathworks.com/matlabcentral/fileexchange/42641-sigmoid-logistic-curve-fit
% warning off
x=x(:);
y=y(:);
if nargin<=1 %fail
fprintf('');
help sigm_fit
return
end
automatic_initial_params=[quantile(y,0.05) quantile(y,0.95) NaN 1];
if sum(y==quantile(y,0.5))==0
temp=x(y==quantile(y(2:end),0.5));
else
temp=x(y==quantile(y,0.5));
end
automatic_initial_params(3)=temp(1);
if nargin==2 %simplest valid input
fixed_params=NaN(1,4);
initial_params=automatic_initial_params;
plot_flag=1;
end
if nargin==3
initial_params=automatic_initial_params;
plot_flag=1;
end
if nargin==4
plot_flag=1;
end
if exist('fixed_params','var')
if isempty(fixed_params)
fixed_params=NaN(1,4);
end
end
if exist('initial_params','var')
if isempty(initial_params)
initial_params=automatic_initial_params;
end
end
if exist('plot_flag','var')
if isempty(plot_flag)
plot_flag=1;
end
end
%p(1)=min; p(2)=max-min; p(3)=x50; p(4)=slope
%f = @(p,x) p(1) + (p(2)-p(1)) ./ (1 + exp((p(3)-x)*p(4))); % Modified function form with base e
f_str='f = @(param,xval)';
free_param_count=0;
bool_vec=NaN(1,4);
for i=1:4;
if isnan(fixed_params(i))
free_param_count=free_param_count+1;
f_str=[f_str ' param(' num2str(free_param_count) ')'];
bool_vec(i)=1;
else
f_str=[f_str ' ' num2str(fixed_params(i))];
bool_vec(i)=0;
end
if i==1; f_str=[f_str ' + (']; end
if i==2;
if isnan(fixed_params(1))
f_str=[f_str '-param(1) )./ ( 1 + exp( (']; % Modified here for base e
else
f_str=[f_str '-' num2str(fixed_params(1)) ')./ (1 + exp((']; % Modified here for base e
end
end
if i==3; f_str=[f_str ' - xval ) *']; end
if i==4; f_str=[f_str ' ) );']; end
end
eval(f_str)
%------added by me--------
% Set optimization options
options = statset('nlinfit');
options.MaxIter = 10000; % Set the maximum number of iterations here
%-------------------------
f
initial_params(bool_vec==1)
[BETA,RESID,J,COVB,MSE] = nlinfit(x,y,f,initial_params(bool_vec==1),options);
stat.param=BETA';
% confidence interval of the parameters
stat.paramCI = nlparci(BETA,RESID,'Jacobian',J);
% confidence interval of the estimation
[stat.ypred,delta] = nlpredci(f,x,BETA,RESID,'Covar',COVB);
stat.ypredlowerCI = stat.ypred - delta;
stat.ypredupperCI = stat.ypred + delta;
% plot(x,y,'ko') % observed data
% hold on
% plot(x,ypred,'k','LineWidth',2)
% plot(x,[lower,upper],'r--','LineWidth',1.5)
free_param_count=0;
for i=1:4;
if isnan(fixed_params(i))
free_param_count=free_param_count+1;
param(i)=BETA(free_param_count);
else
param(i)=fixed_params(i);
end
end
if plot_flag==1
x_vector=min(x):(max(x)-min(x))/100:max(x);
plot(x,y,'k.',x_vector,f(param(isnan(fixed_params)),x_vector),'r-')
xlim([min(x) max(x)])
end
end
  4 个评论
Torsten
Torsten 2024-8-8
编辑:Torsten 2024-8-12
the function is constant over x -values. Also, visually the function seems to fit the data quite well, so I am not able to resolve the warning.
Read @John D'Errico 's answer for an explanation. Your data are simply not suited for this fitting function (and vice versa).
Your fitting function assumes that you have data for a sigmoid, but all you have are data for a "one-sided" sigmoid. The asymptote of your data for x -> +Inf is unknown.
Akankshya Majhi
Akankshya Majhi 2024-8-14
@Torsten: Thank you. Yes, makes sense. Do you have any idea if there is no other suitable model to fit the data and get the relevant parameter (drop off point)?

请先登录,再进行评论。

回答(2 个)

dpb
dpb 2024-8-8
编辑:dpb 2024-8-8
M = readmatrix('datai2j3.txt');
%fPar = sigm_fit_base_e(M(:,1),M(:,2),[],[0 0.2845 9.88 -1],1)
fPar = sigm_fit_base_e(M(:,1),M(:,2),[0 nan nan nan])
f = function_handle with value:
@(param,xval)0+(param(1)-0)./(1+exp((param(2)-xval)*param(3)))
fPar = 1x4
0 0.2817 10.3666 -6.0641
<mw-icon class=""></mw-icon>
<mw-icon class=""></mw-icon>
function [param,stat]=sigm_fit_base_e(x,y,fixed_params,initial_params,plot_flag)
% Optimization of parameters of the sigmoid function
%
% Syntax:
% [param]=sigm_fit(x,y)
%
% that is the same that
% [param]=sigm_fit(x,y,[],[],[]) % no fixed_params, automatic initial_params
%
% [param]=sigm_fit(x,y,fixed_params) % automatic initial_params
% [param]=sigm_fit(x,y,[],initial_params) % use it when the estimation is poor
% [param]=sigm_fit(x,y,fixed_params,initial_params,plot_flag)
%
% param = [min, max, x50, slope]
%
% if fixed_params=[NaN, NaN , NaN , NaN] % or fixed_params=[]
% optimization of "min", "max", "x50" and "slope" (default)
%
% if fixed_params=[0, 1 , NaN , NaN]
% optimization of x50 and slope of a sigmoid of ranging from 0 to 1
%
%
% Additional information in the second output, STAT
% [param,stat]=sigm_fit(x,y,fixed_params,initial_params,plot_flag)
%
%
% Example:
% %% generate data vectors (x and y)
% fsigm = @(param,xval) param(1)+(param(2)-param(1))./(1+exp((param(3)-xval)*param(4)))
% param=[0 1 5 1]; % "min", "max", "x50", "slope"
% x=0:0.1:10;
% y=fsigm(param,x) + 0.1*randn(size(x));
%
% %% standard parameter estimation
% [estimated_params]=sigm_fit(x,y)
%
% %% parameter estimation with forced 0.5 fixed min
% [estimated_params]=sigm_fit(x,y,[0.5 NaN NaN NaN])
%
% %% parameter estimation without plotting
% [estimated_params]=sigm_fit(x,y,[],[],0)
%
%
% Doubts, bugs: rpavao@gmail.com
% Downloaded from http://www.mathworks.com/matlabcentral/fileexchange/42641-sigmoid-logistic-curve-fit
% warning off
x=x(:);
y=y(:);
if nargin<=1 %fail
fprintf('');
help sigm_fit
return
end
automatic_initial_params=[quantile(y,0.05) quantile(y,0.95) NaN 1];
if sum(y==quantile(y,0.5))==0
temp=x(y==quantile(y(2:end),0.5));
else
temp=x(y==quantile(y,0.5));
end
automatic_initial_params(3)=temp(1);
if nargin==2 %simplest valid input
fixed_params=NaN(1,4);
initial_params=automatic_initial_params;
plot_flag=1;
end
if nargin==3
initial_params=automatic_initial_params;
plot_flag=1;
end
if nargin==4
plot_flag=1;
end
if exist('fixed_params','var')
if isempty(fixed_params)
fixed_params=NaN(1,4);
end
end
if exist('initial_params','var')
if isempty(initial_params)
initial_params=automatic_initial_params;
end
end
if exist('plot_flag','var')
if isempty(plot_flag)
plot_flag=1;
end
end
%p(1)=min; p(2)=max-min; p(3)=x50; p(4)=slope
%f = @(p,x) p(1) + (p(2)-p(1)) ./ (1 + exp((p(3)-x)*p(4))); % Modified function form with base e
f_str='f = @(param,xval)';
free_param_count=0;
bool_vec=NaN(1,4);
for i=1:4;
if isnan(fixed_params(i))
free_param_count=free_param_count+1;
f_str=[f_str ' param(' num2str(free_param_count) ')'];
bool_vec(i)=1;
else
f_str=[f_str ' ' num2str(fixed_params(i))];
bool_vec(i)=0;
end
if i==1; f_str=[f_str ' + (']; end
if i==2;
if isnan(fixed_params(1))
f_str=[f_str '-param(1) )./ ( 1 + exp( (']; % Modified here for base e
else
f_str=[f_str '-' num2str(fixed_params(1)) ')./ (1 + exp((']; % Modified here for base e
end
end
if i==3; f_str=[f_str ' - xval ) *']; end
if i==4; f_str=[f_str ' ) );']; end
end
eval(f_str);
%------added by me--------
% Set optimization options
options = statset('nlinfit');
options.MaxIter = 10000; % Set the maximum number of iterations here
%-------------------------
f
initial_params(bool_vec==1);
[BETA,RESID,J,COVB,MSE] = nlinfit(x,y,f,initial_params(bool_vec==1),options);
stat.param=BETA';
% confidence interval of the parameters
stat.paramCI = nlparci(BETA,RESID,'Jacobian',J);
% confidence interval of the estimation
[stat.ypred,delta] = nlpredci(f,x,BETA,RESID,'Covar',COVB);
stat.ypredlowerCI = stat.ypred - delta;
stat.ypredupperCI = stat.ypred + delta;
% plot(x,y,'ko') % observed data
% hold on
% plot(x,ypred,'k','LineWidth',2)
% plot(x,[lower,upper],'r--','LineWidth',1.5)
free_param_count=0;
for i=1:4;
if isnan(fixed_params(i))
free_param_count=free_param_count+1;
param(i)=BETA(free_param_count);
else
param(i)=fixed_params(i);
end
end
if plot_flag==1
x_vector=min(x):(max(x)-min(x))/100:max(x);
plot(x,y,'k.',x_vector,f(param(isnan(fixed_params)),x_vector),'r-')
xlim([min(x) max(x)])
end
end
The model is over-paramaterized -- eliminating trying to estimate the first seems to remove the numerical instability issues...
The output is somewhat confusing; he returns all four possible parameters in the resulting array but the printed model only references the estimated ones. You'll have to fix up the function to use the results but looks like at least reasonable fit given the noise in the data.
  1 个评论
John D'Errico
John D'Errico 2024-8-8
编辑:John D'Errico 2024-8-8
I would disagree about the statement of being over-parameterized.
A good example of a classically over-parameterized model is something like
y = (a+b)*x
where you cannot distinguish between the parameters a and b. All you can ever estimate there is the sum of a and b. And that is not the case here.
In the case of the OP model, param(1) and param(2) have very well defined meanings. And if the data were sufficient, they could indeed be estimated. The fundamental problem is that you cannot estimate the asymptote as x--> +inf. There simply is no information content in the data out there to show when the curve will start to turn around and flatten out on the right end.
This is why the model ends up with a singularity. It also explains why, if you removed one of the parameters, then the other parameter becomes sufficient to converge.

请先登录,再进行评论。


John D'Errico
John D'Errico 2024-8-8
编辑:John D'Errico 2024-8-8
A not uncommon misunderstanding about modeling.
First, look carefully at your data.
M = importdata('datai2j3.txt');
x = M(:,1);
y = M(:,2);
plot(x,y,'o')
AND think about the model you have posed. This is your model:
f = function_handle with value:
@(param,xval)param(1)+(param(2)-param(1))./(1+exp((param(3)-xval)*param(4)))
What happens in that model at plus infinity? That is, what is the limit assuming param(4) is negative? We don't really care what the value is, as long as param(4) is negative. The exponential will itself go to infinity. That means the model reduces to
y(+inf) = param(1)
when x gets large.
Similarly, we can look at what happens for large negative x. Now the exponential goes to zero, and the model reduces to
y(-inf) = param(1) +(param(2) - param(1))/(1 + 0)
so we will have
y(-inf) = param(2)
at minus infinity.
Ok, NOW GO BACK AND LOOK AT YOUR DATA!!!!!!!!!! THINK ABOUT IT!!!!!!!
The left asymptote is well defined. At minus infinity, I'd guess the left asymptote will be somewhere around 0.3, plus or minus some random crap.
What happens at x = PLUS infinity? Is there ANY information about the right asymptote? NOOOOOOOOOO!!!!!!!
There is no information content in your data to estimate the parameters in your model.
Now, finally, go back and read the message you got.
Warning: The Jacobian at the solution is ill-conditioned, and some model parameters may not be estimated well (they are not identifiable). Use caution in making predictions.
There is no meaningful value that can be assigned to param(1), BASED ON THIS DATA. There simply is insufficient information.
If the data were different, where the curve is starting to roll over on the right end, then you would find the model estimation process becomes well-posed. The Jacobian will no longer be singular, and you would have been happy.
  12 个评论
dpb
dpb 2024-8-14
M = importdata('datai2j3.txt');
x = M(:,1);
y = M(:,2);
plot(x,y,'o')
hold on
findchangepts(y,'statistic','mean')
findchangepts(y,'statistic','linear')
findchangepts(y,'statistic','rms')
ix=[findchangepts(y,'statistic','mean');findchangepts(y,'statistic','linear');findchangepts(y,'statistic','rms')];
S={'mean','linear','rms'};for i=1:3,fprintf('Bkpt %s: %d\n',S{i},ix(i)),end
Bkpt mean: 17 Bkpt linear: 16 Bkpt rms: 17
Is one prepackaged way to pick a point...there are probably better, but...

请先登录,再进行评论。

类别

Help CenterFile Exchange 中查找有关 Linear and Nonlinear Regression 的更多信息

产品


版本

R2022b

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by