[result,FVAL,EXITFLAG]=fmincon(test1,X0,[],[],[],[],LB,UB)
means to call test1 with no arguments and to use the result (which would have to be a function handle or string) as the function to minimize over. Try
[result,FVAL,EXITFLAG]=fmincon(@test1,X0,[],[],[],[],LB,UB)
Your code does not use nested functions, it uses a script, and two functions defined with static workspaces. We cannot tell which file test1 is defined in: if it is not in test1.m then you would not be able to obtain a function handle to it from outside whatever file it is in. If test0 and test1 are both in test0.m then you would need to take extra steps to have test0 return the function handle to test1 . For example,
function y = test0(x)
if nargin == 0
y = @test1;
else
y = x.^2;
end
function y = test1(x)
y = x(1) + x(2) + test0(x(1));
end
and
X0=[0.1;0.1];
LB=[0;0.71];
UB=[2;6.53];
fun = test0(); %get handle to real function
[result, FVAL, EXITFLAG] = fmincon(fun, X0, [], [], [], [], LB, UB);
