You can set the stop flag as part of the output function. It is described in the documentation here:
Edited to add an example:
I am stopping using x values but you can get function values from the optimValues structure.
function testerMain
x0 = [10;10;10]; % Starting guess at the solution
A = [-1 -2 -2; ... 1 2 2];
b = [0;72];
options=optimset('Algorithm','interior-point','Display','iter','OutputFcn',@outfun);
[x,fval] = fmincon(@myTestfun,x0,A,b,[],[],[],[],[],options);
disp(x)
end
function f = myTestfun(x)
f = -x(1) * x(2) * x(3);
end
function stop = outfun(x,optimValues,state)
stop = false;
if x(1) > 11
stop = true;
disp('Stopping, x > 11')
end
end