function f = obj_eps1(x,~)
That says that obj_eps1 expects to be passed two input arguments.
[gbest]=PSOalgo(n,maxite,lb,ub,dim,obj_eps1,nonlcon_eps1,EpsVal(i))
That line says that obj_eps1 should be invoked with no inputs, and whatever it returns should be passed in as the 6th input to
function [gbest]= PSOalgo(n,maxite,lb,ub,dim,obj_eps1,nonlcon_eps1,EpsVal)
and
[~,constraintViolation(i,1)] = nonlcon_eps1(pbest(i,:), EpsVal);
That says that whatever is passed as the 6th parameter to PSOalgo must be either an array with at least two indices, or else must be a function handle with at least two inputs.
Therefore, when you invoke obj_eps1 in the line
[gbest]=PSOalgo(n,maxite,lb,ub,dim,obj_eps1,nonlcon_eps1,EpsVal(i))
then you need obj_eps1 to return the handle to a function that accepts at least two inputs.
I would suggest to you that what you wanted was
[gbest] = PSOalgo(n, maxite, lb, ub, dim, @obj_eps1, @nonlcon_eps1, EpsVal(i))
