As the error suggests, you need to refactor your code so that you don't pass function handles across the extrinsic function call boundary. You can wrap up all of that code in yet another function, let's call it myCode.m. Then, declare that whole function as extrinsic and call it from your MATLAB Function block as shown below:
function [y, fval, exitflag] = fcn(lb, ub)
coder.extrinsic('mycode');
y = zeros(1,2);%preintialize this with expected dimensions
fval = zeros(1);%preintialize this with expected dimensions
exitflag = zeros(1);%preintialize this with expected dimensions
[y, fval, exitflag] = myCode(lb,ub);
end
function [y, fval, exitflag] = myCode(lb,ub) %define this in MATLAB workspace
fun = @(x)100*(x(2)-x(1)^2)^2 + (1-x(1))^2;
A = [];
b = [];
Aeq = [];
beq = [];
[y, fval, exitflag] = ga(fun,2,A,b,Aeq,beq,lb,ub);
end
Hope this helps!
