First one:
Undefined function or variable 'dejong2fcn'.
Second one: you have not properly passed the solver names in.
fminuncOptions = optimoptions(@fminunc,'Display','iter','Algorithm','quasi-newton');
options = optimoptions(@ga,'HybridFcn',{@fminunc, fminuncOptions});
options = optimoptions(options,'PlotFcns',{@gaplotbestf,@gaplotstopping});