I don't know why you would want to use the genetic algorithm for such a simple quadratic function. However, the general answer to your question is that you should be using vectorized commands to write these expressions. I.e., you would put your x_ij in a matrix X and write the function something like below. Notice that this works regardless of the size of X.
X=_____;
y=_____;
z=_____;
p_opt = ga(@(p) myfitness(p,X,y,z),______);
function out=myfitness(params, X,y,z)
n=size(X,2);
A=params(1:n);
B=params(n+1:end);
out= norm(X*A-y)^2 +1/6*norm(X*B-z)^2 ;
end
