You are computing k once for each value of i and overwriting it. If I understand what you want, try:
i = 1:5;
j = 0:5;
k = zeros(numel(i),numel(j));
for a = 1:numel(i)
for b = 1:numel(j)
k(a,b) = i(a)^2 + i(a)*j(b) + j(b)^2;
end
end
display(k)
