To evaluate it with a matrix input, each of the variables has to refer to a column (or row) of that matrix.
You have defined ‘X’ as a column vector, so this syntax for your function works with it:
f=@(x) 2*x(1,:).^2 + x(2,:).^2 + 2*x(1,:).*x(2,:) + x(1,:) - x(2,:);
x1=0; x2=1;
X=[x1;x2];
Result_1 = f(X)
X2 = randi(9, 2, 5); % Create Input Data Matrix
Result_2 = f(X2)