The input in calc function is taking a matrix input which is creating this error. Instead of using x(1) and x(2), try using x(1,:) and x(2,:).
Since you are using an array and not a scalar value, you should use .^, .* and ./instead of ^,* and /.
Instead of sending an array input, I suggest you to input x, y separately
function fx = calc(x, y, const)
A = const(1);
B = const(2);
r = sqrt(X.^2 + Y.^2);
fx = A * cos(r).^2 .* exp(-B * r);
end
Then call the function:
Z = calc(x, y, [A, B]);
surf(X,Y,Z)
