You are putting way too many superfluous parentheses in there, and this is causing the error. If you just stick to putting in the parentheses that are actually required, then the code becomes easier to read and the problem is easier to identify. So instead of (x(:)) you should just put x(:), etc., which makes it much easier to keep track of the matching parentheses.
Then you would discover that the function parentheses for sub2ind are closed just after size(gray), and do not include the x and y values at all, as they should.
To make your code clearer and reduce the chance of writing this kind of basic bug, consider allocating the indices first. Then you would have something like this:
idx = sub2ind(size(gray),y(:),x(:));
graydata = gray(idx);
which makes it easier to debug because you can also check the values in idx.
