It’s not clear to me what xvertices and vertices are, but I assume your 180 data points in A correspond to the sampled points (-89.5, -88.5, …, 89.5) and likewise for the (0.5, 1.5, …, 359.5).
If this is true, then you are not using your query points correctly. You are getting a 1x4 output from inpolygon because xq and yq are both 1x4 arrays, not 180x360. To fix this, you need to use set your xq and yq to be vectors like this:
% Set query points for Z(lat,lon)
xq = transpose(0.5:1:359.5);
yq = transpose(-89.5:1:89.5);
% Generate all combinations of [xq(k),yq(k)]:
[xq,yq] = meshgrid(xq,yq);
xq = xq(:);
yq = yq(:);
% Set vertices of enclosing polygon
xvertices = [110,160,160,110];
yvertices = [-10,-10,-40,-40];
% Get points in/on polygon:
[in, on] = inpolygon(xq,yq,xvertices,yvertices);
I think you also flipped your query points with your polygon edge points.
