The points which are within a radius, r, from the origin can be obtained as follows:
index = sum(X.^2,2) < r^2;
Note, squaring r is more efficient than taking the square root for ALL of the points in X.
For a starting point other than the origin, e.g. p (1 by 3), points in X within a threshold radius r from the point p can be obtained as follows:
index = sum(X.^2 - p,2) < r^2;