Your data doesn't seem to fit your plots. The code below should give you an idea of how you could do this. You could also define a cutoff distance and show everything below that distance.
X=XY(:,1);Y=XY(:,2);%XY is the array as you posted in your question
dist_to_origin=sqrt(X.^2+Y.^2);
[~,order]=sort(dist_to_origin);
plot(X(order(1:10)),Y(order(1:10)),'o')
axis([0 80 0 80])
%instead of order(1:10) you can also use this:
L=dist_to_origin<=45;
plot(X(L),Y(L),'o')
