imola - just define an empty array outside of your for loop, and add the point that is inside the rectangle to this array whenever it satisfies the condition. Something like
pointsInRect = [];
for k=1:11
AB=((XY(1,1)-XY(1,2))*(yy(:,k)-XY(2,2))-(XY(2,1)-XY(2,2))*(xx(:,k)-XY(1,2))>=0);
BC=((XY(1,2)-XY(1,3))*(yy(:,k)-XY(2,3))-(XY(2,2)-XY(2,3))*(xx(:,k)-XY(1,3))>=0);
CD=((XY(1,3)-XY(1,4))*(yy(:,k)-XY(2,4))-(XY(2,3)-XY(2,4))*(xx(:,k)-XY(1,4))>=0);
DA=((XY(1,4)-XY(1,1))*(yy(:,k)-XY(2,1))-(XY(2,4)-XY(2,1))*(xx(:,k)-XY(1,1))>=0);
if (AB||BC||CD||DA)==0
pointsInRect = [pointsInRect ; xx(1,k) yy(1,k)];
end
end
As for your conditions, you are saying that if all are false then the point must be in the rectangle. Is that correct?
