This problem can be tackled by separating the points in 3 parts. Let's define some distances for clarity: dsp is the distance between the start and the current point, dep is the distance between the end point and the current point, and dse is the distance between start and end.
- type 1 is all points where sqrt(dse^2+dep^2)>=dsp
- type 2 is all points where sqrt(dse^2+dsp^2)>=dep
- type 3 is all other points
For type 1, the distance to the line segment is simply dep. For type 2, the distance to the line segment is simply dsp. (to see this, draw the line as one side of a right-sided triangle and apply Pythagoras)
Type 3 is between start and end, so there we need the orthogonal distance. We can slightly modify a staff-provided answer:
function d = point_to_line(pt, v1, v2)
% pt should be nx3
% v1 and v2 are vertices on the line (each 1x3)
% d is a nx1 vector with the orthogonal distances
v1 = repmat(v1,size(pt,1),1);
v2 = repmat(v2,size(pt,1),1);
a = v1 - v2;
b = pt - v2;
d = sqrt(sum(cross(a,b,2).^2,2)) ./ sqrt(sum(a.^2,2));
You can easily separate these three cases with logical indexing.
