Why not try a direct computation with the data rather than using 'arrayfun'.? It just might be faster.
X = A(:,1);
Y = A(:,2);
X1 = B(:,1); % I assume that B(:,1) & B(:,2) are x and y coordinates of one end of lines
Y1 = B(:,2);
X2 = B(:,3); % and that B(:,3) & B(:,4) are x and y coordinates of the opposite ends
Y2 = B(:,4);
L = sqrt((X2-X1),^2+(Y2-Y1).^2);
P = (X2-X1)./L;
Q = (Y2-Y1)./L;
R = (X1.*Y2-Y1.*X2)./L;
D = zeros(size(A,1),1);
for k = 1:size(A,1)
D(k) = min(abs(P*Y(k)-Q*X(k)+R));
end
Also I assume that distance here is measured to the closest point on an extended line, not just to the line segment.