You can use the sign of the z-term of the cross product of the vectors defining the line and the points of interest:
vAB = [3 2 0]; % Vector from A to B, or anywhere along the line
vC = [5 2 0]; % From origin to C
vD = [2 4 0]; % From origin to D
X = cross(vAB, vC);
sign(X(3)) % Below the line when negative
Y = cross(vAB, vD);
sign(Y(3)) % Above the line when positive
