You should use more than 3 points to get better noise/error-resistance. Also, if your transformation is a rotation/translation, you need to constrain M to be of this form. This can be done with ABSOR ( Download ),
reg=absor(P,T)


Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!