The functions both do pretty much the same job.
'estimateGeometricTransform2D' takes in the input arguments matchedPoints1 and matchedPoints2 which can be specified as either a KAZEPoints object, cornerPoints object, SURFPoints object, MSERRegions object, ORBPoints object, BRISKPoints object, or an M-by-2 matrix in which each row is a pair of [x,y] coordinates and M is the number of matched points.
'fitgeotrans' takes input arguments movingPoints and fixedPoints, which can be only be specified as as an M-by-2 matrix of type 'double' or 'single'.
'fitgeotrans' supports 'nonreflectivesimilarity', 'similarity', 'affine', 'projective', 'polynomial', 'pwl', and 'lwm' as output tforms.
'estimateGeometricTransform2D' supports 'rigid', 'affine', 'similarity' and 'projective' as output tforms. It also provides additional outputs like statusCode and inlierIndex.
To answer your question, since 'projective' is supported in both, if you have KAZEPoints or other such objects, use estimateGeometricTransform2D. If you have [x,y] co-ordinates, you can use either.
