This is a limitation of the 'imregister' function. The local minimum 'imregister' finds during calculation often does not correspond to the global minimum for this type of image. Specifically, when the image is circularly symmetric (which may confuse the rotation estimation). Using more local features (such as with MSER regions) is likely to be a better approach.
However, you can try to convert the images to grayscale images (using rgb2gray) and then use 'imregister' function(as it expects grayscale arguments). Read the below documentation:
imregister (rgb2gray(moving), rgb2gray(fixed), 'affine', optimizer, metric);