As Image Analyst already told you in https://www.mathworks.com/matlabcentral/answers/231932-is-this-how-to-calculate-mean-square-error-for-two-images#comment_1320422
Look up "image quality metrics" in the help. You'll find functions such as brisque, niqe, piqe, ssim, psnr, immse, multissim, and multissim3. We don't know which would work best for you. Read about them and decide which you think would work best for you.
