Have you looked at the plot produced by showReprojectionErrors? The mean reprojection error should ideally be less than 0.5 of a pixel. If you see image pairs that have relatively high reprojection errors, try excluding those and recalibrating.
If most image pairs have high reprjection errors, take a look at your calibration images. See if they are out of focus, or if the checkerboard is at very oblique angles. Also, make sure that the calibration images give you points in all parts of the image. If you have any of these problems, then you need to take some more calibration images.
