This should be a good entrypoint:
Each Axes has its own Interactions property, whose value is a list of interaction objects, so you can be pretty granular about what interactions are enabled.
One of these interactions is "RulerPanInteraction", which appears to be what you want to remove.