Solved this by following the examples in the "Dynamic Mask Dialog Box" help section.
More specifically, I needed to promote both the output mode (popup) and the output type/scaling to the mask. Then the visibility of the output type and scaling can be controlled in the callback function of the output mode as described in the help text.