Create a colormap that is sufficiently smooth to your liking (e.g. with a few calls to linspace). Then you can set the Ticks property and the TickLabels property. The nice thing for you is that if you choose your scale sufficiently small, all negative values will be below the lowest index (and be colored with the lowest index), and all positive values will be above the highest index (which will be colored with the highest index). That would leave only 0 itself to be colored white.
You should supply a MWE, so we can help you tweak your code to solve this for realistic values. (Use the {}Code button to correctly format your code.)
