Suppose that your task is to detect red objects. Then when you use RGB, the task usually gets designed in terms of whether the object is bright enough. But when you use HSV, the task usually gets designed in terms of whether the object is red enough.
Suppose that your task is to detect objects that stand out from the background, where the background is defined in terms of majority color. Then, when you use RGB, the task usually gets designed in terms of whether the object is bright enough relative to the background, or high contrast enough relative to the background. But when you use HSV, the task usually gets designed in terms of whether the object is a different enough color from the background.
You can think of situations in which the difference would be important. If the background is red and the object is brighter red, then can you detect it? Not if you are relying on hue alone. If the background is black and the object is dark red, then can you detect it? Not if you are relying on brightness alone.
