You can do color segmentation to get the red, black, brown, and blue pixels. You get a binary image that is essentially a map of where those color pixels are in the image. Then use find() on the binary image to get the x and y coordinates of every pixel of that color.
That said, I don't think you described your problem precisely enough so I don't think my answer is what you want, though it does what you asked.
