You could try something like this:
Sounds like 3-D data to me. Or even two separate 2-D images. So try
[rows, columns, numColors] = size(yourBrightnessImage);
hsvImage = ones(rows, columns, 3);
hsvImage(:, :, 3) = 255 * mat2gray(yourBrightnessImage); % Assign intensity channel.
hsvImage(:, :, 1) = yourHueImage; % Assign hue channel.
rgbImage = hsv2rgb(hsvImage);
imshow(rgbImage);
If you need more help, attach your data.
