First, that's not YUV. I know everyone calls it YUV. If it's not analog PAL video, it's probably not YUV. That's the forward transform for YPbPr/YCbCr. If you think you still want YUV, there are conversion examples here. Your outputs are obviously being replaced by the valid results from rgb2ycbcr(), so I don't know what you're trying to do or why. Maybe you were just testing things?
YUV=cat(3,uint8(Y),uint8(U),uint8(V));
Either way, it appears you're trying to convert RGB to 8-bit integer YCbCr. You can't just multiply and be done. If you do that, you'll truncate half your chroma data and you won't have the expected margins.
After multiplication by the forward transform, Cb and Cr will (depending on the color content) span [-128 128] (note A(2,3) and A(3,1)). Converting that to uint8 will truncate all your color information. It needs to be offset.
Margins might be a variable thing, but I'm going to assume that they're expected by whatever will read the data. Certainly, rgb2ycbcr() and ycbcr2rgb() expect standard margins. The synopsis mentions this:
YCBCR is uint8 where Y is in the range [16 235], and Cb and Cr are in the range [16 240].
Here's an example of manual conversion. You're free to adapt it to your needs.
inpict = imread('peppers.png');
A = [0.299 0.587 0.114;-0.1687 -0.3313 0.5;0.5 -0.4187 -0.08131];
yccpict = uint8(reshape(reshape(im2double(inpict),[],3)*(A.*Asc).' + os.',s));
yccpict2 = rgb2ycbcr(inpict);
rgbpict = im2uint8(reshape((reshape(double(yccpict),[],3) - os.')/(A.*Asc).',s));
rgbpict2 = ycbcr2rgb(yccpict2);
The small difference is merely a matter of order of operations.
As a sidenote. I mentioned A(2,3) and A(3,1). Those two terms determine the half-width of the chroma plane in each direction. Since they're neatly half, the chroma data is a nice intmax-width and can be offset by 128 to fit back into [0 255]. Look at those terms in the YUV forward transformation matrix. If you tried to do the same with YUV, V would be [-157 157]. Offsetting won't fit that back into [0 255].
