If you read the documentation for dct2 carefully, it all implies that the image must be 2D rather than 3D. That is, you cannot use dct2 on an RGB or YCrCb image, except perhaps color plane by color plane.
Notice how the example uses rgb2gray() on the sample image, which should correspond to the luminosity information in the Y channel. You could do something like locate the dct2 Y, Cr, and Cb separately, but figure out based upon the Y channel which locations to zero out, and zero those locations out in all 3 dct2 results. Then idct2 each back and splice together the results to get the new YCrCb matrix.
