Couldn't find your image, so I found 'rice.png' within MATLAB
I = im2double(imread('rice.png'));
Display the rice.png image, ising imagesc to autoscale and display the image.
figure
imagesc(I)
T = dctmtx(8);
dct = @(block_struct) T * block_struct.data * T'
D = blockproc(I,[8 8],dct);
In your code, you had mask as a long 1x64 vector, and I think you meant to have it as an 8x8 matrix. I manually put the semicolons after each 8 elements of the mask vector, to turn it into an 8x8 matrix. More elegantly, I could have used the command mask = reshape(mask,8,8) to reshape the 1x64 vector into an 8x8 matrix. Once I made this change, your code ran successfully, and computed new image I2, which I display here using imagesc instead of image, since imagesc autoscales the image before displaying it
mask = [1 1 1 1 0 0 0 0; 1 1 1 0 0 0 0 0; 1 1 0 0 0 0 0 0; 1 0 0 0 0 0 0 0;...
0 0 0 0 0 0 0 0; 0 0 0 0 0 0 0 0; 0 0 0 0 0 0 0 0; 0 0 0 0 0 0 0 0]
B2 = blockproc( D,[8 8],@(block_struct) mask .* block_struct.data);
invdct = @(block_struct) T' * block_struct.data * T
I2 = blockproc(B2,[8 8],invdct);
figure
imagesc(I2)