size(X) = [1 6];
size(Y) = [3 3];
X and Y are not compatible for multiplcation given their sizes.
You can do something like this -
X = [a;b;c]; %size(X) = [3 2]
Z = mod(Y*X,2);
Z = [1 1; 0 1; 0 0];
Z = reshape(Z,1,[]);
Z = [1 1 0 1 0 0]; %If you want it to be a number and not an array, use polyval(z,10)
