dec2bin does not generate a number, but a string. You're calculating the remainder of the ascii code of the characters '0' and '1' divided by the decimal 10 (and not binary 10 == decimal 2).
The only numeric representation that matlab understands is decimal. All other representations use strings. Numerical operations applied to strings use the ascii code of the characters.
The closest you could come to the result you want would be:
b = 0;
d = dec2bin(rem(b, bin2dec('10'))
