yogya - have you verified that the conversion of the hex string x5 to decimal is correct? I think that your hex string has too many characters and so the conversion produces some number that is the maximum allowed. Try modifying x5 slightly and see if the decimal results differ. If you read the documentation for hex2dec, it states that the value of the input hexadecimal string must be smaller than hexadecimal 10,000,000,000,000. I couldn't see a equivalent restriction for base2dec, but there was one for dec2base.
As for bitxor, your code is failing because you are passing strings, r1 and r2, into this function rather than integers.