When calling the mod function with symbolic inputs, the second input must be a matrix (which can be a vector or a scalar) of numbers or symbolic number.
"Divisor (denominator), specified as a number, symbolic number, or a vector or matrix of numbers or symbolic numbers."
The symbolic variable b isn't a number or symbolic number.
As for your statement that mod(a+b, b) is equal to a, that's not quite true.
a = 5;
b = 3;
equalsA = mod(a+b, b) == a % returns false
equivalentA = mod(a+b, b) == mod(a, b) % returns true
If you want to use symbolic numbers, you will need to wrap the commands that define equalsA and equivalentA in isAlways calls to convert them into true or false.
a = sym(5);
b = sym(3);
equalsA = isAlways(mod(a+b, b) == a) % returns false
equivalentA = isAlways(mod(a+b, b) == mod(a, b)) % returns true
