There are two main issues with the code you wrote. The first is that the isinteger function doesn't do what you think it does. It does not check that its input contains integer values but that its input is of an integer type.
isinteger(5)
isinteger(int8(5))
The second issue is that in the case where the input is invalid, you don't assign a value to the variable being returned as output. This will cause MATLAB to throw an error, but it won't be the one you expect or want. I recommend doing as Guillaume suggests and using error instead of disp.
But as Guillaume also pointed out, the factorial function included with MATLAB already does error for noninteger values. So if you wanted to throw a different error message, I would probably just use try and catch. Call factorial inside the try and if you reach the catch throw your own error. I'm not going to post the code since I suspect this is part of a homework assignment, but it's only going to be a couple lines long.
