No way to tell what exactly caused the problem from the snapshot of the model. In the MATLAB Function block editor, you can click "Edit Data" and specify the data types of the inputs and outputs, so you don't have to use the "Data Type Conversion" block.
In the function code, use uint8() to typecast if needed. Try a simple example to make sure the values and data types are all correct. If the output value changes when only the data type of the input changes, then somewhere in the data processing there is an error. Show the code so people can point out the problem.
