One obvious problem (might be a typo), when you assign theta you use atan2d, which returns an angle in degrees, and atan, which returns a value in radians, so the result is gibberish. It's probably best to avoid using inverse trig functions for what you're trying to do, since they're not continuous. It looks like you're using a Given's rotation matrix to change the direction of the ball. It may be more straightforward to use a Householder reflection matrix:
H = eye(2) - v*v';
Where v is the tangent vector of one of the octagon sides. So the direction of the ball would be modified like:
ballVector = -H * ballVector;
This avoids having to use any messy inverse trig functions.
(The code assumes you're vectors are stored in columns)
