This is not a MATLAB question - it's just analytical geometry formulas and algebra. Just draw it out and make sure you see the right angle between the center, the point off the circle, and the point on the circle. Then use the Pythagorean theorem.
[EDIT]
OK some more hints:
Let's say R is the radius, and (xr, yr) is on the circle, and (xc, yc) is the center of the circle, and (xp, yp) is the point away from and off the circle. So by the Pythagorean theorem
length1 = sqrt((xp-xr)^2 + (yp-yr)^2) % Length of side #1
length2 = sqrt((xp-xc)^2 + (yp-yc)^2) % Length of side #2
R^2 + length1^2 = length2^2 % The right triangle.
Plug in for length1 and length 2 :
R^2 + (xp-xr)^2 + (yp-yr)^2 = (xp-xc)^2 + (yp-yc)^2 % The right triangle.
Multiply out and cancel and combine terms and you'll be further along.
