Hi Kim,
Looks like you are trying to implement a Hough transform algorithm from the scratch. MATLAB already has the function “imfindcircles” which detect the circle in the image given a range of radius. Also this function uses the same Hough transform algorithm. But before using the function you need to do some pre-processing, I am following up your code to find the edge, now use this edged image to fill up the holes and now you can use “imfindcircle” function. This might help you.
I am also attaching a snippet of the code for the reference
image = imread('image.png');
image = edge(imgaussfilt(rgb2gray(image), 2), 'Canny');
img = imfill(image,'holes');
imshow(img);