I don't even think you need dbscan for this. You just need to define a length that separates "near enough" and "too far away". Then you just check every point in the array to see if it has 5 that are near enough, and keep those.
nearEnough = 0.02; % Whatever you want.
x = data(:,1);
y = data(:,2);
indexesToKeep = false(1, length(x)); % Initialize to not keeping any of them.
for k = 1 : length(x)
distances = sqrt((x(k) - x).^2 + (y(k) - y).^2);
if sum(distances > nearEnough) >= 5
% At least 5 are close enough to this k'th point, so keep this point.
indexesToKeep(k) = true;
end
end
x = x(indexesToKeep);
y = y(indexesToKeep);
