Hi Manal,
I'm always happy to help a fellow animal researcher!
If you know where the center of the arena is in your heatmap this is not so difficult:
img = peaks(64);
xsize = size(img,2);
ysize = size(img,1);
cent = size(img)/2; % if the heatmap is not centered on the center of the arena you will have to adapt this to the real coordinate
radius = xsize/2; % you will have to adapt this for your arena, from your example I assume it fills the image
% generate a circular mask
[cc,rr] = meshgrid(1:xsize, 1:ysize); % x,y indices of all map pixels
circ_mask = sqrt(sum(([rr(:) cc(:)]-cent).^2,2)) <= radius; % cicular mask based on distance from cent
img_masked = img; % duplicate input image
img_masked(~circ_mask(:)) = NaN; % mask it (everything further than radius away from cent is NaN)
% plot the results
figure
subplot(1,2,1)
imagesc(img);
daspect([1 1 1])
subplot(1,2,2)
im = imagesc(img_masked);
set(im,'alphadata',~isnan(img_masked));
daspect([1 1 1])
If you don't know this then when you generate the ratemap using hist3 you should specify the bin edges such that the coordinate of the center of the arena falls in the center bin. If you don't know the coordinate of the arena center then you will have to work that out, otherwise there is no clear way to do what you want (although you could estimate the center as the centroid of the boundary of the animal's trajectory).
Hope this helps,
M.
