You're not using the proper Matlab syntax for defining function, i.e. you don't need curly braces to define function. Just remove them and it should be fine.
function [jaccard_Idx,jaccardDist]=jacarrd_coefficient(imgorig,imgseg)
%check for logical image
if ~islogical(imgorig)
error('Image must be logical');
end
if ~islogical(imgseg)
error('Image must be logical');
end
%find the intersection of both the image
inter_image=img_orig & img_seg;
%find the union of both the image
union_image=img_orig./img_seg;
jaccard_Idx=sum(inter_image(:))./sum(union_image(:));
jaccardDist=1-jaccard_Idx;
