Hi,
I don't know an exact way, so I just propose two ideas:
1) use "imdilate" with a high value for the parameter, this may smooth the edges but also enlarge the objects. Then, to get back the starting dimensions, use "imerode".
2) try to use 'ConvexImage' property of "regionprops" function:
I hope this could help you.