The ~ represents an output that is discarded. It is essentially equivalent to:
[dummy,idx]=min(dists);
clear dummy
For this example, the code wants to work with the index of the minimum value, not the value itself, so the minimum value that is returned is discarded and only the index is retained.
