I don't fully understand the first paragraph and have never used this function, but looking at the help and what the function does it seems perfectly natural that M is empty.
You told it to match pairs, but told it the cost of not matching a row or column is 0, so the best cost it can achieve is 0 by not matching anything, hence M, the list of matched pairs is empty and uR and uC, the lists of unmatched rows and columns contains every row and every column. Unless there were a row and column containing identical values any matching would have a cost > 0.
When you gave an actual cost for not matching then in some cases actually finding a match was lower cost than that fixed value.
As for finding a value for costUnmatched, it depends entirely what you want. As with anything that uses thresholds it is often trial and error or based on knowledge of the actual inputs you have, which presumably are not just matrices of random numbers in your actual usage?
But if they were random numbers then you can use probabilities to determine a costUnmatched that would, on average match a certain percentage of pairs, it all depends what you want as a final result.
