fmincon is not designed to deal with integer x values. You should try ga() which has an IntCon option or patternsearch() with a round() on the input values.
Of course for a small problem like this there are only 216 unique combinations of x so you could easily brute force it.
[rr,cc,pp] = ndgrid(0:5);
v = (50*rr.^2 + 100)./rr + (175*cc.^2 + 150)./cc + (160*pp.^2 + 250)./pp;
[val, idx] = min(v(:));
[rr(idx) cc(idx) pp(idx)]
