Your mistake is on this line:
A=rand(20,3);
You correctly wrote that "hel.faces is a m*3 matrix", but you seem to have missed that its values are all indices, i.e. positive integers, with values from 1 to numel(mask):
>> numel(mask)
ans =
121368
>> max(hel.faces(:))
ans =
121368
>> min(hel.faces(:))
ans =
1
>> find(mod(hel.faces(:),1)) % only integers
ans =
Empty matrix: 0-by-1
That is why it can be used as an index into another array (e.g. mask): because it it contains indices.
But you generated A with random numbers between 0 and 1, which are not indices (indices must be positive integers). Once you generate A with positive integers, with values from 1 to numel(B), then your code will work:
>> B = rand(20,1);
>> A = randi([1,20],20,3);
>> C = B>0.5;
>> C(A)
ans =
0 1 0
1 1 0
0 0 0
1 0 0
0 1 1
0 1 1
1 1 0
1 1 0
0 1 0
1 0 0
1 0 0
1 0 0
0 0 0
1 1 1
1 0 1
0 1 0
0 1 1
0 0 0
1 1 1
1 1 1
