The problem is that you're not passing num to collatz, so how can it know how many times it has been called? Here is a version that makes num an optional argument, so you don't need to provide it in your call:
function [out, num] = collatz(val,varargin)
if nargin >1
num = varargin{1};
else
num = 0;
end
if val == 1
out = 1;
elseif mod(val, 2) == 0
[out,num] = collatz(val/2,num);
else
[out,num] = collatz(3*val+1,num);
end
num = num+1;
Now if you try
[out,num] = collatz(4)
you get
out =
1
num =
3
