Because of the recursion limit (and other factors), it is smarter to implement this in a loop, instead of a recursive function.
The code below has some assumptions about your function you should check, but it shows the general idea.
x0=0.1;
N=30;
a=0.4;
x=zeros(N,1);
x(1)=x0;
for n=1:(N-1)
if 0<=x(n) && x(n)<a
x(n+1)= x(n)/a;
else
x(n+1)= (1-x(n))/(1-a);
end
end
