Really, you are going about this the wrong way.
Any given pixel will return to its original position if and only if the net transformation matrices applied to it are equivalent to the identity matrix.
First of all, remember the properties of exponentiation and mod: ((x mod P) * (x mod P) * (x mod P) ... * (x mod P)) with a total of N terms, is always equal to (x^N) mod P.
Then, if you examine what you are doing, you will see that net transformation applied to [x,y] after T steps is
and thus the condition that you need to satisfy is that (A^T) mod N is the identity matrix.
You do not need to transform any real matrices and compare them to the original: you just need to keep a running A^T matrix, multiply that by A, take the mod N, and see if you get the identity matrix out.
AI = eye(size(A));
AT = AI;
found = false;
numtries = 3*N;
for T = 1:numtries
AT = mod(AT * A,N);
if isequal(AT, AI); found = true; break;
end
if ~found
disp('did not find period')
else
fprintf('period was %d\n', T);
end
The periods can be tricky to predict by knowing "a" and "b", but perhaps I just did not look at it closely enough.
