I would eliminate the abs call until the end, using it on the integrated function only after you evaluate it numerically.
Try this:
syms a f real
z(f) = exp(1j*2*pi*f);
H(f) = (2/z - 1/z^2)^2;
H = rewrite(H, 'sincos'); % Optional, If You Want a Trigonometric Result
IH = int(H, f, 0, a);
IH = simplify(IH, 'Steps', 20)
