When you do the syms call, you should break it down into symbols that are certain to be real valued and those that are not certain, and you should add the "real" qualifier when possible. For example,
syms a f h m x real
syms A %potentially complex valued
We know that A is potentially complex valued because they used 
instead of just 
. In turn that implies that your use of A^2 in the code is not correct: it might need to be abs(A)^2
instead of just . In turn that implies that your use of A^2 in the code is not correct: it might need to be abs(A)^2 You should also add any assume() calls that are reasonable. For example,
assume(x >= 0)
With all of your symbols being potentially complex valued (because you did not say otherwise), and since any of your values might be 0, then MATLAB is unable to determine the complex sign of the expression, so it has to just talk in general about limits, with it not able to prove anything about the properties of that limit.
