Hi,
From my understanding you are getting the expected ifft result when the first element of the input is a real number and a different result when the first element is a complex number.
In ifft , the conjugate symmetry holds good from 2nd element to last element of the input. For a real signal, the first element of fft is always real.
So, while computing ifft, the same is expected. The required real signal can be obtained when the first element is a real value.
Hope it helps!