When you fft() N points, you get out N points. If the original N points were all real-valued, then the N output points of fft() will be in two halves that will be complex conjugates of each other (and reversed left-to-right). Except, that is, for the first output point, which does not get included in this conj(fliplr(X)). If you know the first output value, and you know the first half segment of the output, and you know the original data was real, then you can reconstruct the complex conjugate portion prior to taking the ifft() -- but you need that complex conjugate portion to be there in order for the ifft() to function properly.
If any of the original N points were complex-valued, then you need all N points of the fft() output, as the fft() in such a case is not conjugate-symmetric.
ifft() cannot simply be applied to "the first half" of the fft() output in order to regenerate the original data, as it would not be able to distinguish that case from the case of being asked to take the inverse fft of data that was originally complex.
