‘Is there a way to speed up the calculation compared to just applying ifft and generating the entire signal x?’
No. The Fourier transform calculates a series of complex exponentials (with a real cosine and imaginary sine term) to your entire signal, both when transforming to the frequency domain and when inverse-transforming to the time domain. Because of this, you cannot transform only one value. You have to transform the entire signal.
