The error being received is expected as the lengths of variable 't' and 'Y' do not match.
After the line:
>> X = X(1:N/2+1)
is executed, length of 'X' becomes half of length of variables 'x' or 't' and when ifft is performed on 'X', the result 'Y' has same length as 'X' which is half of 't' and thus the error.
There are various ways to avoid this error and as the question asks for accessing the time domain values without noise, one way to do this would be taking the ifft of the vector 'x' instead of 'X'.