The error was because you were trying to set the cutoff frequency at the Nyquist frequency (half the sample rate).
The error message is pretty clear.
Below is modified to format your code properly and adjust the cutoff frequency to a valid value. You can adjust to get the behavior you are looking for.
% Define the bandpass signal (sine wave)
fs = 1000; % sampling frequency
t = 0:1/fs:1-1/fs; % time vector
f1 = 100; % signal frequency
f2 = 300; % signal frequency
x = sin(2*pi*f1*t) + sin(2*pi*f2*t); % bandpass signal
% Obtain the Fourier transform of the bandpass signal
X = fft(x);
% Define the cutoff frequency for the lowpass filter
fc = 400; %500; % cutoff frequency <<< You can't set a cutoff at the Nyquist frequency (half the sample rate)
% Create a Butterworth lowpass filter
N = 4; % filter order
fc/(fs/2) % <<< valid (less than 1)
[b,a] = butter(N, fc/(fs/2));
% Apply the filter to the Fourier transform of the signal
Y = filter(b, a, X);
% Obtain the time-domain representation of the filtered signal
y = real(ifft(Y));
% Plot the original and filtered signals
figure;
subplot(2,1,1);
plot(t,x);
title('Bandpass Signal');
xlabel('Time (s)');
ylabel('Amplitude');
subplot(2,1,2);
plot(t,y);
title('Lowpass Filtered Signal');
xlabel('Time (s)');
ylabel('Amplitude');
