A low passband with a very high sampling frequency is asking a lot of any filter. I am somewhat surprised that lowpass used a FIR design in that instance, when a IIR design is more appropriate. (I could not get a FIR filter to work with those constraints, either.)
This one works:
Tend = 10e-3;
dt = 100e-9;
t = 0:dt:Tend-dt;
fsig = 500;
sig = 100*sin(2*pi*fsig*t) + 20*sin(2*pi*fsig*100*t);
% [sig_filt filter] = lowpass(sig, 1000, 1/dt);
Fs = 1/dt;
Fn = Fs/2;
Wp = 950/Fn;
Ws = 1050/Fn;
Rp = 1;
Rs = 60;
[n,Wn] = ellipord(Wp,Ws,Rp,Rs);
[z,p,k] = ellip(n,Rp,Rs,Wp,'low');
[sos,g] = zp2sos(z,p,k);
sig_filt = filtfilt(sos,g,sig);
[h,f] = freqz(sos, 2^16, 1/dt);
Fs = 1/dt;
Fn = Fs/2;
L = numel(sig);
FTsig = fft(sig)/L;
Fv = linspace(0, 1, fix(L/2)+1)*Fn;
Iv = 1:numel(Fv);
figure
loglog(Fv, abs(FTsig(Iv))*2)
hold on
plot(f, abs(h), 'LineWidth',1.5)
hold off
xlim([0 5E+6])
grid
legend('Signal','Filter Characteristic', 'Location','S')
EDIT — (6 Aug 2020 at 13:35)
Consider forcing an IIR design:
[sig_filt filter] = lowpass(sig, 1000, 1/dt, 'ImpulseResponse','iir');
When I tried this, I got much better results than with the default FIR design.
.
