I do not have the RF Toolbox, however creating a filter as an anonymous function with appropriate arguments (I just use the passband frequencies here) is certainly possible with these filters, Signal Processing Toolbox filters, and Control System Toolbox system objects. You can use the first approach, or you can combine the frequencies (and other parameters) in one vector (as illustrated in the second approach) and then refer to the elements of the vector in the appropriate places in the rffilter object.
It should then be possible to use the anonymous function version of your rffilter object as part of an objective function in your machine learning application.
Example —
robj = @(f1,f2) rffilter('ResponseType','Bandpass','Implementation','LC Tee','PassbandFrequency',[f1 f2], ...
'StopbandFrequency',[f1 f2].*[0.95 1.05],'PassbandAttenuation',3,'StopbandAttenuation',40);
F1 = 950E+06;
F2 = 2200E+6;
frequencies = linspace(0,2*F2,1001);
figure
rfplot(robj(F1,F2), frequencies)
robj = @(params) rffilter('ResponseType','Bandpass','Implementation','LC Tee','PassbandFrequency',[params(1) params(2)], ...
'StopbandFrequency',[params(1) params(2)].*[0.95 1.05],'PassbandAttenuation',3,'StopbandAttenuation',40);
F1 = 950E+06;
F2 = 2200E+6;
P = [F1 F2];
frequencies = linspace(0,2*P(2),1001);
figure
rfplot(robj(P), frequencies)
.
