You need a scientific decision for a filter, not an opinion from the forum. Is there any evidence that a median filter does not delete the most important information? Maybe a high frequency change in HR or RR is hint for a specific problem. Then the filtering destroys the measurement.
Analyse the signal you want to obtain and the noise. If you have scientific reasons to identify noise securely, a matching filter can be created and applied to the measured signals, the HR and RR. Afterwards you can calculate the score.