Try median filtering only on points segmented by color..
In a blunt quick pass, I was able to fade away the blue and the red lines, considerably, using:
% Read the image and make a copy
A=imread(image.jpg');
B=A;
[m,n,~]=size(A);
% Blunt color segmentation (can improve by working on the color space)
M=(A(:,:,1)>A(:,:,2))&(A(:,:,2)<A(:,:,3));
[i,j]=find(M);
mask=find((i>w)&(i<m-w)&(j>w)&(j<n-w));
% Median filtering on the segmented pixels (can fine tune the right width)
w=15;
for k=1:numel(mask)
x=i(mask(k));
y=j(mask(k));
B(x,y,1)=median(A(x-w:x+w,y-w:y+w,1),'all');
B(x,y,2)=median(A(x-w:x+w,y-w:y+w,2),'all');
B(x,y,3)=median(A(x-w:x+w,y-w:y+w,3),'all');
end
% Result in B
With some fine tuning you might get it to work better. Could also think of a vector implementation to make it quicker.
The black line is harder to segment with color, so would have to use some directional filter (which could mess up the hair).
You can also start by smoothing out the halftone.
