I suggest you switch to using animatedline() to update your graphics.
At present, you are pulling out the old data for the line object, appending one new sample to it, and writing the data back. The first time you do that you are working with a vector of length 1, the second a vector of length 2, and so on. So the time involved to get to point N is 1+2+3+...+N which is N*(N+1)/2 and that adds up.
animatedline pre-initializes a buffer (usually of length 1000) and has a count of how many samples are in the buffer, and when the buffer gets full, it gets written as a whole into the list of complete buffers. In this way, the only need for expansion is in the writing of complete buffers; I do not recall at the moment how it keeps track of those efficiently.
