No real world data truly follows a perfectly linear model. Even where there is an essentially linear domain, there are always limits, where at the ends perhaps, things start to deviate more from linearity. Very likely here this is what you are seeing.
The patterned residuals suggest to me that your data pushes the limits just a bit. Those larger residuals at each end are sufficient to bias the line just a bit. Or, perhaps, there is just more noise at the ends of the curve. This is also not uncommon behavior, where you have a non-uniform noise variance.
So, yes, your model is very likely approximately linear. Is it perfectly so? Of course not. If I wanted to ask the question "Is it NOT linear", I might next fit a quadratic model. Then look if the quadratic coefficient you obtain has bounds that contain zero. A simple question to ask, so TRY IT! If those confidence bounds allow the possibility that the quadratic coefficient is zero, then I would say that your data is not sufficient to show the model is NOT linear. What you cannot do is use what you have done so far to prove that your data does truly follow a linear model.
However, since we do not have your data, nor have you shown the results of that higher order fit, then all I can do is suggest that you do the test I have suggested.
