Sure. You can do a modified median filter. You take the median filter of the data. Then you identify outliers, like where the actual value is more than some value away from the median. For those that are outliers, you replace only those with the median value. I think that's what my salt and pepper noise removal demo (attached) does.
You definitely don't want interpolation or regression like you listed in the tags.
Alternatively, if you know for a fact that it is only the edges that have outliers and want to replace the whole row or column, you can just replace the edges with the next innermost column or row
data(end,:) = data(end-1,:); % Replace last row.
and so on for the other 3 edges.
