Distortion is an optical aberration. it is a non-linear mapping of points that cannot be done with a linear process like convolution. The formulas for it are given in that link. You'd have to use something like nlfilter() or else program up a scanning filter on your own.
I don't know how least squares would be involved in your problem at all. Not sure what you were thinking but correcting, or creating, distortion is a straightforward mathematical operation. You would use a polynomial interpolation though in deciding where to pull the input pixel from, since it will likely lie in between integer rows and columns.
Your formula for A is really messed up. Please read the documentation on zeros for how to pass the proper input arguments.
