Sigh. Without seeing your numbers, this is hard to tell. Just telling us the size of the matrix is irrelevant.
I will tell you that you are doing something that is foolish.
Do NOT do this:
cpts = (A'*A)\(A'*b);
That squares the condition number of the problem.
Whereas this does not:
cpts = A\b;
Whoever told you to use the form you are using was steering you to a very poor way of solving the problem.
In the end, what matters is what the rank and condition number of A is in terms of linear algebra. So what does this tell you?
cond(A)
If that is too slow, then just do
condest(A)
It won't be to far off. If the condition number is greater than roughly 1e8, then the form you were using will have a condition number of roughly 1e16, which will then potentially yield NaNs for a result.
So at the very least, report the condition number of A.
