You cannot compute the 100 billion pairwise distances, because the matrix won't fit in memeory. (Ok, symmetry reduces it to be only 50 billion or so distances.)
But do you seriously, really, desperately need to compute ALL such pairwise distances? You want to know the mean distance. Do you desperately need to know the EXACT value of that mean distance?
The point is, just use a Monte Carlo sampling to get a very good estimate of the mean pairwise distance.
For example, I'll do this using a smaller matrix, just so we can get some result out in a reasonable time.
X = randn(10000,160);
mean(pdist(X))
ans =
17.868
Now, compute a simple random sample. Then compute just a very small subset of the total distances you would need for an exact solution.
mean(vecnorm(X - X(randperm(size(X,1)),:),2,2))
ans =
17.864
mean(vecnorm(X - X(randperm(size(X,1)),:),2,2))
ans =
17.868
mean(vecnorm(X - X(randperm(size(X,1)),:),2,2))
ans =
17.856
mean(vecnorm(X - X(randperm(size(X,1)),:),2,2))
ans =
17.872
mean(vecnorm(X - X(randperm(size(X,1)),:),2,2))
ans =
17.874
So 5 simple random permutations gave 5 different estimates. All were correct to within about 0.1% error. For a matrix with 330000 rows, the error would be even smaller. And if even that much error bothers you, then just take the average or median of several such samples.
I would point out that even for an array of size 330000x160, the computation I describe takes about a half second on my computer, almost as much time as it took to generate the array in the first place.
X = randn(330000,160);
tic,mean(vecnorm(X - X(randperm(size(X,1)),:),2,2));toc
Elapsed time is 0.585973 seconds.
