Euclidean Distance (huge number of vectors)
4 次查看(过去 30 天)
显示 更早的评论
I have a 70,000 x 300 matrix. These are basically 70,000 vectors of 300 elements each. I need the distance matrix (distances between each pair of vectors). It is too large to just use pdist. Does anybody have general programming tips on how to handle this problem?
0 个评论
回答(4 个)
Richard Brown
2012-4-30
You could do something like this and leave it running overnight :| Obviously you'd define your histogram edges to be relevant to the vectors you were using. The disp statement is purely there as a primitive progress bar
n = 70000;
x = rand(300,n);
edges = linspace(0, 10, 100);
edges2 = edges.^2;
h = zeros(1, numel(edges));
for i = 1:n-1;
d2 = sum(bsxfun(@minus, x(:, i+1:n), x(:, i)).^2);
h = h + histc(d2, edges2);
disp(100*i/n)
end
And then to display it,
bar(edges, h, 'histc')
EDIT Adjusted to reflect Walter Roberson's comment below to eliminate the use of sqrt
3 个评论
Image Analyst
2012-4-30
That's 70,000 * 69,999 / 2 = 2.4 billion distances. What are you going to do with all those? Can you figure out a way where you only need to calculate some specific distances for some limited number of specific circumstances/criteria rather than all of them? Do you really need all 2.4 billion of them?
2 个评论
Image Analyst
2012-4-30
If there is not pattern and they're more or less uniformly distributed, then you would probably get the same essential shape of the histogram after only a few thousand distances. You can do that for a few thousand of them, then a few million, and see if the shape of the histogram changes substantially. If the histogram shape equilibrates/stabilizes after a few thousand calculations, why do all 2.4 billion of them?
Sean de Wolski
2012-4-30
Break it into a bunch of smaller problems and solve each one (based on what you want as an end result).
Some questions to ask yourself: Do the vectors follow any pattern? Could you downsample or merge some of them? What are you doing with the results?
The more background you can give us the more we can help.
0 个评论
另请参阅
类别
在 Help Center 和 File Exchange 中查找有关 Classification Ensembles 的更多信息
产品
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!