Hi Med,
When calculating pairwise distances for your dataset "Dataset_1.mat," you encounter memory constraints due to the following reasons:
- X = 60000 x 2
- Pairwise distance matrix size= 60000 x 60000 = 36 x 10^8 elements
NVIDIA libraries that MATLAB uses for GPU computation store array lengths as 32-bit integers, limiting arrays to a maximum of intmax('int32') elements. This imposes a hard limit on array size.
As a workaround, you can try applying the pairwise distance calculation to smaller chunks of your dataset. This method involves processing smaller portions of the data and storing the results in a final cell array. It achieves the same outcome as applying the “pdist2” function to the entire dataset but handles it in more manageable parts, which are then aggregated.
load('Dataset_1.mat')
X = single(Dataset(:,[2 4]));
X_CPU = X(1:60000,:);
X_GPU = gpuArray(X_CPU);
chunkSize = 10000; % Adjust chunkSize parameter based on available memory
numChunks = ceil(size(X_GPU, 1) / chunkSize);
% D_parts would store the final result of the pairwise distance of your dataset
D_parts = cell(numChunks, numChunks);
for i = 1:numChunks
idx1 = (i-1)*chunkSize + 1 : min(i*chunkSize, size(X_GPU, 1));
for j = 1:numChunks
idx2 = (j-1)*chunkSize + 1 : min(j*chunkSize, size(X_GPU, 1));
% Calculate distances for each chunk
D_chunk = pdist2(X_GPU(idx1, :), X_GPU(idx2, :), 'squaredeuclidean');
% Store the result in a cell array
D_parts{i, j} = gather(D_chunk); % gather to CPU if GPU memory is a constraint
end
end
You can refer to the resources below for more information:
