How to know the GPU memory needed when training a detector network like faster R-CNN?

Question

jingxue chen 2020-7-9

0
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/562304-how-to-know-the-gpu-memory-needed-when-training-a-detector-network-like-faster-r-cnn

评论： Mahesh Taparia 2020-7-17

I have a GPU which only have 6G total memory. When training a faster R-CNN detector, although I have set the input size to 224*224*3, the miniBatchSize can only be set as 2. If I set miniBatchSize as 4 or 8 or larger, there are errors that out of memory on device and the data no longer exists on the device. Now I want to buy a new gpu which support me to train 1920*1080 picture with miniBatchSize set to be 64 or 128. But I don't know how to compute the memory and other paraneters the GPU needed. So how can I decide which GPU to choose?

0 个评论
显示 -2更早的评论隐藏 -2更早的评论

请先登录，再进行评论。

请先登录，再回答此问题。

Answer 1

Mahesh Taparia 2020-7-14

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/562304-how-to-know-the-gpu-memory-needed-when-training-a-detector-network-like-faster-r-cnn#answer_465524

Hi

I think 6GB GPU is enough for your code. Check if the code is running on CPU/ GPU. To run the code on GPU, set ExecutionEnvironment to 'gpu' , if you are using trainingOptions and trainNetwork function for training. You can refer this document for that. For custom training loop, you need to convert the array to gpuArray. For more information, refer this documentation of gpuArray.

2 个评论
显示无隐藏无

jingxue chen 2020-7-15

编辑：jingxue chen 2020-7-15

在 MATLAB Online 中打开

inputSize = [224 224 3];
preprocessedTrainingData = transform(trainingData, @(data)preprocessData(data,inputSize));
numAnchors = 5;
[anchorBoxes, meanIoU]  = estimateAnchorBoxes(preprocessedTrainingData,numAnchors)
featureExtractionNetwork = resnet50;
featureLayer = 'activation_40_relu';
numClasses = width(vehicleDataseti)-1;
lgraph = fasterRCNNLayers(inputSize,numClasses,anchorBoxes,featureExtractionNetwork,featureLayer);
%Data Augmentation 
augmentedTrainingData = transform(trainingData,@augmentData);
augmentedData = cell(4,1);
for k = 1:4
    data = read(augmentedTrainingData);
    augmentedData{k} = insertObjectAnnotation(data{1},'rectangle',data{2},data{3});
    reset(augmentedTrainingData);
end
figure
montage(augmentedData,'BorderSize',10)
%Preprocess Training Data 
trainingData = transform(augmentedTrainingData,@(data)preprocessData(data,inputSize));
validationData = transform(validationData,@(data)preprocessData(data,inputSize));
data = read(trainingData);
I = data{1};
bbox = data{2};
annotatedImage = insertObjectAnnotation(I,'rectangle',bbox,label);
annotatedImage = imresize(annotatedImage,2);
figure
imshow(annotatedImage)
%Train Faster R-CNN
options = trainingOptions('sgdm',...
    'MaxEpochs',4,...
    'MiniBatchSize',2,...
    'InitialLearnRate',1e-3,...
    'CheckpointPath',tempdir,...
    'ValidationData',validationData);
  % Train the Faster R-CNN detector.
    % * Adjust NegativeOverlapRange and PositiveOverlapRange to ensure
    %   that training samples tightly overlap with ground
    %   truth.                                                   
[detector, info] = trainFasterRCNNObjectDetector(trainingData,lgraph,options, ...
        'NegativeOverlapRange',[0 0.3], ...
        'PositiveOverlapRange',[0.6 1]);

Thanks a lot！

But when I am trying to train resnet50 + faster R-CNN on CPU use 1920*1080 images，set inputsize to 224*224 and miniBatchSize to 8，I found that sometimes memory used more than 24GB. （when training with such big miniBatchSize on GPU an error occurs as “The data no longer exists on the device.”）

When I set inputsize to 500*500 and miniBatchSize to 16，an error occurs that I am requesting 32*32*1024*13166（51.4GB）array. It' so big.

So I think maybe it's because there are some issue in the code（autually it's from the faster R-CNN example），or CPU and bad GPU needs more memory？（my GPU is GTX 980 Ti）

Mahesh Taparia 2020-7-17

在 MATLAB Online 中打开

Hi

If you are using 1920*1080 size (HD) image, try to reduce its size and then start the training. Big images require more memory. Also, set the 'Execution Environment' to 'gpu' in the training option in your code, i.e

options = trainingOptions('sgdm',...
    'MaxEpochs',4,...
    'MiniBatchSize',2,...
    'InitialLearnRate',1e-3,...
    'CheckpointPath',tempdir,...
    'ValidationData',validationData,...
    'ExecutionEnvironment','gpu',...
    );

and check if the problem get resolve.

请先登录，再进行评论。

How to know the GPU memory needed when training a detector network like faster R-CNN?

0 个评论
显示 -2更早的评论隐藏 -2更早的评论

采纳的回答

2 个评论
显示无隐藏无

更多回答（0 个）

另请参阅

类别

标签

产品

版本

Community Treasure Hunt

How to know the GPU memory needed when training a detector network like faster R-CNN?

0 个评论 显示 -2更早的评论隐藏 -2更早的评论

采纳的回答

2 个评论 显示 无隐藏 无

更多回答（0 个）

另请参阅

类别

标签

产品

版本

Community Treasure Hunt

0 个评论
显示 -2更早的评论隐藏 -2更早的评论

2 个评论
显示无隐藏无