GPU for ConvNN training out of memory

8 次查看（过去 30 天）
David Brenes
David Brenes 2017-6-13
评论： Andrea Picciau 2019-6-21
Hi.
I am working with applying one of the MATLAB neural network examples to a data set that I have. My operating system is MacOS Sierra 10.12.5. When I run the program on the CPU there are no errors. However, when I run it on the GPU I am getting an error:
"Out of memory on device. To view more detail about available memory on the GPU, use 'gpuDevice()'. If the problem persists, reset the GPU by calling 'gpuDevice(1)'."
CUDADevice with properties:
Name: 'GeForce GT 750M'
Index: 1
ComputeCapability: '3.0'
SupportsDouble: 1
DriverVersion: 8
ToolkitVersion: 8
MaxThreadsPerBlock: 1024
MaxShmemPerBlock: 49152
MaxThreadBlockSize: [1024 1024 64]
MaxGridSize: [2.1475e+09 65535 65535]
SIMDWidth: 32
TotalMemory: 2.1470e+09
AvailableMemory: 182575104
MultiprocessorCount: 2
ClockRateKHz: 925500
ComputeMode: 'Default'
GPUOverlapsTransfers: 1
KernelExecutionTimeout: 1
CanMapHostMemory: 1
DeviceSupported: 1
DeviceSelected: 1
The code is below. I was suggested to run smaller batch sizes, but I am already running 1 batch size and it does not work.
rootFolder = 'Desktop/Neural_Net/T_v1/';
categories = {'Good', 'Bad'};
Training_path = [rootFolder, 'Training/'];
rootFolder_Test = [rootFolder, 'Test/'];
TrainingSet = imageDatastore(fullfile(Training_path,
categories), 'LabelSource', 'foldernames');
TestSet = imageDatastore(fullfile(rootFolder_Test,
categories), 'LabelSource', 'foldernames');
tbl_train = countEachLabel(TrainingSet)
minSetCount = min(tbl_train{:,2}); % determine the smallest
amount of images in a category
TrainingSet = splitEachLabel(TrainingSet, minSetCount,
'randomize');
tbl_train_cut = countEachLabel(TrainingSet)
alex = alexnet;
layers = alex.Layers;
clear alex,
layers(23) = fullyConnectedLayer(2);
layers(25) = classificationLayer; % rms different sgdm
opts = trainingOptions('sgdm', 'InitialLearnRate', 0.001,
'MaxEpochs', 1, 'MiniBatchSize', 1, 'ExecutionEnvironment',
'gpu', 'OutputFcn', @plotTrainingAccuracy);
TrainingSet.ReadFcn = @readAndPreprocessImage;
myNet = trainNetwork(TrainingSet, layers, opts);
disp('Past myNet')
TestSet.ReadFcn = @readAndPreprocessImage;
predictedLabels = classify(myNet, TestSet); % PRobability
normalized. Observer the softmax layer configure to output
sofmax
accuracy = mean(predictedLabels == TestSet.Labels)
% High specificity
function plotTrainingAccuracy(info)
persistent plotObj
if info.State == "start"
plotObj = animatedline;
xlabel("Iteration")
ylabel("Training Accuracy")
elseif info.State == "iteration"
addpoints(plotObj,info.Iteration,info.TrainingAccuracy)
drawnow limitrate nocallbacks
end
end
function Iout = readAndPreprocessImage(filename)
I = imread(filename);
% create an RGB image.
if ismatrix(I)
I = cat(3,I,I,I);
end
% Resize the image as required for the CNN.
Iout = imresize(I, [227 227]);
end
  3 个评论
显示 1更早的评论
Joss Knight
Joss Knight 2017-6-14
single is the default, so that shouldn't be it. What is the result of gpuDevice when you first start up MATLAB? According to the above, you only have 182 MB of free memory, which is barely enough to hold a single copy of AlexNet.
Ishtiaq Bercha
Ishtiaq Bercha 2018-5-1
Joss, I am using GTX 960m GPU and have a 3 layer CNN. I am getting the gpu out of memory error down to a mini batch size of 4. The GPU has a 2 GB RAM and 8 GB shared memory (although I am not sure how that benefits) on a 16GB ram alienware laptop. What are your recommendations. I am using medical images that are 512 x 512 x 3 in size. I am thinking about reducing the image size but are there any other ways I could get rid of the error? Thanks ---Ish Here is my code, btw.:
clc clear all % Train Network for Image Classification %Load the data as an ImageDatastore object.
digitDatasetPath = fullfile('C:','Users','ihbercha', ...
'Desktop','Machine Learning','Proj_F','Img_Data');
imds = imageDatastore(digitDatasetPath, ...
'IncludeSubfolders',true, ...
'LabelSource','foldernames');
% The training set has 150 % images and the testing set has the remaining images from each %label % (260 x 2 total images) .
numTrainingFiles =150;
[imdsTrain,imdsTest] = splitEachLabel(imds,numTrainingFiles,'randomize');
% Define the convolutional neural network architecture.
layers = [ ...
imageInputLayer([512 512 3])
convolution2dLayer(3,8,'Padding',1)
reluLayer
maxPooling2dLayer(3,'Stride',3)
convolution2dLayer(3,16,'Padding',1)
reluLayer
maxPooling2dLayer(3,'Stride',3)
convolution2dLayer(3,32,'Padding',1)
reluLayer
maxPooling2dLayer(3,'Stride',3)
fullyConnectedLayer(2)
softmaxLayer
classificationLayer];
Set the options to the default settings for the stochastic gradient descent with momentum. Set the maximum number of epochs at 20, and start the training with an initial learning rate of 0.01.
options = trainingOptions('sgdm', ...
'MaxEpochs',12,...
'InitialLearnRate',1e-2, ...
'Verbose',0, ...
'Plots','training-progress','MiniBatchSize',4, 'ExecutionEnvironment','gpu');
% Train the network.
net = trainNetwork(imdsTrain,layers,options);
% Run the trained network on the test set, which was not used to train the % network, and predict the image labels (digits).
YPred = classify(net,imdsTest);
YTest = imdsTest.Labels;
cf = confusionmat(YTest, YPred)
% Calculate the accuracy. Accuracy is the ratio of the number of true labels
% in the test data matching the classifications from |classify|, to the number
% of images in the test data.
accuracy = sum(YPred == YTest)/numel(YTest)

请先登录，再进行评论。

请先登录，再回答此问题。

回答（2 个）

Joss Knight
Joss Knight 2017-6-14
编辑：Joss Knight 2017-6-14
You're going to struggle massively with a small mobile chip like the 750M with only 2GB of memory and a large network like AlexNet, especially when that memory is being shared by the graphics. MATLAB R2017a's deep learning support does memory paging back to CPU to improve the capacity so if you are on 16b you may find upgrading helps (but it might not). Also, minimize the amount of memory being used by other processes by closing everything else and switching off all hardware graphics acceleration. This includes MATLAB which you should start up in software opengl mode ( matlab -softwareopengl ).
  2 个评论
David Brenes
David Brenes 2017-6-14
I do have MATLAB R2017a, is there anything that must be specified in the code if I want to have memory paging back to CPU?
Joss Knight
Joss Knight 2017-6-14
No, it just happens.
We have MacBook Pros with the 750M in our test system and we just can't reliably train AlexNet on them because memory is so short. The system takes so much of it and then AlexNet needs a good GB to train. Probably if you want to play with Deep Learning on this machine you're going to have to reduce the size of the network - or use the CPU.

请先登录，再进行评论。

ioannisbaptista
ioannisbaptista 2019-6-20
I have the same problem.
The message "Out of memory on device. To view more detail about available memory on the GPU, use 'gpuDevice()'. If the problem persists, reset the GPU by calling 'gpuDevice(1)'." appears when I try to evaluate my trained CNN. But in my case, I'm using a GeForce 1060M GTX 6GB RAM.
Here's a piece of my code:
testData = load('testROI.mat');
[test_imds, test_pxds] = pixelLabelTrainingData(testData.gTruth);
testDataSet = pixelLabelImageDatastore(test_imds, test_pxds);
unetPxdsTruth = testDataSet.PixelLabelData;
unetpxdsResults = semanticseg(test_imds, unet); % error is caused by this line
unetMetrics = evaluateSemanticSegmentation(unetpxdsResults, unetPxdsTruth);
The command gpuDevice() shows the results below:
CUDADevice with properties:
Name: 'GeForce GTX 1060'
Index: 1
ComputeCapability: '6.1'
SupportsDouble: 1
DriverVersion: 9.2000
ToolkitVersion: 9.1000
MaxThreadsPerBlock: 1024
MaxShmemPerBlock: 49152
MaxThreadBlockSize: [1024 1024 64]
MaxGridSize: [2.1475e+09 65535 65535]
SIMDWidth: 32
TotalMemory: 6.4425e+09
AvailableMemory: 5.0524e+09
MultiprocessorCount: 10
ClockRateKHz: 1670500
ComputeMode: 'Default'
GPUOverlapsTransfers: 1
KernelExecutionTimeout: 1
CanMapHostMemory: 1
DeviceSupported: 1
DeviceSelected: 1
As you can see, there are more than 5GB of free memoy but, for some reason I don't understand, the out of memory problem happens. The curious thing is it doesn't happen with 500 images the training stage, but happens with 100 images in the test evaluating stage. It's important to emphasize that this evaluation atempt uses a pretrained CNN, so the training data is not in the GPU memory while doing this.
Does anyone please knows what might be going on?

类别

在 Help Center 和 File Exchange 中查找有关 Image Data Workflows 的更多信息

标签

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by