MATLAB Dropout layer during prediciton

17 次查看（过去 30 天）
rakbar
rakbar 2019-1-7
评论： Miles Brim 2022-5-11
The Documentation for a Dropout layer states that:
"At prediction time the output of a dropout layer is equal to its input."
I assume this means that during prediction, there is no dropout.
Is there a method in MATLAB to enable Dropout during prediction time?
  1 个评论
Michael Phillips
Michael Phillips 2021-3-12
Has there been any progress on this in recent Matlab updates? I would also like to make an uncertainty assessment of my network using the Monte Carlo dropout method proposed by Gal and Ghahramani in 2016.

请先登录，再进行评论。

请先登录，再回答此问题。

回答（5 个）

Michael Hesse
Michael Hesse 2020-11-18
Any updates for this topic? I'm also interested in using dropout at prediction time to make an estimate for the uncertainty of the network. Yarin Gal showed that this procedure is equivalent to bayesian approximate inference.
Michael Hesse
Michael Hesse 2020-11-18
classdef Dropout < nnet.internal.cnn.layer.FunctionalLayer ...
& nnet.internal.cnn.layer.CPUFusableLayer
% Dropout Implementation of the dropout layer
% Copyright 2015-2019 The MathWorks, Inc.
properties
% LearnableParameters Learnable parameters for the layer
% This layer has no learnable parameters.
LearnableParameters = nnet.internal.cnn.layer.learnable.PredictionLearnableParameter.empty();
% Name (char array) A name for the layer
Name
end
properties (Constant)
% DefaultName Default layer's name.
DefaultName = 'dropout'
end
properties
% Learnables Empty
Learnables
end
properties(SetAccess=protected, GetAccess=?nnet.internal.cnn.dlnetwork)
% LearnablesName Empty
LearnablesNames
end
properties (SetAccess = private)
% InputNames This layer has a single input
InputNames = {'in'}
% OutputNames This layer has a single output
OutputNames = {'out'}
% HasSizeDetermined Specifies if all size parameters are set
% For this layer, there are no size parameters to set.
HasSizeDetermined = true
% Fraction The proportion of neurons to drop
% A number between 0 and 1 which specifies the proportion of
% input elements that are dropped by the dropout layer.
Probability
end
methods
function this = Dropout(name, probability)
this.Name = name;
this.Probability = probability;
% Dropout layer doesn't need X or Z for the backward pass
this.NeedsXForBackward = false;
this.NeedsZForBackward = false;
end
function Z = predict(~, X)
Z = X;
end
function [Z, dropoutMask] = forward(this, X)
% Use "inverted dropout", where we use scaling at training time
% so that we don't have to scale at test time. The scaled
% dropout mask is returned as the variable "dropoutMask".
if ~isa(X, 'dlarray')
superfloatOfX = superiorfloat(X);
else
superfloatOfX = superiorfloat(extractdata(X));
end
dropoutScaleFactor = cast( 1 - this.Probability, superfloatOfX );
dropoutMask = ( rand(size(X), 'like', X) > this.Probability ) / dropoutScaleFactor;
Z = X.*dropoutMask;
end
function [dX,dW] = backward(~, ~, ~, dZ, mask)
dX = dZ.*mask;
dW = []; % No learnable parameters
end
function outputSize = forwardPropagateSize(~, inputSize)
outputSize = inputSize;
end
function this = inferSize(this, ~)
end
function tf = isValidInputSize(~, ~)
% isValidInputSize Check if the layer can accept an input of
% a certain size
tf = true;
end
function outputSeqLen = forwardPropagateSequenceLength(~, inputSeqLen, ~)
% forwardPropagateSequenceLength The sequence length of the
% output of the layer given an input sequence length
% Propagate arbitrary sequence length
outputSeqLen = inputSeqLen;
end
function this = initializeLearnableParameters(this, ~)
end
function this = prepareForTraining(this)
this.LearnableParameters = nnet.internal.cnn.layer.learnable.TrainingLearnableParameter.empty();
end
function this = prepareForPrediction(this)
this.LearnableParameters = nnet.internal.cnn.layer.learnable.PredictionLearnableParameter.empty();
end
function this = setupForHostPrediction(this)
end
function this = setupForGPUPrediction(this)
end
function this = setupForHostTraining(this)
end
function this = setupForGPUTraining(this)
end
end
methods(Access=protected)
function this = setFunctionalStrategy(this)
% No-op
end
end
methods (Hidden)
function layerArgs = getFusedArguments(~)
% getFusedArguments Returned the arguments needed to call the
% layer in a fused network.
layerArgs = { 'passthrough' };
end
function tf = isFusable(~, ~, ~)
% isFusable Indicates if the layer is fusable in a given network.
tf = true;
end
end
end
How can i copy the forward method into the predict method?
  2 个评论
Michael Phillips
Michael Phillips 2021-3-12
Hey Michael, I'm trying to solve the same problem - that is, I want to use dropout during testing to estimate my network uncertainty. Did you ever get a version of this code working? If so, would you be willing to share it with me? Thanks!

请先登录，再进行评论。

Vishal Bhutani
Vishal Bhutani 2019-1-10
Based on my understanding dropout layer is used to avoid over-fitting of the neural network. The term "dropout" refers to dropping out units (both hidden and visible) in a neural network. This type of functionality is required at time of training of network. At the time of testing whole network is considered i.e all weights are accountable. So during testing or prediction output of dropout layer is equal to its input.
Its better if you tell your usecase, it might help to understand issue in more detail.
Hope it helps.
  1 个评论
rakbar
rakbar 2019-1-10
Some recent studies and tests have shown that when the Dropout layer is also active during prediction times, the prediction interval (or confidence interval) of the target can also be estimated. See for Example:
"A Theoretically Grounded Application of Dropout in Recurrent Neural Networks" https://arxiv.org/abs/1512.05287
At prediction time, the idea is to perform a few Monte-Carlo like loops for each target to obtain a distriution predictions. Then reporting the mean and standard deviation as the final target prediction and its error. These can be done in Python/Keras/Tensorflow.

请先登录，再进行评论。

Greg Heath
Greg Heath 2019-1-11
help dropout
... It is important to note that when creating a network, dropout will only be used during training.
Hope this helps.
Thank you for formally accepting my answer
Greg
I
Don Mathis
Don Mathis 2019-1-17
You could write yourself a custom dropout layer that does dropout in both the forward() and predict() methods. For dropout rate p, it would set each activation to 0 with probability p and then multiply all activations by 1/(1-p).
I'm not sure, but you might be able to give it a writeable 'p' property so you could set it to 0 after training if you want.

类别

在 Help Center 和 File Exchange 中查找有关 Image Data Workflows 的更多信息

标签

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by