Main Content

deep.gpu.deterministicAlgorithms

Set determinism of deep learning operations on the GPU to get reproducible results

Since R2024b

collapse all in page

    Syntax

    previousState = deep.gpu.deterministicAlgorithms(newState)
    state = deep.gpu.deterministicAlgorithms

    Description

    previousState = deep.gpu.deterministicAlgorithms(newState) returns the current determinism state of GPU deep learning operations as 1 (true) or 0 (false) before changing the state according to the input newState. If newState is 1 (true), then subsequent calls to GPU deep learning operations use only deterministic algorithms. This function requires Parallel Computing Toolbox™.

    example

    state = deep.gpu.deterministicAlgorithms returns the current determinism state of GPU deep learning operations as 1 (true) or 0 (false). If state is 1 (true), then subsequent calls to GPU deep learning operations use only deterministic algorithms.

    Tip

    • Use this function only if you require your GPU deep learning operations to be exactly reproducible because using only deterministic algorithms can slow down computations.

    • This function only controls the algorithms selected by the NVIDIA® cuDNN library. To enable reproducibility, you must also control other sources of randomness, for example, by setting the random number generator and seed. In most cases, setting the random number generator and seed on the CPU and GPU using the rng and gpurng (Parallel Computing Toolbox) functions, respectively, is sufficient. For more information, see Limitations and Tips.

    example

    Examples

    collapse all

    This example uses:

    Open Live Script

    To make GPU deep learning operations use only deterministic algorithms, set the determinism option to true. Store the previous state, so that you can restore it later.

    previousState = deep.gpu.deterministicAlgorithms(true);

    To make your GPU deep learning operations repeatable, also set the random number generators and seeds on the CPU and GPU using the rng and gpurng functions, respectively. 

    rng("default")
gpurng("default")

    After performing deep learning operations on your GPU, such as training or inference, revert the setting back to its original state.

    deep.gpu.deterministicAlgorithms(previousState);

    This example uses:

    Open Live Script

    This example shows how to train a network several times on a GPU and get identical results.

    Ensuring the reproducibility of model training and inference on the GPU can be beneficial for experimentation and debugging. Reproducing model training on the GPU is particularly important in the verification of deep learning systems.

    Prepare Training Data and Network

    Use the supporting functions prepareDigitsData and prepareAutoencoderLayers to prepare the training data and the network architecture. These functions prepare the data and build the autoencoder network as described in the Prepare Datastore for Image-to-Image Regression example, and are attached to this example as supporting files.

    [dsTrain,dsVal] = prepareDigitsData;
layers = prepareAutoencoderLayers;

    Define Training Options

    Specify the training options. The options are the same as those in the Prepare Datastore for Image-to-Image Regression example, with these exceptions.

    • Train for 5 epochs. Five epochs are not sufficient for the network to converge, but are sufficient to demonstrate whether or not training is exactly reproducible.

    • Return the network corresponding to the last training iteration. Doing so ensures a fair comparison when you compare the trained networks.

    • Train the network on a GPU. By default, the trainnet function uses a GPU if one is available. Training on a GPU requires a Parallel Computing Toolbox™ license and a supported GPU device. For information on supported devices, see GPU Computing Requirements (Parallel Computing Toolbox).

    • Disable all visualizations.

    options = trainingOptions("adam", ...
    MaxEpochs=5, ...
    MiniBatchSize=500, ...
    ValidationData=dsVal, ...
    ValidationPatience=5, ...
    OutputNetwork="last-iteration", ...
    ExecutionEnvironment="gpu", ...
    Verbose=false);

    Check whether a GPU is selected and is available for training.

    gpu = gpuDevice;
disp(gpu.Name + " selected.")
    NVIDIA RTX A5000 selected.

    Train Network Twice and Compare Results

    Train the network twice using the trainnet function. To ensure that random number generation does not affect the training, set the random number generator and seed on the CPU and the GPU before training using the rng and gpurng (Parallel Computing Toolbox) functions, respectively.

    rng("default")
gpurng("default")
net1 = trainnet(dsTrain,layers,"mse",options);

rng("default")
gpurng("default")
net2 = trainnet(dsTrain,layers,"mse",options);

    Check whether the learnable parameters of the trained networks are equal. As the training uses nondeterministic algorithms, the learnable parameters of the two networks are different.

    isequal(net1.Learnables.Value,net2.Learnables.Value)
    ans = logical
   0

    Plot the difference between the weights of the first convolution layer between the first training run and the second training run. The plot shows that there is a small difference in the weights of the two networks.

    learnablesDiff = net1.Learnables.Value{1}(:) - net2.Learnables.Value{1}(:);
learnablesDiff = extractdata(learnablesDiff);

figure
bar(learnablesDiff)
ylabel("Difference in Weight Value")
xlabel("Learnable Parameter Number")

    Set Determinism Option and Train Networks

    Use the deep.gpu.deterministicAlgorithms function to set the GPU determinism state to true, and capture the previous state of the GPU determinism so that you can restore it later. All subsequent calls to GPU deep learning operations use only deterministic algorithms. 

    previousState = deep.gpu.deterministicAlgorithms(true);

    Train the network twice using the trainnet function, setting the CPU and GPU random number generator and seed each time. Using only deterministic algorithms can slow down training and inference.

    rng("default")
gpurng("default")
net3 = trainnet(dsTrain,layers,"mse",options);

rng("default")
gpurng("default")
net4 = trainnet(dsTrain,layers,"mse",options);

    Check whether the learnable parameters of the trained networks are equal. As only deterministic algorithms are used, the learnable parameters of the two networks are equal.

    isequal(net3.Learnables.Value,net4.Learnables.Value)
    ans = logical
   1

    Plot the difference between the weights of the first convolution layer between the first training run and the second training run. The plot shows that there is no difference in the weights of the two networks.

    learnablesDiff = net3.Learnables.Value{1}(:) - net4.Learnables.Value{1}(:);
learnablesDiff = extractdata(learnablesDiff);

figure
bar(learnablesDiff)
ylabel("Difference in Weight Value")
xlabel("Learnable Parameter Number")

    Restore the GPU determinism state to its original value.

    deep.gpu.deterministicAlgorithms(previousState);

    Input Arguments

    collapse all

    New state of GPU determinism, specified as 1 (true) or 0 (false).

    If newState is 1 (true), then subsequent calls to GPU deep learning operations use only deterministic algorithms. If newState is 0 (false), then subsequent calls to GPU deep learning operations use the fastest available algorithms, which might be nondeterministic.

    Data Types: logical

    Limitations

    • This function only affects deep learning computations on the GPU in MATLAB®. It does not affect:

      • Deep learning operations on a CPU, for example, training a network using the trainnet function with the ExecutionEnvironment training option set to "cpu" or "parallel-cpu".

      • Deep learning code generated using GPU Coder™ or MATLAB Coder™.

      • Predictions using the predict and minibatchpredict functions when the Acceleration option is set to "mex".

      • Deep learning operations in Simulink®.

    • When using only deterministic algorithms, computations can be slower.

    • As the NVIDIA algorithm selection depends on several factors, including the hardware and the current GPU memory usage, your workflow might not give identical results on different GPUs.

    • Training a network is not reproducible, even if you use this function and set the random number generator and seed, if:

      • You use the trainnet function with the PreprocessingEnvironment training option set to "background" or "parallel".

      • You train a network using a minibatchqueue object with the PreprocessingEnvironment property set to "background" or "parallel".

    Tips

    • Sources of randomness in your deep learning workflow can include:

      • Learnable parameter and state value initialization — Initialization functions that sample from a distribution using random numbers generated on the CPU.

      • Data shuffling during training — If the Shuffle training option is set to "once" or "every-epoch", the trainnet function shuffles the training data using random numbers generated on the CPU.

      • Dropout layers — If you are training using a CPU, dropout layers generate random numbers on the CPU. If you are training using a GPU, dropout layers generate random numbers on the GPU.

      • Custom layers — The code inside your function layer dictates where the random numbers are generated. For example, rand(10) generates random numbers on the CPU while rand(10,"gpuArray") generates random numbers on the GPU.

    • If you are performing deep learning operations in parallel, for example, by using the trainnet function with the ExecutionEnvironment option set to "parallel-gpu", then set the random number generator and seed on each of the workers. For more information, see Control Random Number Streams on Workers (Parallel Computing Toolbox).

    • You can change the default algorithm and seed for the random number generator from the MATLAB Preferences window. To ensure that rng("default") uses the same algorithm and seed in different MATLAB sessions, ensure that the sessions have the same default algorithm and seed preferences. Alternatively, you can avoid using the preferences by specifying the seed and algorithm. For example, call rng("twister") to use the Mersenne Twister algorithm with a seed of 0.

    Version History

    Introduced in R2024b

    See Also

    | (Parallel Computing Toolbox) |

    Topics