Use Multiple GPUs in Local Machine

If you have access to a machine with multiple GPUs, you can train a network using the trainnet function by setting the ExecutionEnvironment training option to "multi-gpu" using the trainingOptions function.

The "multi-gpu" option allows you to use multiple GPUs in a local parallel pool. If there is no current parallel pool, trainnet automatically starts a local parallel pool using your default cluster profile settings. The pool has as many workers as the number of available GPUs.

For information on how to perform custom training using multiple GPUs in your local machine, see Run Custom Training Loops on a GPU and in Parallel.

Use Multiple GPUs in Cluster

For training with multiple GPUs in a remote cluster, set the ExecutionEnvironment training option to "parallel-auto" or "parallel-gpu" using the trainingOptions function.

If there is no current parallel pool, trainnet automatically starts a parallel pool using your default cluster profile settings. If the pool has access to GPUs, then only workers with a unique GPU perform training computation. If the pool does not have GPUs, then training takes place on all available CPU workers instead.

For information on how to perform custom training using multiple GPUs in a remote cluster, see Run Custom Training Loops on a GPU and in Parallel.

Optimize Mini-Batch Size and Learning Rate

Convolutional neural networks are typically trained iteratively using mini-batches of images. This is because the whole data set is usually too large to fit into GPU memory. For optimum performance, you can experiment with the mini-batch size by changing the MiniBatchSize option using the trainingOptions function.

The optimal mini-batch size depends on your exact network, data set, and GPU hardware. When training with multiple GPUs, each image batch is distributed between the GPUs. This effectively increases the total GPU memory available, allowing larger batch sizes. A recommended practice is to scale up the mini-batch size linearly with the number of GPUs, in order to keep the workload on each GPU constant. For example, if you are training on a single GPU using a mini-batch size of 64, and you want to scale up to training with four GPUs of the same type, you can increase the mini-batch size to 256 so that each GPU processes 64 observations per iteration.

Because increasing the mini-batch size improves the significance of each iteration, you can increase the learning rate. A good general guideline is to increase the learning rate proportionally to the increase in mini-batch size. Depending on your application, a larger mini-batch size and learning rate can speed up training without a decrease in accuracy, up to some limit.

You can use the Experiment Manager app to find optimal training options by sweeping through a range of hyperparameter values or by using Bayesian optimization. For more information on how to use Experiment Manager, see Create a Deep Learning Experiment for Classification.

Select Particular GPUs to Use for Training

If you do not want to use all of your GPUs, you can select the GPUs that you want to use for training and inference directly. Doing so can be useful to avoid training on a poor-performance GPU, for example, your display GPU.

If your GPUs are in your local machine, you can use the gpuDeviceTable (Parallel Computing Toolbox) and gpuDeviceCount (Parallel Computing Toolbox) functions to examine your GPU resources and determine the index of the GPUs you want to use.

For single GPU training with the "auto" or "gpu" options, by default, MATLAB uses the GPU device with index 1. You can use a different GPU by selecting the device before you start training. Use gpuDevice (Parallel Computing Toolbox) to select the desired GPU using its index:

gpuDevice(index)

For multiple GPU training with the "multi-gpu" option, by default, MATLAB uses all available GPUs in your local machine. If you want to exclude GPUs, you can start the parallel pool in advance and select the devices manually.

For example, suppose you have three GPUs but you only want to use the devices with indices 1 and 3. You can use the following code to start a parallel pool with two workers and select one GPU on each worker.

useGPUs = [1 3];
parpool("Processes",numel(useGPUs));
spmd 
    gpuDevice(useGPUs(spmdIndex)); 
end

trainnet automatically uses the current parallel pool when you set the ExecutionEnvironment option to "multi-gpu" (or "parallel-auto" or "parallel-gpu" for the same result).

Train Multiple Networks on Multiple GPUs

If you want to train multiple models in parallel with one GPU each, start a parallel pool with one worker per available GPU, and train each network on a different worker. Use parfor or parfeval to simultaneously execute a network on each worker. Use the trainingOptions function to set the ExecutionEnvironment name-value option to "gpu" on each worker.

For example, use code of the following form to train multiple networks in parallel on all available GPUs:

options = trainingOptions("sgdm",ExecutionEnvironment="gpu");

parfor i=1:gpuDeviceCount("available")
    trainnet(…,options); 
end

To run in the background without blocking your local MATLAB, use parfeval. For examples showing how to train multiple networks using parfor and parfeval, see

Advanced Support for Fast Multi-Node GPU Communication

Some multi-GPU features in MATLAB, including the trainnet function, are optimized for direct communication via fast interconnects for improved performance.

If you have appropriate hardware connections, then data transfer between multiple GPUs uses fast peer-to-peer communication, including NVLink, if available.

If you are using a Linux^® compute cluster with fast interconnects between machines such as Infiniband, or fast interconnects between GPUs on different machines, such as GPUDirect RDMA, you might be able to take advantage of fast multi-node support in MATLAB. Enable this support on all the workers in your pool by setting the environment variable PARALLEL_SERVER_FAST_MULTINODE_GPU_COMMUNICATION to 1. Set this environment variable in the Cluster Profile Manager.

This feature is part of the NVIDIA NCCL library for GPU communication. To configure it, you must set additional environment variables to define the network interface protocol, especially NCCL_SOCKET_IFNAME. For more information, see the NCCL documentation and in particular the section on NCCL Environment Variables.