For training, you can use either the CPU or the GPU. For certain Deep Learning problems which can a long time to train on a CPU, such Image Classification, training on the GPU is orders of magnitude faster. In general, if you have access to a reasonably powerful GPU use it for training. CPU can help when you need to scale up and run parallel jobs to utlize multiple cores on a machine that doesn't have access to multiple GPU's. An example, running an experiment to tune hyperparameters by training multiple networks at once in parallel using the Experiment Manager.
For inference (calling a trained model), CPU is generally sufficient. As an example for image classiciation you can get ~30 predictions / second on a CPU. However if you require very high frame rates or need to run a batch job, using a GPU will help you achieve faster results.
On overhead time of transferring data, here is a post which discusses it. It shows a few ways you can use various time functions for measuring execution time on a gpu. However I don't see data transfer time being a deciding factor between CPU / GPU.
My recommendation:
Training a single model
If you have access to a GPU, either locally or through a cloud platform, create a test to check if your training converges quicker or not. Good news is this is easy to do. Simply:
- Set the 'ExecutionEnvironment' setting to 'cpu' in your trainingoptions.
- Start training your model and monitor the training progress plot. Make note of how long each epoch takes to train.
- Stop the training.
- Set the 'ExecutionEnvironment' setting to 'gpu' in your trainingoptions.
- Start training your model and monitor the training progress plot. Make note of how long each epoch takes to train.
- Stop the training.
In most cases you'll observer the GPU is faster.
Hopefully this helps answer your question.
Regards,
