Vgg16 requires a lot of GPU memory and you don't have very much of it. The error message says "Try reducing 'MiniBatchSize' using the trainingOptions function." Did you try that? Otherwise, I would recommend using a network that uses less memory, for example, GoogLeNet or SqueezeNet.
You can use any pretriained network available in MATLAB for transfer learning in this example: https://www.mathworks.com/help/deeplearning/examples/train-deep-learning-network-to-classify-new-images.html
For a list of pretrained networks, see https://www.mathworks.com/help/deeplearning/ug/pretrained-convolutional-neural-networks.html
