Overcoming VRAM limitations on Nvidia A100

Question

Christopher McCausland 2023-3-13

0
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/1928115-overcoming-vram-limitations-on-nvidia-a100

评论： Joss Knight 2023-3-14

I have access to a cluster with several Nvidia A100 40GB GPU's. I am training a deep learning network on these GPU's, however using trainNetwork() only makes use of around 10GB of the GPU's vRAM. I beleive this is a limitation of Nvidia Cuda, see here.

I have two related questions;

Other cluster users are writting in python with the 'DistributedDataParallel' module in PyTorch and are able to load in 40Gb of data (over the cuda limitation) onto the GPU's; is there a similar work around for MATLAB?
If this isn't the case is there any way to use Multi-instance GPU's, so essentially split the physical card into several smaller virtual GPU's and compute in parrellel?

Ideally I would like to speed up computation, so having a 3/4 of the vRAM empty which could otherwise be used for mini-batches is a little heart breaking.

0 个评论
显示 -2更早的评论隐藏 -2更早的评论

请先登录，再进行评论。

请先登录，再回答此问题。

Answer 1

Joss Knight 2023-3-14

2
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/1928115-overcoming-vram-limitations-on-nvidia-a100#answer_1192575

Just increase the MiniBatchSize and it'll use more memory.

6 个评论
显示 4更早的评论隐藏 4更早的评论

Christopher McCausland 2023-3-14

在 MATLAB Online 中打开

Hi Joss,

Thank you for the answer. This was my inital port of call.

My initial MiniBatchSize was 30 which was about 10GB of vRAM, as this is a 40GB card I then doubled MiniBatchSize to 60 expecting around 20GB of vRAM usage. At that stage I get the dreaded;

Maximum variable size allowed on the device is exceeded.

However, when I monitor vRAM usage in real time I seem to 'max' out at 12GB. (Right hand graph, Red plot. A second idle A100 can be seen in the backgound as blue)

I can see GPU utilisation is around 90% for this GPU which is good (Left hand graph, green plot), but also that there are regular dips which I assume is a pause to load in more data. (There could also be spikes of vRAM usage which I dont't see due to a slow polling rate.)

I am using the inbuilt trainNetwork with a datastore, I can share this code if needed however it is all very vanilla.

net = trainNetwork(dstrain,layers,options);

@Joss Knight, am I correct in saying that the expected behaviour is that increasing MiniBatchSize should allow me to use most of the vRAM on the card, say ~36GB. (With some vRAM reserved for Nvidia tasks etc.)?

Christopher

Christopher McCausland 2023-3-14

Hi Joss,

That makes much more sense, thank you for the explination too.

I will be able to eek out the final 10% of GPU utilsation by finding the exact minibatchsize that cases the fail. Regardless, as you mentioned, down-sampling the data should allow for larger minibatchsize size too.

I will wait for an answer to https://uk.mathworks.com/matlabcentral/answers/1926685-deep-learning-with-partitionable-datastores-on-a-cluster?s_tid=srchtitle as i'll need partition to be true before I can use DispatchInBackground. Ideally, I would like to distrabute this over multiple GPU workers so hopefully I can get partition working.

In the mean time I will mark this question as answered and will @ you in the next one if I can get partition behaving.

Thank you!

Christopher

Joss Knight 2023-3-14

You may never get that 10% so don't get your hopes up! Also, the best utilization is not necessarily at the highest batch size.

Why not ask a new question where you show your code for your datastore and one of us can help you make it partitionable.

请先登录，再进行评论。

Overcoming VRAM limitations on Nvidia A100

0 个评论
显示 -2更早的评论隐藏 -2更早的评论

采纳的回答

6 个评论
显示 4更早的评论隐藏 4更早的评论

更多回答（0 个）

另请参阅

类别

标签

产品

版本

Community Treasure Hunt

Overcoming VRAM limitations on Nvidia A100

0 个评论 显示 -2更早的评论隐藏 -2更早的评论

采纳的回答

6 个评论 显示 4更早的评论隐藏 4更早的评论

更多回答（0 个）

另请参阅

类别

标签

产品

版本

Community Treasure Hunt

0 个评论
显示 -2更早的评论隐藏 -2更早的评论

6 个评论
显示 4更早的评论隐藏 4更早的评论