How to implement rematerialization when using autodiff?
1 次查看(过去 30 天)
显示 更早的评论
I am using autodiff instead of finite differences and I am running into a memory problem. I read about rematerialization but I don't know how to or if it is possible to implement on matlab.
0 个评论
回答(1 个)
Zuber Khan
2024-5-10
Hi,
I am assuming that by rematerialization, you are referring to Gradient Checkpointing. Based on my understanding, there is no in-built functionality in Matlab so far to minimize memory consumption by discarding certain intermediate outputs and recalculating them during the backward pass.
You might need to write a custom code to save the important intermediate outputs (checkpoints) and recompute the rest during the backward pass with the help of these checkpoints.
Since you have not provided any details, it is difficult to write the exact code for identification and usage of these checkpoints. Further, in order to know how to use them for recomputation, it would require understanding of your objective function as well as the computational graph of derivatives.
I hope this will give you certain clarity.
Regards,
Zuber
0 个评论
另请参阅
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!