Hi Bipin,
When a deep CNN network produces NaN outputs during training, it typically indicates that there is an issue with the numerical stability of the network. Some reasons for NaN outputs can be:
- Improper weight initializations
- High Learning Rate
- Batch normalizations.
To debug and solve the issue, you can try lowering the learning rate, gradient clipping and check and debug custom layers and their corresponding weight initializations and loss functions. Temporarily, you can use a smaller version of your network to see if the issue persists. This can help isolate whether the problem is due to the network size or specific layers.
For more help on network producing NaN outputs, refer the following links:
- https://www.mathworks.com/matlabcentral/answers/337587-how-to-avoid-nan-in-the-mini-batch-loss-from-traning-convolutional-neural-network
- https://www.mathworks.com/matlabcentral/answers/1917165-training-loss-is-nan-deep-learning
- https://www.mathworks.com/matlabcentral/answers/92319-how-are-nan-values-in-the-input-data-for-a-neural-network-taken-into-account-while-training-the-netw
Hope this helps.
