Hi @Diego,
I see that you are using Mini-Batch gradient descent to train the network and you want to avoid having these abnormal spikes in the accuracy graph.
The spikes can be attributed as an unavoidable consequence of Mini-Batch Gradient Descent in general. This happens as some minibatches might have some outliers in them which can reduce training accuracy temporarily. You can avoid the spikes by using the following techniques:
- Full Batch Gradient descent: Here, the batch size is equal to the entire training set, so the entire training data is used for gradient and accuracy computation.
- Using higher batch sizes: This is supposed to reduce the depth of the spikes but not completely eliminate them. This is however better than Full Batch Gradient descent as training time is much less compared to Full Batch Gradient Descent.
- Outlier removal: It is not recommended as model will more likely not be robust enough and it is quite tedious given the amount of data you are using.
For further information on how mini-batch gradient descent works you can use the following: A Gentle Introduction to Mini-Batch Gradient Descent and How to Configure Batch Size
