Hi,
Based on your description, I understand that you are experiencing unusual behaviour in the validation error and training error. In a typical neural network training process, the training error consistently decreases, while the validation error initially decreases and then starts to increase, indicating overfitting. However, in your case, both the training error (blue line) and the validation error (red line) initially decrease and then both start to increase after some epochs. This suggests a few points to consider and correct in your training procedure:
- Shuffling the Validation Set: You mentioned shuffling the validation set before each epoch. This can lead to inconsistent evaluation metrics. The validation set should remain fixed throughout the training process to provide a consistent measure of the model's performance.
- Using Validation Set for Training: You mentioned processing the validation set through forward and backward steps to update the weights, which is not standard practice. The validation set should only be used to evaluate the model’s performance, not to train it. Use the validation set solely for evaluation. Do not perform backpropagation or weight updates using the validation set.
- Learning Rate and Hyperparameters: An inappropriate learning rate or other hyperparameters might cause unstable training. Experiment with different learning rates, batch sizes, and other hyperparameters.
By addressing these issues, you should be able to achieve more stable and predictable training behaviour, with the validation error and training error behaving as expected.
Hope this helps!
