As you mentioned, this looks like an overfitting problem. Terminating the training early is one way that you can avoid overfitting. There are also training options that you can specify to automate this.
For example, see the validation section on the following documentation page:
In particular, setting the option OutputNetwork as best-validation-loss will return the network at the iteration with the minimal validation loss. From your plot, this looks to be around 0.5 * 10^4 (although there is a spike around 1.5 * 10^4 that looks sporadically lower). You might also want to set the ValidationPatience. This specifies the number of times that the loss on the validation set can be larger than or equal to the previously smallest loss before network training stops. This should help terminate the training before the upward drift in the validation loss. You may also want to increasing the spacing between validation loss evaluation to remove the oscillations and help isolate the upward drift. This can be done by setting the ValidationFrequency value, which is the number of iterations between evaluations of validation metrics.
