In general, to create a machine learning model, you would:
1. Collect data.
2. Split the data into training, test and validation sets.
3. Train a machine learning model using both the training and test sets.
4. Validate that your trained model on the validation set to verify that it can still reliably predict "unseen" data.
5. Use the model to predict real world data.
From the workflow above, you can see that we can only assess the accuracy of the model (before really using it in real world) by evaluating the prediction it outputs on the validation set.
If the predicted values on the validation set is within some reasonable accuracy that you desire, then, you can use the model to predict real world data with the assumption that it would also predict these new data with the same level of accuracy.
Yet, the validation set itself had to first be manually collected and labeled.
Furthermore, it is counter-intuitive (if not impossible) to be able to *automatically *assess the accuracy of your model on new unseen (and unlabeled) data. If you have another model that can assess whether your existing model is predicting new data correctly vs. incorrectly, you would certainly have used that model instead.
