Hello Naina,
I doubt what you are doing is beneficial in the sense of handling the data.
You haven't split your data into train/test set or train/validation/test set.
In order to estimate the performence of your algorithm on data that it wasn't trained on you need to split your data.
Since you have only 512 samples (small dataset in the sense of deep learning or almost any other machine learning algorithm) I would go on cross validation
After you train your model, try testing the predictions using the test data and see your result.
I have a feeling you are overfitting to your training set.
