Hi,
As per my understanding it is required to cluster the images into several groups using unsupervised learning. I would suggest using trainAutoencoder to learn a network which can generate meaningful features of the images, these features can be extracted from the last layer of encoder. The last layer of encoder is a lower dimensional representation of the image which can be used in kmeans for clustering the features in K groups. This way the images will be classified into several groups but be sure that the clustering is based on some high-level similarities between the images.
For more detailed information you may refer to the research papers available for Unsupervised Clustering.
