By my understanding, you want to train a Semantic Segmentation on different set of images. One thing which you should do make all images of same size, uneven size will also work but all images should be of same size. After making all images to one size, you can do one thing is to make changes in the following command:
>> inputSize = [size1 size2 3];
>> imgLayer = imageInputLayer(inputSize)
where size1 and size2 specify your image size. Specify 3 for RGB images and 1 for grayscale images. Hope it helps.