Partially labelled semantic segmentation

Question

byungchan 2020-10-1

0
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/602917-partially-labelled-semantic-segmentation

评论： byungchan 2020-10-10

Hi, I'm trying to use partially labelled images as a groud truth for the semantic segmentation training(there are lots of ambiguous regions in my images, so i am hoping to train only with the apparent regions; by inputting the class weight 0(techinically it was 10^-20 as I cannot input zero) to the unlabeled class in pixel classification layer)

And I found that the mini-batch accuracy is fluctuates around 35-40% and never converges.

Are there any ways i could fix this problem?

Is it related to excessive non-labeled regions?

I would appreciate any of your advice concerning this problem.

0 个评论
显示 -2更早的评论隐藏 -2更早的评论

请先登录，再进行评论。

请先登录，再回答此问题。

Answer 1

Raunak Gupta 2020-10-3

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/602917-partially-labelled-semantic-segmentation#answer_504562

Hi,

Assigning zero class weight to unlabeled pixel will help in removing them from loss calculation during training however when the accuracy metrics are calculated after training, it will include all the pixels and thus giving random categories to unlabeled pixel (I am assuming there is more than 1 class in labeled pixels).

Instead I will recommend assigning a background category to unlabeled pixel and keeping the class weight same as labeled pixel so that network can assign proper categories to all the pixels (background pixel can also be trained). This way network will be more robust for all parts of the image. So, adding one more class named background to the previous classes may help you converge to a higher accuracy.

7 个评论
显示 5更早的评论隐藏 5更早的评论

byungchan 2020-10-3

Thank you for your answers Gupta,

What I actually want to do is to classify the pixel labels for the ambiguous region(unlabelled region), using a network trained only with the apparent regions(labeled region).

However, if I assign a "background" label to the unlabelled regions, and train with those images, semantic segmentation results from the trained network would segment those ambiguous areas as a "background" rather than specific categories that I wish to verify(my goal is to see those ambiguous regions were classified as one of the categories that I have predefined).

So my question is this.

When calculating the mini-batch accuracy during the network training, does it accounts for all the regions in the image? (i.e. if the network assigns certain categories(my case "wet sand", "dry sand") to the non labeled region("NaN" label), would it affect the mini-batch accuracy during training?)
If that is the case, is there any possible way that I could see the mini-batch accuracy calculated only by using the labeled regions?

Thank you very much in advance!

byungchan 2020-10-9

Hello Gupta, I was working on this after I've got your tips, and I've got several more things to ask which I hope you don't mind.

1.I don't know the reason why the training loss have high loss function(always converges over 0.5 after only around 5 epochs, even if I tried numerous trials with different initial rate and different learning rates) when training with partially labeled images.

I can understand that the training accuracy could be small, as it accounts for all the pixels in the image, but I still don't know the reason for this high loss function.

Do you have any clues for this? Or do you think it's enough as it shows convergence?

Also, I don't know how the Matlab calculates its cross-entropy loss for images with unlabeled regions.

Would the sum of loss functions for unlabeled regions calculated as zero in the Matlab?

2. In regards to question 1, and your advice which you've given previously, I tried to make the unlabeled region in the original image as NaN(it automatically becomes 0 and the unlabeled regions become black) and pass it through the trained network, using 'ValidationData'(though it is validation, I used the same image as with the training to know the "true" training accuracy).

But the accuracy is even lower than 30% which I think still accounts for the "unlabeled region."(This is clear that if I use the function "evaluateSemanticSegmentation" manually, the class accuracy for each label is mostly over 80-90%.) Loss function, on the other hand, is around zero which is quite surprising that I thought it would be around 0.5. (Hence, I am asking above question #1.)

So is there anyway I could fix this problem? What I really want to do is to verify the "true" training accuracy and loss function (which only accounts the labeled regions) in "real-time" during the training, so that I can validate the training with some confidence.

Thank you very much in advance and have a wonderful day!