I understand that you are looking for an approach to improve your deep learning object detector for an LED Matrix after transfer learning with YOLO did not work well.
First, I'd suggest taking a closer look at your training images. Make sure you have lots of examples of the LED Matrix with different lighting, from different angles, and against various backgrounds. It's also super important to make sure your bounding box labels are drawn tightly and accurately around the object. Beyond the data, you can try tweaking the training process. You can create custom anchor boxes using the "estimateAnchorBoxes" function so they better match the shape of your matrix. Sometimes, simply adjusting the learning rate in "trainingOptions" or letting it train for more epochs can make a big difference.
If you're still not getting the results you want, it might be worth giving other models like SSD or Faster R-CNN a shot.
Please refer to the following documentation for more information on Object Detection Using Deep Learning: www.mathworks.com/help/vision/ug/object-detection-using-deep-learning.html
