You are taking a network trained to detect objects and you are trying to use it to detect a concept ("diseased") . It should not be surprising if the resulting network focuses on figuring out what object the input image most closely resembles.
Generally speaking, if you have high training accuracy but low testing accuracy, that often indicates that you overclassified . Not enough different inputs, not enough variations in the inputs.
There is an old story about how a branch of the US military achieved 100% success in training a network to distinguish US fighter aircraft from Russian fighter aircraft, but that when the network was deployed, it was a complete failure. It turned out that all of the training images for the US aircraft were pointing towards the right, and all of the training images of the Russian aircraft were pointing towards the left, and so what the network had effectively ended up training on was the orientation of the aircraft rather than any detail of the aircraft. It was overtrained on one kind of input image.
