Some plants just cannot be controlled with a PID controller. In that case you need to look at other types of control. You can design arbitrary controllers using Control System Designer app , where you can uze root locus editor and bode plot editor to change controller gain, add poles and zeros.
You can also use Control System Tuner app that tunes specified controller structure. For example, you can start with a state-space controller of say order 4 and let the app tune it to get a stable response.
