According to _Signal processing and linear systems_ by Lathi, the transfer function \$H(j\omega)\$ does not exist for systems with poles in the RHP: -
This makes sense to me, since \$H(j\omega) = \frac{Y(j\omega)}{X(j\omega)} \$. However, since the system is unstable \$Y \$ is unbounded (and growing) and the fourier transform doesn't exist for such functions. So \$Y(j\omega)\$ doesn't exist and therefore, \$H(j\omega) \$ must be meaningless for unstable systems.
BUT! we use \$H(j\omega) \$ when designing controllers for unstable systems anyway - and it works. We look at the Bode Plot, we look at the Nyquist Plot both of which you need to know \$H(j\omega)\$ and design a controller based on what we see - and the controller actually works!
How can this be? How can there be this contradiction between systems and signals theory and control theory? It seems that concepts like **region of convergence** and **existence of fourier integral** are only dealt with on coursework and once that's done, you don't hear from them ever again.
