H(jω) does not exist for unstable systems, but we still use it when designing controllers - contradiction?

I don't have that book, but the statement you cite (emphasis mine):

... when s=jω the integral on the RHS of eq. 2.48 does not converge for unstable systems

doesn't mean the transfer function for an unstable system doesn't exist.

The book makes a lot of confusion between Transfer Function (TF) H(s), which is the Laplace transform of the impulse response h(t), and Fourier Transform (FT) of h(t).

Let's take a simple unstable system, i.e. one that has a transfer function with a pole $p = 2$ in the RHP:

\[ H(s) = \frac {1} {s-2} \]

This corresponds to an impulse response:

\[ h(t) = e^{2t}u(t) \]

The proof is simple:

\[ \begin{align} H(s) = \int_{-\infty}^{+\infty}h(t)e^{-st}dt = \int_{-\infty}^{+\infty}e^{2t}u(t)e^{-st}dt = \\[2em] = \int_{0}^{+\infty}e^{-(s-2)t}dt = - \frac {1}{s-2} \biggl[e^{-(s-2)t}\biggr]_{0}^{+\infty} = \frac {1}{s-2} \biggl[e^{-(s-2)t}\biggr]_{+\infty}^{0} = \\[2em] = \frac {1}{s-2} \biggl[e^{-(s-2)\cdot 0} - e^{-(s-2)\cdot (+\infty)}\biggr] = \frac {1}{s-2} \biggl[1 - e^{-(x + jy -2)\cdot (+\infty)}\biggr] = \\[2em] = \frac {1}{s-2} \biggl[1 - \underbrace{ e^{-(x-2)\cdot (+\infty)} } _{\substack{ \text{converges} \\[4pt] \text{to 0 iff} \\[6pt] x - 2 > 0 }} ~~\cdot \overbrace{ e^{-jy \cdot (+\infty)} } ^{\substack{ \text{complex number} \\ \text{with} \\ \text{bounded} \\ \text{magnitude} \\ }} \biggr] = \frac {1}{s-2} \biggl[1 - \underbrace{0}_{\text{iff x>2}} ~\biggr] = \\[2em] = \frac {1}{s-2} \quad \forall s=x+jy ~~\text{and}~~ x>2 \end{align} \]

In other words, the integral that defines H(s) converges for every point in the complex plane to the right of the pole. That is its region of convergence.

This also means that the imaginary axis lies outside that convergence region, hence the Fourier transform of h(t) doesn't exist, i.e. the defining integral of the Fourier transform does not converge.

So why does the Nyquist criterion for stability still work? Because that criterion doesn't depend on the Fourier transform. Simply substituting s with jω int H(s) doesn't mean that you get a Fourier transform and that because this latter doesn't exist you can't do that substitution.

In other words, H(jω) is the Fourier transform of h(t) only if the system is stable and H(s) region of convergence includes the imaginary axis. If this doesn't happen, H(jω) is still a valid complex function of a real variable.

In fact if you read Wikipedia about Nyquist stability criterion, you'll see that nowhere the Fourier transform is mentioned.

That's true even for other expressions which remain valid, like Y(s) and X(s) and the fact that H(s)=Y(s)/X(s) even for unstable systems.

The problem is that putting s=jω when you deal with an unstable system is misleading, since it could make you believe you are dealing with Fourier transforms, which is not the case.

The fact that "everything is OK" is a consequence that H(s) still exists and substituting jω inside the expressions before rather than after obtaining H(s) just doesn't change the end result when you apply Nyquist.

The real point here is that you can't interpret that ω as a real frequency existing in the system. You can't measure H(jω) for an unstable system.

Please note that the text refers to the integral, or the mathematical evaluation through integration which, indeed, cannot be obtained. But that doesn't mean you can't obtain the Laplace transfer function directly. A hypothetical RLC filter with a negative resistor is very much possible. In fact, it can be made in practice with emulated elements, and its transfer function will give a pole in the RHP:

$$H(s)=\dfrac{\dfrac{1}{LC}}{s^2-\dfrac{1}{RC}s+\dfrac{1}{LC}}$$

This can be obtaind by considering the raw Laplace equivalents of the elements. Is it unstable? Yes. Can it be done practically? Yes, here is the concept of it:

You can clearly see the phase going up. Does that violate the textbook? No -- the textbook only picks on the mathematical aspect (Olin says it better: semantics).

I think this is partly semantics.

For example, consider designing a compensator for a power supply. The transfer function under consideration is essentially the open loop impulse response of the system. If that goes nuts, then you have other problems to fix first. You are right in that it needs to be stable (not oscillate or grow exponentially or something) in response to a single blip.

But again, that's the open loop response. You can certainly make a mess and cause closed loop instability with the wrong feedback, but that's not the transfer function being quantified. In the end, of course, we do care about the closed loop transfer function. By that time, we've designed the compensator (feedback) to make sure the system is stable.