What's next is to solve for the eigenvalues λ using the quadratic formula in terms of α. As with every quadratic equation, there are 3 possibilities:
I. two distinct real solutions, λ1 and λ2
II. one real solution λ
III. complex conjugate solutions, λ = a±ib
As to the solution for the system, these correspond to the following cases:
I. two independent real exponential functions, eλ1t and eλ2t
II. two functions of the form eλt and t*eλt
III. two independent trig-exponential functions, eatsin(bt) and eatcos(bt)
The phase portraits are qualitatively different for these three cases. The critical (or degenerate) case is case II. This happens when the discriminant of the quadratic formula (the term under the root sign) equals 0.
In your example, the characteristic equation is
Use the quadratic formula and get the eigenvalues
λ=1/2( α-4±√(α²+8α-24) ),
so the discriminant is α2+8α-24. Set this discriminant to zero and use the quadratic formula again to find the critical values of α:
α2+8α-24=0 ⇒ α=-4±2√10, about -10.3 and 2.32
If α lies between these two values, i.e.
-4-2√10 < α < -4+2√10,
the discriminant is negative, so you are in case III, complex-conjugate eigenvalues giving rise to trig-exponential functions.
Now what if α lies outside these two critical values, i.e.
α ≤ -4-2√10 or α ≥ -4+2√10 ?
Then you get two real eigenvalues (case I: real exponential functions) EXCEPT at the boundaries,
α = -4-2√10 and α = -4+2√10,
when there is only one eigenvalue, namely
λ=-4+√10 and λ=-4-√10
They claim there is a third critical value at α=5/2, corresponding to λ=0 and λ=-3/2. I don't see anything special about this value though.