The critical points can be found by solving x'=0 and y'=0. In this problem, we will have a system of equations,
(y-2)(2x+y-α) = 0 and (x-4)(x+2y-β) = 0.
The first equation gives us y=2 or 2x+y = α. The second equation gives us x=4 or x+2y = β. So we have four critical points in total. They are (4,2), (4,α-8), (β-4,2), and ((2α-β)/3,(-α+2β)/3).
The jacobian is given by
In this case, we have
Next, let us find the values of α and β that satisfy the conditions. The classification of the critical points are given by the eigenvalues of the jacobian.
At point (4,2), the jacobinian is
J(4,2) = ⌈0 8-α⌉
The eigenvalue of J(4,2) is given by solving (x-0)(x-0)-(8-α)(6-β)=0. We have x=±√(8-α)(6-β).
The eigenvalue of J(4,α-8) is given by solving x(x-2α-20)-(α-10)(2α-β-12)=0.
The eigenvalue of J(β-4,2) is given by solving x(x-2β+16)-(-α+2β-6)(β-8)=0.
The eigenvalue of J((2α-β)/3,(-α+2β)/3) is given by solving ...
However, we do not need the explicite expression of the eigenvalues. All we care about is that the we have one saddle points and one ellipse stable. Saddle points means the eigenvalues are real numbers with different signs and ellipse stable points means the eigenvalues are negative numbers.
So assume (4,2) is a saddle point. Then we have (8-α)(6-β)>0. Suppose (4,α-8) is the ellipse stable point, then we should have -2α-20<0, (α-10)(2α-β-12)<0, and the discriminant (-2α-20)2+4(α-10)(2α-β-12)>0 to make sure that we will have two negative eigenvalues. One possible choice of α, β is α=0, β=-13.