The easiest way to proceed on this one is to note that if there is no solution, the determinant of the coefficient matrix must be zero. So the determinant of
1 1 5
1 2 -2
4 11 k must be zero. This determinant can be worked out by hand resulting in the equation
2k - 8 + 55 -40 +22 - k = 0
The solution is k = -29. With k = -29 there are two possibilities. There could be no solution or an infinite number of solutions. If the latter (which we would like to rule out), there will be a solution for which z = 0. Plugging z =0 into the original equations we get:
x + y = -3
x + 2y = 3
4x + 11y = 31
The first two of these can be solved to get x = -9 and y = 6. Plugging these into the last equation yields
30 = 31 - a contradiction. Thus there is no solution for z = 0, so this is not a case of an infinite number of solutions.
In conclusion with k = -29 there is no solution.
