A good hunch would be "d." The first choice doesn't seem reasonable: multiplying a generic 3x1 vector by A will produce non-zero results in every element.
For the second choice, the transpose AT is:
1 a b
0 1 a ≠ A
0 0 0
How the determinant could be negative for ALL values of a and b (as answer c states) is hard to imagine, so by POE, answer d makes sense, but we'll calculate it to be sure.
However, det A = 1-a
This is not equivalent to a2- b, and it is not always <0, so the answer is "none of the above."