This question is all about order of operations.
The mnemonic device to help you remember order of operations is PEMDAS. This stands for Parentheses, Exponents, Multiplication, Division, Addition, and Subtraction. You must perform the operations according to this order to get the correct answer. Some math books teach students to remember these six letters by remembering the phrase "Please Excuse My Dear Aunt Sally".
We start with 5-3(-4-6)^2. Parentheses have the highest precedence, so we do what's in the parentheses first (-4-6). This gives us -10:
5-3(-10)^2. Next we have to take care of the exponent. Raising -10 to the 2 power gives 100:
5-3(100). Next we perform the multiplication, 3 * 100 = 300:
5-300. Finally we do the subtraction, 5 minus 300 gives us -295.