I don't know why that happened, but it really doesn't matter. All that really happened was that it went from being (35 MINUS 30) MINUS 40 to being (35 PLUS NEGATIVE 30) PLUS NEGATIVE 40. The equation didn't really change. When yo0u subtract an integer, you add its opposite. 35 MINUS POSITIVE 30 is the same thing as 35 PLUS NEGATIVE 30.
Now to solve this equation, use the order of operations. PEMDAS. Parenthesees come first. 35-30=5, so then it becomes:
3+2[10*5-40]. Brackets are the same as parenthesees, so you deal with what's going on inside the brackets before you deal with what's going on outside the brackets. Then, multiplication comes before addition or subtraction. 10*5=50
3+2[50-40]. 50-40=10
3+2[10]. Now the brackets have been simplified as much as they could possibly be simplified. Now, when you have a number right next to a parenthesis or bracket with no symbol in between, that means you multiply the number that is next to the parenthesis to what's inside the parenthesees. In the order of operations, PEMDAS, multiplication comes before addition. Hence, you do not add the 3 and the 2. You multiply 2[10] to get 20. That leaves you with:
3+20.
So the solution to the problem is 23.

William N.

Just as a disclaimer: I have never actually seen a calculator do this before and so this is simply an educated hypothesis, albeit one that is grounded in the evidence, and must be taken as such. Still, the more important part is that the second equation displayed by your calculator is identitical to the input equation and will still lead to the same answer if you know how to read it.

09/24/12