This is a classic optimization problem in calculus. Let s be the length of the sides of the squares to be cut out.
Then the dimensions of the open box will be 20 - 2s by 20 -2s by s .
The volume is thus V = (20 - 2s)2 s
To find the value of s which maximizes this, compute the derivative dV/ds and set it equal to zero.
Using the product rule, power rule and chain rule dV/ds = (20 - 2s)2 - 4 s (20 - 2s)
Setting this equal to zero gives (20 - 2s) - 4 s = 0 with the solution
s = 10/3
The maximum volume is then (20 - 20/3)2 (10/3) = 16000/27