I cannot see your figure, but I will assume you will cut squares from each corner of the rectangle of length x.
The the volume V = x(20-2x)(16-2x) = 4x(10-x)(8-x)
dV/dx = 0 = 80-36x+3x2
The maximum occurs at x = (approximately) 2.95 and the maximum value is (approximately) 105.
Ooops, if you don't have calculus, I am not sure how to tell you to do this problem except by graphing V as a function of x.