Let's assume the rectangle is oriented with the 20 inch side along a horizonal axis and the 6 inch side along the vertical.
We'll be cutting a square of side length x from each of the four corners of the rectangle. Once folded, this will give an open box whose length and width will both be reduced by 2*x, and whose height will be x.
So the dimensions of the open box will be 20 - 2x, 6 - 2x, and x. (Note that this constrains x to be < 3 for a valid solution since otherwise the removal of the square corner will completely remove the side of length 6.)
With these dimensions, the volume V of the box is length * width * height =
(20 - 2x)(6 - 2x)(x) =
(20 - 2x)(6x - 2x2) =
120x - 12x2 - 40x2 + 4x3 = 120x - 52x2 + 4x3
To find the min or max values for this expression, take the first derivative:
dV/dx = 120 - 104x + 12x2
Setting this expression to 0 and using the quadratic formula yields two roots: 7.30 and 1.37. (approx)
The first value can be eliminated because it's larger than the 3" constraint above.
To confirm that the second value is in fact a maximum, plug it into the second derivative:
d2V/dx2 = 24x - 104
Evaluating at x=1.37 yields -71. The fact that this is negative confirms that we had found a max value.