When the squares each with side x are cut out from each corner, the length and with are each shortened by 2x (one length of x from each end). So, the bottom of the box will be 17 - 2x by 11 - 2x. The sides that are turned up have the height equal to the side of the square which is x.
So, the volume of the box is V = (17 - 2x)(11 - 2x)(x).
x must be less than 5.5 inches because if it is larger, there would be nothing remaining of the side that is 11 inches. If you put this Volume into your graphing calculator, with a winow for xmin = 0 and xmax = 5.5, you see that you probably need to adjust the ymax also. set it at 200. Use, 2nd Calc, - max(4).
This gives you x = 2.1778 and y = 182.9667.
I'm not sure what you are to round to, but lets round x = 2,
Then x = 2, 17-2x =13 and 11-2x = 7 these give you a volume = 182 in3