Well, let's start by guessing that making a square will maximize the area.
Now let's do the math to back that up:
Start by making a careful sketch, noticing that this rectangle will lie in QI, with its top right vertex on the line x + y = 12. That line has a slope = - 1, and has a y-int of (0 , 12) and an x-int of (12 , 0).
Next place a bottom right vertex on the positive x-axis at an arbitrary point, x, somewhere between 0 and 12. Now extend the right side of the rectangle up to the line x + y = 12 and place the top right vertex there. Then complete the rectangle by drawing the top side parallel to the x-axis.
The dimensions of this rectangle are x by y but we will define y in terms of x to get an equation for its area in terms of x only, take the derivative, set it = 0, and solve for the ideal x:
A = x·y but for any point on the line x + y = 12, the y-coordinate is given by y = (12 - x).
A = x·(12 - x) = -x2 + 12x
A' = -2x + 12 = 0. , x = 6. , y = 6 , Max area = 36