Any point P on the ellipse can be represented as (a·cos(θ), b·sin(θ)), where θ can range from 0 to 2π.
Let us assume A is the corner of the inscribed rectangle that is located in the first quadrant. Then θ only ranges from 0 to π ⁄ 2.
Then, the lengths of the sides of the inscribed rectangle are 2a·cos(θ) and 2b·sin(θ). And the area of the rectangle is 4ab·cos(θ)·sin(θ) = 2ab·sin(2θ) since 2·sin(θ)·cos(θ) = sin(2θ).
In summary, the area of the rectangle is 2ab·sin(2θ) which in turn has a maximum value of 2ab since sin(2θ) has a maximum value of 1.
Answer : 2ab.
Hope that helps!