This is a related rates problem, except we have two variables with two different rates of change. Let's start with what we are being asked to find, which is the rate of change of the area outside the circle but inside the square. If we draw a picture and think about this, we have a smaller circle inside a larger square. The full area of the square minus the full area of the circle leaves us with the area in between the two (which is what the problem is essentially asking for).
A = Asq - Acir
At this point, let's define a few variables. Let "x" equal the side length of our square and let "r" equal the radius of the circle.
Asq = x2
Acir = πr2
A = x2 - πr2
Next, lets implicitly differentiate with respect to time, t, so we can use our related rates.
dA/dt = 2x(dx/dt) - 2πr(dr/dt)
Now, at the particular time in the problem statement, we know the following:
x = 18 m
dx/dt = -2 m/min (watch the negative sign as the sides are decreasing in length)
r = 3 m
dr/dt = -3 m/min (again, watch the negative sign)
Next, simply substitute into our above equation and solve for dA/dt (or the rate of change of the area).
dA/dt = [2*(18)*(-2)] - [2π*(3)*(-3)]
dA/dt = (-72) - (-18π)
dA/dt = -72 + 18π m2/min or approx. -15.45 m2/min