Draw an equilateral triangle, the draw your square inside. You are given that the length of the sides is:
8(2√3+3), we'll use this in a minute.
Notice in your drawing that the square (centered on the base) forms to identical right triangles on each side.
these are 30-60-90 triangles with the 60° angles in each bottom corner. The base of these two triangles added together, when subtracted from the length of the side gives us the base of the square.
Lets call the base of the square a. To find a we need to subtract these two bases from the original length of our side. Notice that the base of a triangle, you can call it b, divided by the upright side of our square, a is the cotangent of the angle 60°.
So cot 60° = b/a which means that b = a cot 60°. Cot 60° is 1/√3. so we can now write:
8(2√3+3) - 2b = a, and with b = a√3
8(2√3+3) - 2a/√3 = a
multiply everything by √3 and we have 8√3(2√3+3) - 2a = a√3
we can combine terms and simplify to 8(2·3 + 3·√3) = (2 + √3)a
bring the factor 3 outside of the () gives 8·3(2 + √3) = (2 + √3)a
we divide through tho remove the (2 + √3) and are left with a = 24
then our square is a² = 24² = 576