Declare the length of the side of a square to be q and the length of the side of the triangle to be t
we know that the length of the 20ft piece of wire must equal 4q+3t
They split we can say is s
The area of the square is q2
The area of the triangle is slightly tricky we initially don't know the height but if we bisect the equilateral triangle we get two right triangles with angles of 30,60, and 90. That means that we know they are related in the the opposite side of the 90 degree angle is t(because that must be the biggest side) and therefore the 30 degree angle is t/2 and the 60 degree angle which is the height is (√3/2) *t which means that we have an area for that triangle of (t2 *√3)/4. We have two such triangles so total area is (t2 *√3)/2.
A=(1/16)*s2 + √3*(20-s)2/18
A'=s/8 + (-√3/9)*(20-s)
Set it = 0 or graph and you will find the minima at 12.125 and the maxima at 0 for s where the triangle takes up all the area.