The approach to solving this problem is coming up with an equation that relates the length of the ladder to the distance it is placed behind the fence. Once this equation is known, we can take its derivitive and solve for the relevant zero.
I solved this by using two variables (x and θ) to represent the distance between the fence and the ladder and the angle the ladder makes with the ground, and l to represent the length of the ladder. The last piece of the puzzle is knowing that in order for the ladder to be as short as possible, it should touch the top of the fence.
|\ Simple trig gets us: l = x/cos(θ) + 6/cos(θ)
| \
| \ Because we know the ladder touches the fence, we can get: tan(θ) = 5/x
| \
| \ Substituting θ for x, we get: l = 5/sin(θ) + 6/cos(θ)
| \
| 5 | \
|_____| θ\
6 x
Now that we have the equation, we should take its derivitive. (1/sin(θ) = csc(θ), 1/cos(θ) = sec(θ))
dl/dθ = -csc(θ)*cot(θ)*5 + sec(θ)*tan(θ)*6
We are only interested in angles 0-90 so we need to solve the zeroes of this equation. Its easiest for me to use a graphing calculator in these situations, just make sure it is set to degrees.
Solving the zero of the derivitive, we get θ = 43.26 degrees.
Now we use x = 5/tan(θ) to solve x = 5.31
Finally we use x = 5.31 and l = (x+6)/cos(θ) to get l = 15.53 ft