This is a very standard optimization question in a Calculus AB curriculum. You are given a quantity to minimize or maximize (in this case, the cost), which you will do by taking its derivative, setting it = 0, and solving. You are also given a constraint (in this case, each pen must enclose 4,500 sq. ft.), which will allow you to write a function for the quantity you are optimizing in one variable only.
Draw a diagram letting w represent the width of each pen along the river and letting l represent the length of each pen away from the river, the perimeter that needs enclosing is 4w + 3l. However, the 2w of stones along the river costs $80 / ft while the remaining 2w+ 3l of fencing is only $10 / ft. (This difference in cost, btw, suggests that the ideal w to minimize cost will be a lot smaller than the ideal l.)
Total cost, C, is therefore C = 160w + 20w +30l or C = 180w + 30l. We would like to take a derivative, set it = 0, and solve, but our cost function is still expressed in two variables. This is where we use the constraint.
lw = 4,500. Isolating l, we get l = 4,500 / w. Substituting that expression in for l in the cost equation yields:
C = 180w + 135,000 / w.
C' = 180 - 135,000w-2 Set = 0 and solve for w: 180 - 135,000w-2 = 0 135,000w-2 = 180 w-2 = 1 / 750
w2 = 750 and w ~ 27.4 ft. l = 4,500 / 27.4 ~ 164.3 ft You can plug w = 27.4 into the cost function to find the minimum cost of the project.
If the question asks you to justify that this is a minimum, you can show that C' goes from - to + at w = 27.4 ft