Here's my best shot:
The first step is generate an equation for the volume of the cylinder, V=pi(r^2)h. It's radius can be the independant variable, but you now need height. This height can be found by finding an equation for the "sloped" surface of the cone.
You know that at x=0, y=5. At x=3.5, y=0. Generate the equation of the line: y=mx+b, plug in rise/run, and you get y=-(5/3.5)x + 5, or made proper, y=-(10/7)x +5 (note negative slope).
So now at any radius x within the cone, you know the height of the cylinder where it hits the cone will be given by that equation. Relating this back to the equation for volume, x=r and y=h.
So now you need to go back to what you are trying to optimize, the volume. You had two variables on the right side, you need to get it down to one. You now have an equation that relates the two, so plug in the equation for the height:
A=pi(r^2)h = pi (r^2) (-(10/7)r+5)).
Multiply it out: A=-pi(10/7)r^3 + 5pi(r^2)
This will give you the volume at any point X.
To find a min/max, you take the derivative and find the roots (set equal to 0).
-(30/7)pi(r^2) + 10pi(r) = 0
factor out 10pi(r)
10pi(r) * (-(3/7)r + 1)
One root is obviously 0, which doesn't work. To find the other, set the other factor to 0:
-(3/7)r +1 = 0
Solve for r: r=7/3, or 2.33333333
that is your radious
plug back into equation for y way above for height:
y = -(10/7)(7/3) + 5 = 5/3 or 1.6666667
Should work! But please check!!!