The radical line has to extend over all the things it is under. As it is written it suggests only the 20 and 80 are being square rooted but this means no simplifications since the x and y do not have the same power and are not like terms.
Now if it were written like this where the parenthesis show where the radical ends
3*√(20x^2)*y^3-2x*√(80)*y^3
We could start by factoring out the like terms (y^3)
y^3(3*√(20x^2)-2x*√(80))
From here in order to simply we need the x's to have the same power. To do that we can put the square root on each term
√(20x^2)=√20*√(x^2)=x√(20)
y^3(3*x√(20)-2x*√(80))
Now we pull out the x like we did with the y
x*y^3(3*√(20)-2*√(80))
In order to simply the radicals they need to be the same base 80=20*4=20*2^2
So we can pull out the four by square rooting it
x*y^3(3*√(20)-2*2*√(20))
Factoring the radical will show us:
x*y^3√(20)(3-4)
-x*y^3√(20)
Hope it helps