a^3b - ab^3

could mean

a^(3b) - a(b)^3 or

a^(3b) - (ab)^3 or

a^3(b) - a(b)^3 or

a^3(b) - (ab)^3

most likely though is

a^3(b) - a(b)^3

where you can factor out an ab to get

ab[a^2-b^2] = ab(a-b)(a+b)

but a^3(b) - (ab)^3 is possible, sort of 2nd most likely

it's the same as a^3(b) - a^3(b^3) which factors into

a^3(b)[1-b^2] = (a^3)(b)(1-b)(1+b)

but you really just meant what is the greatest common factor of these two terms

(a^3)b and -a(b^3)

both have an a and both have a and b

so ab is the greatest common factor

if you meant (a^3)b and -(ab)^3 or -a^3b^3 then

the greatest common factor is

(a^3)b,