There is some ambuguity to what you wrote, but based on the context I think this is what you mean:
(a^3)*b - a*(b^3)
Notice that both terms have one 'a' and one 'b'. We can rewrite this as:
(a^3)*b - a*(b^3) = (a^2)*ab - (b^2)*ab = a*b*(a^2 - b^2)
[since a^3 = a*a*a = a*(a^2) = (a^2)*a]
So 'ab' is the GCF.