1) In order to multiply two polynomials, one must multiply each term in the second polynomial by each term in the first one and add the results, also combining the like terms. In your case, first polynomial has two terms: a and b. Second polynomial has three terms: a^{3}, -3ab, and -b^{2}. Now let us multiply each of those three by the first term of the first polynomial, that is, a.

a*a^{3}=a^{4}

a*(-3ab)=-3a^{2}b

a*(-b^{2})=-ab^{2}

Next, do the same with the second term of the first polynomial, that is b.

b*a^{3}=ba^{3}

b*(-3ab)=-3ab^{2}

b*(-b^{2})=-b^{3}

Now we shall add all six results together:

(a+b)(a^{3}-3ab-b^{2})=a^{4}-3a^{2}b-ab^{2}+ba^{3}-3ab^{2}-b^{3};

Now we collect like terms. Those are terms that has the same variable structure. In your case those are the third and the fifth terms in the product, -ab^{2} and -3ab^{2}. Notice, they both have
*a* to the first power, and *b* to the second power, multiplied together. Their sum is -4ab^{2}. Notice also, that the second term, though looking similar, is not the same, since it has
*a* raised to the second power and *b* raised to just the first power. Therefore, it cannot be combined with the other two we just added together. So, we finally obtain:

(a+b)(a^{3}-3ab-b^{2})=a^{4}-3a^{2}b-4ab^{2}+ba^{3}-b^{3};