Draw a picture of two intersecting circles. The left circle stands for the family, the right for the pets. Their intersection contains 5, the pictures of both. The total in left circle is 14, or 9 + 5 in the intersection. The right circle total is 8. 5 in the intersection plus 3 in the circle outside the intersection.
The total number of pictures is 9+5+3=17.
I have no understanding of the reason for your remark about GCF, LCM, and prime factorization, but they are certainly good to know.