A set is a collection of things, called elements, that share a common property. So yes, family members are a collection of things (people) who share the common property of being in your family. Other examples:
The set of calico cats (thing=cats, property=calico)
The set of pens with green ink (thing=pens, property=green ink)
The set of integers less than 5 and greater than -2 (thing=integer, property = less than 5 and greater than -2)