John R. answered 03/12/23
Calculus, Probability, and Stat Tutor, Math Degree, 20+ years Exp.
There are n! ways of arranging n objects in a line, so there are 5! ways of arranging 5 books on a shelf. More generally, the number of arrangements of size r ≤ n that can be made from n (distinct) objects is n! / (n-r)!. (Remember that 0!=1.)
If we want both math books to be to the left of the 3 engineering books, we can arrange the 2 math books in 2! ways, and for ANY of those math book arrangements we can arrange the 3 engineering books in 3! ways to the right of them. So there are a total of 2!*3! ways of arranging the 5 books with the math books on the left.
If we only care about the number of subsets of size 3, without respect to the order in which they are drawn, we use the formula for combinations, not permutations. If we are selecting r objects from a set of n (distinct) objects, there are n!/ [r!(n-r)!] subsets of size r we can make, so 5!/[3!(5-3)!] for this problem.