This is an example of a permutation, because the order of the books matters. If the order didn't matter, it would be a combination.
The formula for the number of permutations of n objects taken r at a time is given by this formula:
where ! is the factorial operator. In our example, n=5 and r=3, so we have:
P(5,3) = 5!/(5-3)!