You can select the books to be arranged in 5*4*5 = 100 ways. And you can arrange the sets selected on the shelf in 6 ways.
If the order does matter, that is the order is specified ahead of time, there is only 1 way to do that; therefore, the 600 ways above is all the possible arrangements.
But if each set is to be arranged within itself then you need to multiply the 600 by 5!*4!*5! which is a big number which I'll leave for you to calculate.