draw a tree diagram
3 ways for choice of the 1st book
2 remaing choices for the 2nd book
1 choice for the 3rd book
3x2x1 = 6 total ways to arrange the books.
but that's just if you "normally" shelve them in the traditional manner
you could also stack them one on top of the other, in 6 more orders, getting 6+6 = 12 ways to arrange the books.
or you could stack 2 on top of each other, with the 3rd book next to the other 2
then you'd get 3 choices for top book, 2 more for the bottom book for another 3x2x1 = 6 more ways
Now we're up to 12+6 = 18 total ways to shelf the 3 books
that 3rd book could be vertical with the other two books horizontal, or the 3rd book could be horizontal too, then add in another 6 ways, up to 24 total ways to stack the books.
but I'd go with six, in any math problem like this. Ignoring reality and go with what's expected.