Fun question! And as you can see, the complete answer isn't short (sorry!) There are three main points:
- The most common reason to put verbs before subjects is to turn statements into questions. "You are ok." vs. "Are you ok?" In older forms of English, e.g., Shakespearean, most questions were formed this way. "You heard the news." vs. "Heard you the news?" In modern English, we usually replace the predicate verb with an auxiliary verb like "do" or "have" and move that in front of the subject, followed by the infinitive of the original verb. So "You heard me." vs. "Did you hear me?" Obviously that's not what's happening in the Tolkien, but it's the most common reason to put verbs in front of subjects.
- In the passage you quoted, the subject of "lived" is actually "there," which is the impersonal pronoun of existence (in other words, when you just want to say that something exists), and not, in this case, a preposition indicating location. Compare the "there"s in this sentence: "There is a man over there." Tolkien is using the first one, so in this case, the subject DOES precede the verb, and it's not an example of the device you're describing.
- But you are right that that is a literary device, or anyway, used to be, so yeah, let's talk about it! In older forms of English, switching subject and verb order was more common, not only to form questions, but in other situations too. You see it all the time in Shakespeare. Look at Egeus's first line in Midsummer, "Full of vexation come I..." or King Lear's early announcement, "divide we in three our kingdom." Besides fitting into a poetic meter, changing word order shifts the emphasis of the sentence. Egeus wants you to know how vexed he is as soon as possible, Lear wants to make sure no one misses the fact of the division.
Coming back to Tolkien, let's try it two ways, pretending, like Tolkien's original audience, we've never heard of hobbits (and taking out the "there" to nullify point 2). "In a hole in the ground, a hobbit lived." Oh, ok, something or other lived in a hole in the ground, I didn't catch what it was, but it's probably not important. Now try: "In a hole in the ground lived a hobbit." A what? What's that? Tell me more!
This device has mostly faded from English in the last hundred years or so, so the main purpose of using it now would be to make the sentence sound older, possibly Shakespearean in tone. The one exception I can think of off the top of my head would be, like Tolkien, beginning a story. Compare the impact of: "Mary lived in New York" and "In New York lived Mary." Which one makes you want to know more about Mary? Like I said, I can't immediately think of any other times when you'd do that in a contemporary voice, but if you find one, let me know!