You probably find it difficult to see that the second verb listen is actually an infinitive (to listen).
The reason you don't see it as an infinitive is that it is usual with this structure to omit the word "to."
Take a look just below and notice how I identify each part of the sentence.
Subject verb object
He let his daughter (to) listen to the music.
The words "listen to the music" all together are the object. Listen cannot be the second verb you want it to be because it is already part of the object of the sentence.