This is a great question, and it comes down to harmony.
In a lot of melodies we'll hear it start off on a certain pitch, and move away from that pitch before finally returning to it at the end. In those examples, it's fairly easy to discern what the key is as it's most likely that starting and ending pitch. But, what about when things aren't so clear, and like you say, it uses pitches that come from a few different scales? Well, we can tell what key a melody is in usually by the harmonies that a composer uses to support that melody. The harmony that we can hear the melody driving toward; where our ears finally feel a sense of resolution between harmony and melody, that is how we know what key a melody belongs to.
Introducing new pitches does not necessarily mean that the key has changed. When we hear that sense of resolution occurring with a different harmony than it previously did, then we can say that a piece has modulated. Typically we would expect a full harmonic progression from tonic area to predominant area to dominant area and back to tonic before we could say that something has truly modulated. Adding a few different notes could color a key, but not necessarily change it. For example if you were to borrow a few pitches from the parallel minor key you could change the overall sound of a passage without necessarily changing the key. This actually happens all the time in popular music, for example a "blue note" is technically a borrowed pitch, but using it doesn't mean that a song or melody has modulated.