This gets complicated and muddy very quickly. Would you rate someone with a 4.99 rating and 100 ratings higher or lower than someone with a 4.98 rating and 200 ratings?
You could change your rating system from 1 to 5 to instead -10 to +10, and simply add ratings together. Someone with 99 +10's and 1 +9 would then rate higher than someone with 50 +10's.
Or you could make the ratings simply +1, 0 or -1, and add them up.
The math can't tell you what you value more.