The energy of the charged species must be more sensitive to hydrogen bonding than uncharged species. Polar protic solvent generally slows down Sn2 reactions with charged nucleophiles because it stabilizes the reactants, mainly the nucleophile, more than it stabilizes the transition state.
Polarizable nucleophiles are also less electronegative, meaning they have a higher energy HOMO. This gives better energy match with the LUMO of the alkyl halide electrophile. This aspect is less often discussed in undergraduate texts, at least American ones.