Which accent is most difficult for ESL students to understand depends on their native language, and what kind of accent the speaker has. Students commonly recognize English speakers who come from the same geographic areas, for example an Asian student who is hearing another ESL speaker who can speak an Asian language.
Another important reason is the root of which language is being spoken. Here's are two examples: English and German come from the same language family (Proto-Germanic), and Spanish, Italian and French all come the root language of Latin.