If we try to divide 2 by 0, why isn't that just infinity? Well, infinity isn't really a number. It's just a concept and it represents a place that you can't ever really get to. Math deals with numbers, and it just can't handle this.
Here's a way to think about it. If I divide 20 by 5, I get 4 as the answer. Turning that around, it says that I would need 4 groups of size 5 to make 20. So what would 20 divided by zero mean? How many groups of size zero would I need to make 20? There is no number of groups that would ever add up to 20. Even if you could add up an infinite number of zeros, the total would still be zero. You just can't get to 20! The answer is therefore undefined.
This is by no means mathematically rigorous, but it gives one an intuitive picture of why you just can't do that.