Yes and no. Take a look at two of the best swimmers ever:
Muscles? You bet. But would you say they had "built up" muscles if you saw them fully dressed? Probably not.
Consider the other extreme.
The point is, find something you like doing that's healthy. Do it a lot, and before long you'll look a little more like other people that do that activity.
In the meanwhile, forget about how you look and enjoy what you're doing. That's far more attractive in a person anyway.