Because minerals are made at very specific conditions. For example, diamonds are made from carbon (common), that is heated, (less common), under extreme pressure (really uncommon), for extended periods of time (really really uncommon). So when you "add" that up essentially, It becomes rare to get a diamond. Other minerals are made with rapid cooling after volcanic eruption. You can take the same element(s) and if they cool slowly after volcanic eruption, they form a completely different mineral.
So again, its all because each mineral is made after the material reaches very specific conditions.