It has to do with the ability of the ions produced to interact with the water molecule. For example, when cations such as Al3+ are introduced into solution, the 3+ charge on the cation causes the electron distribution in the water molecule to shift more towards the oxygen, essentially making it easier for protons (hydrogen ions) to fall off the water molecule. When the concentration of the hydrogen ion increases, the pH decreases (becomes more acidic). Basically (pun intended) the water molecule becomes more acidic.
Certain anions can also affect the pH if they are capable of abstracting (taking) protons from water. When this happens hydroxide is left behind from the water molecule which increases the concentration of OH- and raises the pH.
Many ions are spectators in acid-base chemistry, Alkali metal ions, conjugates of the strong acids / bases etc. These ions would not affect the pH and we would expect solutions that contain exclusively those ions to remain neutral.