Good insight present in your question!
A real battery has what is called internal resistance. This acts as a relatively small resistance when compared to other resistances in the circuit - at least when the battery is fresh.
Any other resistors will be in series with the internal resistance of the battery. The effective resistance of the circuit is then added to the internal resistance of the battery to give the total resistance of the entire circuit. As the circuit resistance gets smaller, the total resistance of the circuit gets smaller, and the current through the circuit (and this includes the battery) increases.
So, since P = I2R, as the external resistance decreases, the current increases, and then the power dissipated by the internal resistance of the battery increases. [Keep in mind that the internal resistance is fairly constant, at least for a fresh battery. As the battery is used, the internal resistance starts to increase, and increases even more as the current increases, which makes the heating even more pronounced!]
