The population size of a species within a location is general location is determined through physical surveys and statistical models.
Generally speaking, the "appropriate" number of individuals within a species can be determined by the supply of the prey, foraging items, habitat size, and other factors. For example, if the number of deer increase within a specific area to the point where they are depleting the habitat's resources faster than it can produce (ex: eating grass or forage material and this material declines), then that would be "too much" of this species within this local population.
On the opposite side of the spectrum, the decline of a species is concerning. The IUCN Red List has a list of categories to place animal species ranging from: least concern, near threatened, vulnerable, endangered, critically endangered, and extinct. The placement of a species depends on what percentage the population has reduced over a certain time span, if the total population size is less than a set number, and other factors.
Regarding pandas, it is hard to replenish the current population because the reproductive success of captive pandas is low (likely because the "mating ritual" of pandas have not been studied in depth). Furthermore, it takes time to create a breeding colony in which a group of captive animals, such as pandas, could be taught the skills necessary to successfully thrive in the wild by other members of its species. This is often done through slow re-introduction over generations, especially if a "parent" panda cannot pass on these skills to their offspring.
Another factor against the reintroduction of pandas is the increasing rate of deforestation and urbanization. Pandas are mobile animals and need a large amount of space to thrive; therefore, the space left in the future may not suit them.