
Michael H. answered 06/20/19
In-depth knowledge combined with clunky use of technology!
That's a great question!! Here are a couple of factors you may want to consider.
In addition to being influenced by a polygenic set of alleles that are the same for both sexes, height is also sexually dimorphic -- that is, if a man and a women have exactly the same set of the alleles mentioned above, he will ordinarily grown to be taller than she will (think hormones). So sexual selection or cultural preferences you cited will not necessarily cancel out and result in stablizing selection at all.
On the other hand, in the absence of sexual selection there is an optimal height for human beings based both on their environment (niche -- like job and, as you noted, food supply. If protein is scarce, shorter might be a better strategy) and based on the need for their various systems to function harmoniously (think breeds of dog that have hip problems). So the sexual selective pressure for the population to grow taller will perhaps be counterbalanced by the disadvantages of the parts no longer working together as well as they did at some earlier height.