Chromosomes take up physical space. Thus, the more copies an organism has, the more crowded their nucleus and in conjunction, the more crowded their overall cells will be. To compensate, polyploid organisms typically have larger cells. However, in order to have larger cells, one of two things must happen: the organism must grow significantly larger, or it must have fewer cells.
Polyploid plants have chosen the latter-that is, they have less cells to accommodate for their expansive cell size. This seems to work out okay for them, as they arguably don't need as much genetic activity as animals. Animals, however, require so many cells to perform specialized functions that they can't really get rid of cells in favor of cell size expansion. On the other hand, growing larger to accommodate increased cell size can be unfavorable from an energy standpoint, so both options seem out of the question.
Hope this helps!