Why is the whole note used as reference for "note length"?

We have a whole note, and then half note, quarter note, eighth, etc. I am wondering, why is it that way? If we think in term of beat, then the quarter note is the reference: there are 2 eighth note in a quarter note, 4 16th note, etc. When using a metronome, a beat is also the atomic unit. Thus, why not using a system where a 8th note is... A half note (2 per beat)? I don't think anyone thinks as a 32nd note as a note "that's repeated 32 times in a 4/4 measure". A "note, the length of which is 1/8th of a beat" sounds easier and more concrete to me.