Why is multicollinearity an issue for some clustering methods?

Question

Imagine you have data for a group of patients in a medical study. Your features include height, blood pressure, oral temperature, forehead temperature, sex, weight and a battery of other symptoms associated with an illness you're interested in studying. You cluster the patients, but find that your clusters look odd - what about your data might be causing this issue and why?

Answer 1

Many clustering methods define similarity using distance. When two variables measure nearly the same underlying feature, including both effectively gives that feature extra weight.

For example, oral and forehead temperature are likely highly correlated. In a Euclidean-distance method such as k-means, differences in body temperature would therefore be counted twice. The algorithm might form clusters driven mainly by temperature while underweighting other symptoms that are more relevant to the illness.

Standardizing the variables is important because it prevents variables with larger numerical scales from dominating, but standardization alone does not remove correlation. Possible solutions include removing one redundant variable, combining correlated measurements into a meaningful summary, or using PCA or another dimensionality-reduction method. The choice should be guided by the scientific question: if the difference between oral and forehead temperature carries useful information, that difference could be modeled directly rather than simply discarding one measurement.

Also, a feature such as sex is categorical, so it may require a distance measure or clustering method designed for mixed data.

Answer 2

Multicollinearity causes redundant features to dominate distance calculations. If correlated features are weighted equally, they effectively get counted multiple times, distorting the true similarity between data points.

Result: Clusters group primarily by these correlated physical characteristics rather than the illness symptoms you actually care about. The redundant features drown out the important but less-correlated symptom patterns.

Answer 3

You realize that your results are dominated by temperature measurements - individuals with similar temperature measurements are clustered together to a relatively high degree, while individuals with similarities in other measurements are not. This is because when you added forehead and oral temperature, you essentially included the same concept, body temperature, twice. This is bad for two reasons - one: it over-weights body temperature as a concept (remember how it seemed like our clusters put the people with similar temperatures together but didn't necessarily do the same for other types of symptoms?) and two: it ruins the story our cluster model tells. Clustering is useful for its ability to segment data (patients, customers, etc) into intelligible and meaningful groups, so when we overweight one concept, we make it harder for our audience to read the segments in the underlying data.

Note that this assumes that both temperature readings are correlated and essentially representative of the same underlying concept. If there were some meaningful information to be gleaned from the difference between the two measurements then we might want to include both features.