Should we adjust for multicollinearity before or after feature selection?

Question

I am conducting a regression analysis with a data set that has high multicollinearity with high VIF values. I adjusted for it before my feature selection, and I also tried adjusting for it only after my feature selection. Both gave me different best models, so I'm wondering which one is statistically significant or better as the latter model has fewer explanatory variables than the former.

Answer 1

It depends!

Do you prefer an intuitive model with fewer features that can be more easily explained and efficiently deployed, or do you primarily care about predictive power? How does each model perform on test and unseen data? What kind of algorithms you are using for the fit, data hungry algorithms (eg nnet or xgboost) usually work better with more data and more features but simpler models may respond better with fewer carefully selected features. You may use also lasso for finding highly effective features first and then use those features. The bottom line is you need to carefully calibrate and validate (using cross validation and train-test splitting) to select the best features and best model.