How do I tune hyperparameters in XGBoost or Random Forest?

Answer 1

Common approaches for tuning hyperparameters in XGBoost and Random Forest:

Methods:

1. Grid Search - Try all combinations of specified parameters
2. Random Search - Randomly sample parameter combinations
3. Bayesian Optimization - Intelligently search based on previous results
4. Cross-Validation - Use k-fold CV to evaluate each parameter set

Key hyperparameters to tune:

Random Forest:

1. n_estimators: number of trees (start with 100-500)
2. max_depth: maximum tree depth (None, or 10-50)
3. min_samples_split: minimum samples to split a node (2-10)
4. min_samples_leaf: minimum samples in leaf node (1-5)
5. max_features: features to consider for splits (sqrt, log2, or fraction)

XGBoost:

1. n_estimators: number of boosting rounds (100-1000)
2. learning_rate: step size (0.01-0.3)
3. max_depth: tree depth (3-10)
4. min_child_weight: minimum sum of instance weight (1-10)
5. subsample: fraction of samples per tree (0.5-1.0)
6. colsample_bytree: fraction of features per tree (0.5-1.0)
7. gamma: minimum loss reduction (0-5)

Answer 2

In general, after you have tried fitting with default values and early stopping (for XGBoost), you would do a grid search or random search over a chosen range (wider initially) for around 2-4 hyperparameters at a time. Then, once you narrow down the range, you run a finer search, rinse and repeat until the gains plateau or you hit your goal.

More specifically for Random Forest:

The main hyperparameters you'd want to tune are n_estimators (number of trees) and max_depth. If necessary, max_features is also good to tune.

1. Use RandomizedSearchCV to sample 20-50 random combinations for n_estimators and max_depth (and max_features, optionally.) Fit it with your cross-validation to pick out the best settings.
2. You could fine-tune it further by doing a GridSearchCV around those best settings you picked out.

For XGBoost:

The main hyperparameters you'd want to tune are learning_rate, max_depth, and n_estimators. Let early stopping pick the n_estimators (for example, stop if there is no improvement after 20 rounds).

1. Choose and try out a few learning rates, and pick the best one.
2. Set your learning rate, then choose your early stopping rounds and fit it to find your n_estimators.
3. Use RandomizedSearchCV on max_depth with one or two regularizers like subsample and gamma.
4. Fine-tune it further with GridSearchCV if necessary.

Remember to watch out for overfitting and validate your model on a separate test set. Good luck!