Valentius Kryptix

Hyperparameter Tuning: Grid Search vs Random Search on a Random Forest

Every machine learning model has settings you choose before training starts. These are hyperparameters, and examples include the number of trees in a Random Forest, how deep each tree can grow, and how many features each split considers. Unlike model weights, they aren’t learned from the data. You have to pick them, and the choice can affect how well the model performs.

Hyperparameter tuning is the process of searching for a good combination of these settings. In this post I compare two popular approaches, GridSearchCV and RandomizedSearchCV from scikit-learn, by tuning a Random Forest on the breast cancer dataset (569 samples, 30 features).

The experiment setup

To keep the comparison fair, I set up the experiment carefully:

• I held out 20% of the data as a stratified test set. Neither search ever sees it.
• All tuning used 5-fold cross-validation inside the remaining 80%.
• The scoring metric was ROC AUC.
• I compared four setups: the untuned default model, a grid search over 54 candidates, a random search with the same budget of 54 iterations, and a random search with only one third of that budget (18 iterations).

How the two methods differ: Grid search tries every combination from a fixed list of values. It is thorough, but the cost grows quickly as you add parameters. Random search samples combinations from ranges or distributions, so it can explore a wider space and you decide how many tries to spend.

The results

What I learned

1. Tuning barely helped on this dataset. All methods landed within about 0.002 ROC AUC of each other in cross-validation. The default Random Forest was already close to the best result. The test set has only 114 samples, so the small differences in test scores amount to a few samples and are within noise. Tuning isn’t always worth a lot of effort, and an easy dataset like this one is a good example.

2. Equal fits don’t mean equal time. Grid and random search with the same budget both ran 270 fits, yet random search took about twice as long (80.5s vs 37.5s). Its search space allowed larger forests of up to 300 trees and deeper trees, so each fit was more expensive. The number of candidates is not the same as the cost.

3. A smaller random budget got close. Random search with one third of the budget finished in 24.9 seconds and came within about 0.0016 CV ROC AUC of the best result. That is the usual argument for random search: you get most of the benefit for a fraction of the compute.

4. The plots tell you which parameters matter. In the grid, max_depth=5 scored best, while depths of 10 and None gave identical scores. That suggests the trees rarely grow deeper than 10 on this data. In the random search, lower max_features fractions tended to score slightly better, though the trend was weak.

5. The grid looked better partly because I designed it. Grid candidates scored higher on average because I built the grid around sensible values. The random space was wider and included weaker regions. When a method looks better, ask whether the search space itself was the reason.

Caveats

These results come from a single run with a fixed seed, and timings will vary by machine. For a more rigorous comparison, I would repeat the experiment across several seeds and datasets, including harder ones where tuning has more room to matter.

Takeaways

• Start with the default model and measure it. Tuning only makes sense if you know what you’re improving on.
• Keep your test set untouched until the end, and tune only with cross-validation on the training data.
• When you have many hyperparameters or a limited compute budget, random search is a strong starting point. Grid search works well when you already know which values are worth trying.
• Judge the cost of a search by actual time as well as by the number of candidates.

You can find the full code, results and plots on GitHub: github.com/yeshi-rnj/hp-tuning