Skip to content02 / From recipes to software
Models have settings, too.
In machine learning, there is a useful distinction between what a model learns from data and the choices that control how it learns.
- The model learnsParameters
- The values fitted from training data, such as a neural network’s weights.
- You configureHyperparameters
- Choices such as learning rate, maximum tree depth, or the number of trees. A search can help select them.
Our software example: a random forest classifier
We’ll vary maximum tree depth and minimum samples per leaf, then compare validation accuracy. Different settings can change how well the model generalizes; tuning does not guarantee an improvement.
The same search pattern works beyond ML: trading strategy settings, simulation inputs, and other programs with settings and a measurable result. This is often called parameter optimization.
03 / Choose what to try next
Three ways to explore the same space.
The task and allowed settings stay the same. The search method changes how the next combination is chosen. Switch methods, then step through the trials.
Trial 12 · depth 16 · leaf size 12
Grid search
Cover a fixed set of combinations.
Choose a list of values for each setting, then test every combination in that grid. Previous scores do not change the order.
Here: 4 depth values × 3 leaf sizes = 12 trials.
- When it helps
- Useful when the space is small and you want to cover specific values.
- Keep in mind
- The number of combinations grows quickly as you add settings.
Keep covering the chosen combinations.
Trial 12 · depth 6 · leaf size 8
Random search
Sample across the search space.
Draw settings from the ranges or distributions you define. Each new choice is independent of previous scores.
Here: 12 sampled combinations within the same ranges.
- When it helps
- A useful baseline when you have a fixed trial budget or many settings.
- Keep in mind
- It does not use completed results to guide later choices.
Continue sampling independently of the scores.
Trial 12 · depth 4 · leaf size 6
Bayesian optimization
Let results guide the next choice.
Use completed trials to model which settings may work well. Balance testing promising settings with exploring less certain areas.
Here: broad initial trials, followed by an illustrative guided sequence.
- When it helps
- Often useful when each trial is expensive and the budget is limited.
- Keep in mind
- The outcome depends on the workload, search space, and available results.
Use results to explore and refine promising settings.
Illustration, not a benchmark. Scores and trial sequences are fictional; no optimizer runs in this demo. Real optimizers only see the results of trials they have run.
04 / Read the results
Keep the settings behind every score.
Here are the top three trials from the illustrative Bayesian sequence above. Each result stays connected to the exact combination that produced it.
Random forest classifier
- Depth range
- 2–16
- Minimum leaf size
- 1–12 samples
- Trial budget
- 12 runs
| Trial | Max. depth | Min. leaf size | Validation accuracy |
|---|---|---|---|
| #010Best found | 10 | 3 | 92.5% |
| #006 | 11 | 4 | 91.2% |
| #007 | 9 | 2 | 91.2% |
Fictional scores for illustration. This is a comparison of tested settings, not a claim about model performance.
- Search space
- The values each setting is allowed to take.
- Trial
- One run with one combination of settings.
- Objective
- The metric to improve, and whether to maximize or minimize it.
- Best found
- The strongest measured result among the trials you ran.
05 / Check what holds up
A promising score is a starting point.
Searching more settings gives you more chances to find a high score—including a lucky one. Check that the result holds up beyond the data used to choose it.
Compare against a baseline.
Evaluate the starting configuration and new trials using the same metric and validation procedure.
Keep final test data separate.
Use validation data to select settings. Evaluate the chosen model on untouched test data; repeatedly tuning against the test set makes it part of the search.
Check variation between runs.
When results depend on randomness, repeat promising configurations with different seeds. Look for a reliable improvement, not just the highest single score.
Read about selection bias and validation in scikit-learn
A few practical questions
How many trials do I need?
Start with a budget you can afford and compare against a baseline. The useful trial count depends on the search space, run time, and how much scores vary. There is no universal number.
Will it find the best possible settings?
There is no general guarantee. A search finds the best result among the combinations it evaluates, using the metric and data you provide.
Does running in parallel change the search method?
The search method chooses settings. Parallel execution runs several trials at once. It can reduce elapsed time when enough compute is available; adaptive methods can use a completed result only after that trial finishes.
06 / Put it to work
Run your first parameter search.
HyperOptimizer brings search methods, parallel cloud execution, and results into one workflow. You define the experiment; we run the trials and collect their metrics and logs.
Bring your program
A Docker image with your code and dependencies.
Define the search
Settings, a metric, and a trial budget.
Compare the results
Every run, with its parameters and logs.
7-day Starter trial · payment details required
Prefer your terminal? Define the search in hpo.toml and launch with the CLI