Skip to contentSearching many parameter combinations makes it easier to find a configuration that fits the data you searched. That result still needs evaluation on data that did not influence selection. A larger search is not a substitute for validation.
Separate the windows before searching
Choose a search window and an untouched evaluation window. Keep the split chronological for a time-dependent backtest. Record the dates, data source, instrument universe, costs, and strategy/dependency versions.
Use only the search window to select parameters. A pipeline that prepares features must avoid bringing information from the evaluation window into earlier decisions. Include enough warmup history without allowing future observations to influence signals.
Define what you will compare
Choose objective directions before starting. Return, drawdown, and trade count describe different aspects of a backtest. A high return from very few trades may be difficult to interpret, and drawdown can change your assessment of that return.
Compare candidates with a fixed baseline under the same costs and execution assumptions. Inspect several promising configurations, including nearby parameter values, instead of treating a single top-ranked point as a robust region.
Freeze candidates before evaluation
Export the selected parameter sets and keep their image/data versions. Run each candidate on the untouched window. Preserve the objectives and extraction rules. Record both the search and evaluation results rather than replacing the former.
If you change the strategy or choose another candidate after looking at evaluation results, that window has participated in selection. It no longer provides the same independent check. Reserve additional evaluation data when you need another iteration.
Inspect the execution behind the number
Check that the backtest completed the intended window, that objective metrics came from actual results, and that missing fields were not replaced with zero. Inspect trade/order counts and logs. Include costs appropriate to your workload and understand the assumptions your backtest engine makes about fills.
Historical results do not predict live returns. A managed optimizer handles execution and comparison; it does not establish that a strategy generalizes.
Keep the workflow repeatable
Use separate experiments for search and evaluation, preserve their configurations, and describe which window each uses. Start from the Freqtrade or NautilusTrader guide.
The same principle applies to model validation and simulation evaluation. HyperOptimizer supports any containerized workload with tunable parameters, repeatable trials, and objective metrics. Your workload owns the data split and evaluation logic.