Hyperparameter Tuning

Hyperparameter tuning searches over choices that are not directly learned by gradient descent but strongly affect training and generalization. Examples include learning rate, weight decay, batch size, model width, dropout rate, augmentation strength, and scheduler settings.

A tuning problem can be written as:

$${\lambda }^{\text{*}}=\arg \underset{\lambda \in \Lambda }{\min }{\hat{R}}_{\mathrm{val}}\left(f_{\mathrm{Train}({\mathcal{D}}_{\mathrm{train}},\lambda )}\right)$$

where $\Lambda$ is the search space and ${\hat{R}}_{\mathrm{val}}$ is validation risk. The final claim should not use validation risk as the test estimate.

Search Methods

• Manual tuning: small, interpretable changes guided by diagnostics.
• Grid search: exhaustive combinations over a small discrete space.
• Random search: sampled configurations, often stronger than naive grids.
• Bayesian optimization: model-based search over promising regions.
• Successive halving or bandit search: early resource allocation based on partial runs.
• Population-based training: jointly changes weights and hyperparameters during training.

Each method changes the selection budget and must be recorded when it affects the final model.

Common Hyperparameters

Area	Examples	Main risk
Optimization	learning rate, warmup, schedule, optimizer	unstable or slow training
Regularization	weight decay, dropout, label smoothing	overfit or underfit
Data	batch size, sampling, augmentation	biased gradient or distribution mismatch
Architecture	depth, width, heads, hidden size	capacity and compute tradeoff
Selection	early-stopping patience, checkpoint rule	validation overuse
Inference	threshold, temperature, decoding	metric mismatch or calibration drift

Budget and Overfitting

If many trials are evaluated on the same validation set, tuning can overfit validation:

$${\lambda }^{\text{*}}=\arg \underset{\lambda \in \Lambda }{\min }\left[R(f_{\lambda })+{ϵ}_{\lambda ,\mathrm{val}}\right]$$

The selected ${\lambda }^{\text{*}}$ can exploit noise ${ϵ}_{\lambda ,\mathrm{val}}$. A larger search space should therefore be paired with a stronger final test boundary or nested validation design.

Checks

• Is the search space documented before interpreting results?
• Are random seeds, failed trials, and early-stopped trials included in the selection story?
• Are preprocessing parameters fit only inside the train split for each trial?
• Does the tuning metric match the final deployment metric?
• Is the final test set untouched until the selected configuration is fixed?
• Is compute budget comparable when comparing methods?

Related

• Model selection
• Regularization
• Weight decay
• Batch size
• Learning rate schedule
• Training stability
• Cross-validation
• Leakage
• Experiment tracking