NextGen UI documentation > AI experimentation > Evaluate with model insights > Hyperparameter Tuning

Hyperparameter Tuning¶

Tab	Description
Details	Allows manually setting of model hyperparameters, overriding the DataRobot selections for a model. In some cases, by experimenting with hyperparameter settings, you can improve model performance.

Hyperparameter Tuning allows you to manually set model hyperparameters, overriding the DataRobot selections for a model, and create a new Leaderboard model using the new settings. In some cases, experimenting with hyperparameter settings can improve model performance. When you create models via this insight, the newly created Leaderboard models can later be blended together or further tuned.

When you provide new exploratory values, save, and build using new hyperparameter values, DataRobot creates a new child model using the best of each parameter value and adds it to the Leaderboard.

You can further tune that model and create a child of the child. In other words, you do not iterate and continue to tune a single child model, instead you modify the child and create another new model, providing a lineage of changes. You can also create a new child from the original parent.

Notes on hyperparameter tuning usage

To compute scores for tuned models, DataRobot uses an internal "grid search" partition inside of the training dataset. Typically the partition is a 80/20 training/validation split, although in some cases DataRobot applies five-fold cross-validation.
Grid search is only available preprocessing parameters; tuning is available for all parameters.
You cannot tune blended models.
See also architecture, augmentation, and tuning options specific to Visual AI image augmentation projects.

	Element	Description
1	Search	Filters the display to include only those hyperparameters matching the search strings. If a task has no matching hyperparameters, it does not display.
2	Task name	Provides the pre- or post-processing steps (tasks) that make up the model, with a link to the DataRobot model documentation, which describes the task in detail. Under each task are listed: Parameter: The individual hyperparameters, which are tunable, that make up that task. Current value: The single best value of all searched values for preprocessing or final prediction model hyperparameters. This is the value that was used to build the model. On the editing page, this value is displayed in the Parent column, because tuning results in a new model. Searched values: All values DataRobot searched before selecting the best value.
3	Tune parameters	Opens the editing window, allowing you to modify hyperparameter values.

Create a tuned model¶

Customize hyperparameter settings, testing individual values or ranges, to create new Leaderboard models. You can set new values for preprocessing tasks, with availability dependent on the model and project type. To create a new model—a version (child) of the parent but with hyperparameter modifications—do the following:

Click Tune parameters to open the hyperparameter tuning editor.
Select the hyperparameter to tune, either by scrolling though the list or filtering by search string. Click on the name.
Edit the value; save changes for each modification.
Select a search method
Click Train new model. Once clicked, DataRobot begins training one or more new models. Expand the right panel to follow progress.
On the Leaderboard, find the new child by searching for the Tuned badge and comparing the model IDs. The model card also includes a summary of changes.
Open the new model, return to Details > Hyperparameter tuning and evaluate the new child compared model.

Edit hyperparameter values¶

To edit a hyperparameter value, first click Tune parameters and then click a selection to open. You can modify up to 12 hyperparameters per model.

Avoid memory errors

The following situations can cause memory errors:

Setting any hyperparameter that accepts multiple values such that it results in a grid search that exceeds 25 grid points. Be aware that the search is multiplicative, not additive (for example, max_depth=1,2,3,4,5 with learning_rate=.1,.2,.3,.4,.5 results in 25 grid points, not 10).
Increasing the range of hyperparameters searched so that they result in larger model sizes (for example, by increasing the number of estimators or tree depth in an XGBoost model).

There are a variety of accepted value types and ranges; each hyperparameter includes instruction and reports the parent value.

CategoricalSingle valueMultiple values

Many values provide the option to select User-defined. When selected, the entry expands to include a field for entering the desired value. Validation checks the entry, indicating the issue and turning from red to white when the entry passes.

Click Save changes after each modification; hyperparameters are marked with an Edited badge and the entry is added to the edited parameters list in the right panel. Before training a new tuned model, you must modify at least one parameter.

Select a search method¶

Set the search type to configure the level of search detail, which in turn affects resource usage.

Search type	Description
Bayesian search	Provides a balanced search by using Bayesian optimization to intelligently balance exploration with time spent tuning. That is, it uses previous results to direct next search actions. When selecting Bayesian, also set the max iterations, max time (in hours), and random seed.
Smart search	Performs a sophisticated pattern search (optimization) that emphasizes areas where the model is likely to do well and, for faster results, skips hyperparameter points that are less relevant to the model. DataRobot explores parameter values by starting with initial grid points and then iteratively finding the best-performing points. Results are returned when all promising regions are explored or maximum iterations are reached.
Brute force	Evaluates each data point and all possible parameter value combinations for maximized accuracy, likely increasing time and resource use.

Deep dive: Smart search heuristics

The following describes how DataRobot performs Smart Search:

Use a start value if specified. Otherwise, initialize the first grid pass to be the cross-product of the 25th and 75th percentile among each parameter's grid points and score.
Find the best-performing grid points among those that were searched.
From the unsearched grid points, find the neighbor grid points of the best-performing searched grid point and score for those points.
Add those values to the "searched" list and repeat from Step 2.
If no neighbors are found, search all adjacent neighbors (edges). If neighbors are found, repeat from Step 2.
If still no neighbors are found, reduce the search radius for neighbors. If neighbors are found, repeat from Step 2.
Repeat Steps 2-5 until there are no new neighbors to search or the maximum number of iterations (max_iter) is reached.

Once a method is selected, optionally enter a tune description. The panel also provides a summary of all hyperparameters modified, reporting an error if too many have been tuned.

Evaluate child models¶

Once a child model is trained, it is listed on the Leaderboard with the Tuned badge and a summary of modifications. Open the model and navigate to the same insight—Details > Hyperparameter Tuning—to see changes. From there, you can:

Understanding iterations

Model type	Description	Generation
Parent	The original model built in the experiment.	First
Child	A model in which hyperparemters of the parent were modified.	Second
Child of the child	A model in which hyperparemters of a child were modified.	Third and later

	Element	Description
1	Parameters table/Grid search	Compare and evaluate changes—use Parameters table for list view or Grid search for a graphical format.
2	Search parameters	When in table view, use Show only tuned parameters to see only those you have changed. If you choose to re-tune the model, all parameters are displayed.
3	Tune parameters	Re-tune the model based on the settings of this child model.
3	Open parent model	Click to change the view to the parent model. If you are making modifications to a child model, the listed parent model is the original parent, not the child that the "child of the child" was trained from.

Parameters table¶

The table view lists all tunable hyperparameters of the model, in the same breakdown as the when tuning was available for the parent model. In this view, however, in addition to the current and searched values, DataRobot displays the value used by the original parent.

From here, you can create an iteration of the child by clicking Tune parameters or return to the parent (Open parent model) to create a "second generation" model

Grid search¶

The grid search visualization, available on models for which DataRobot ran a grid search, plots parameters against score for child models. It visualizes patterns across numeric parameter-to-score relationships, helping to identify which parameter combinations lead to the best model performance. The number and detail of the graphs vary based on model type. Not all model types or hyperparameters generate grid search graphs since not all use a grid search to find the best value.

DataRobot graphs those parameters that take a numeric or decimal value and displays them against a score. In the example above, the top three graphs each plot one of the parameters used to build the current model. The points on the graph indicate each value DataRobot tried for that parameter. The starred dot is the value selected, and dots are represented in a worst-to-best color scheme.

The additional rows of graphs provide parameter-to-parameter information that illustrates an analysis of co-occurrence—plotting the parameter values against each other. The graphs can be helpful in experimenting with parameter selection because they provide a visual indicator of the values that DataRobot tried. So, for example, to try something completely different, you can look for empty regions in the graph and set parameter values to match an area in the empty region. Or, if you want to try tweaking something that you know did well, you can identify values in the region near to the star that represents the best value.