Manage the Leaderboard¶

Once you start modeling, Workbench begins to construct your model Leaderboard, a list of models ranked by performance, to help with quick model evaluation. The Leaderboard provides a summary of information, including scoring information, for each model built in an experiment. From the Leaderboard, you can click a model to access visualizations for further exploration. Using these tools can help to assess what to do in your next experiment.

DataRobot populates the Leaderboard as it builds, initially displaying up to 50 models. Click Load more models to load 50 more models with each click.

After Workbench completes Quick mode on the 64% sample size phase, the most accurate model is selected and trained on 100% of the data. That model is marked with the Prepared for Deployment badge.

There are several controls available, described in the following sections, for navigating the Leaderboard and exploring models:

• Viewing experiment information
• Filtering models
• Sorting models
• Shorthand controls
• Managing experiments from the Use Case directory

Experiment information¶

Click View experiment info to access the following tabs:

Setup tab¶

The Setup tab reports the parameters used to build the models on the Leaderboard.

Partitioning details¶

The partitioning section's backtest summary and derivation log provide insight into how DataRobot handled the data in preparation for modeling.

• Click View backtests for a visual summary of the observations and testing partitions used to build the model. You cannot change the partitions from this window, but you can make modifications to retrain the model from the Model Overview.

  

• Click View partitioning log to see (and optionally export) a record of how DataRobot created backtests, assigned data based on window settings, and applied other setup criteria.

  

Original data tab¶

The Original data tab provides summary analytics of the data used in the experiment setup. To view exploratory data insights, click the dataset preview link.

By default, the display includes all features in the dataset prior to feature derivation. You can view analytics for features specific to a single feature list by enabling Filter by feature list and selecting a list.

When viewing the list, you can click on the arrow or more info next to a column name to change the sort order.

Derived modeling data tab¶

The Derived modeling data tab is populated after the feature derivation process runs. It shows basic data analytics as well as information that is not available in the Original data tab. Use the Filter by feature list toggle, described in the Original data section, to update the display. Values update to the selected list, as applicable.

Feature lists tab¶

DataRobot automatically constructs time-aware features based on the characteristics of the data. Multiple periodicities can result in several possibilities when constructing the features and, in some cases, it is better to not transform the target by differencing. The choice that yields the optimal accuracy often depends on the data.

After constructing time series features for the data, DataRobot automatically creates multiple lists of features that are used to build models. Then, when experiment building begins, DataRobot automatically runs blueprints using several feature lists, selecting the list that best suits the model type.

Click the Feature lists tab to view all feature lists associated with the experiment. The display shows both DataRobot's automatically created time series feature lists and any custom feature lists that were created prior to model training.

The following actions are available for feature lists from more options to the right of the Created by column:

Custom feature lists can be created prior to modeling from the data explorer or after modeling from the Derived modeling data or Feature lists tabs, as described below.

Create feature list¶

To create a custom feature list, select features individually or use bulk actions. Use the checkbox to the left of the Feature name column to add or clear selections.

To select features individually:

1. From the Derived modeling data or Feature lists tab, select + Create feature list.

   

2. Use the Show features from dropdown to change the displayed features that are available for selection. The default display list features from the Time Series Extracted Features list. All automatically generated and custom lists are available from the dropdown.

   

3. Select the box next to each feature you want to include.

   

4. Optionally, use the search field to update the display to show only those features, within the Show features from selection, that match the search string.

5. Save the list.

Bulk feature list actions¶

To add multiple features at a time, choose a method from the Bulk selection dropdown:

Save feature list¶

Once all features for the list are selected, optionally rename the list and provide a description in the Feature list summary. The summary also provides the count and type of features included in the list.

Then, click Create feature list to save the information. The new list will display in the listing on the Feature lists tab.

Blueprint repository tab¶

The blueprint repository is a library of modeling blueprints available for a selected experiment. Blueprints illustrate the tasks used to build a model, not the model itself. Model blueprints listed in the repository have not necessarily been built yet, but could be as they are of a type that is compatible with the experiment's data and settings.

There are two ways to access the blueprint repository:

• From a Leaderboard model's Blueprint tab.

  

• Click the View experiment info link and select the Blueprint repository tab.

  

Filter models¶

Filtering makes viewing and focusing on relevant models easier. Click Filter models to set the criteria for the models that Workbench displays on the Leaderboard. The choices available for each filter are dependent on the experiment and/or model type and will potentially change as models are added to the experiment. For example:

Sort models by¶

By default, the Leaderboard sorts models based on the score of the validation partition, using the selected optimization metric. You can, however, control the basis of the display parameter when evaluating models.

Note that although Workbench built the experiment using the most appropriate metric for your data, it computes many applicable metrics on each of the models. After the build completes, you can redisplay the Leaderboard listing based on a different metric. It doesn't change any values within the models; it simply reorders the model listing based on model performance in this alternate metric.

See the page on optimization metrics for detailed information on each.

Controls¶

Workbench provides simple, quick shorthand controls:

Icon	Action
	Reruns Quick mode with a different feature list. If you select a feature list that has already run, Workbench replaces the deleted models or make no changes.
	Duplicates the experiment with an option to reuse just the dataset, or the dataset and settings.
	Deletes the experiment and its models. If the experiment is being used by an application, you cannot delete it.
	Slides the Leaderboard panel closed to make additional room for, for example, viewing insights.

Manage experiments¶

At any point after models have been built, you can manage an individual experiment from within its Use Case. Click the more options icon to the right of the experiment name to delete it. To share the experiment, use the Use Case Manage members tool to share the experiment and other associated assets.

What's next?¶

After selecting a model, you can, from within the experiment:

• View experiment insights.
• Add models to experiments.