Skip to content

Click in-app to access the full platform documentation for your version of DataRobot.

Leaderboard reference

The Leaderboard provides a wealth of summary information for each model built in a project. When models complete, DataRobot lists them in the Leaderboard with scoring and build information. Text below the model provides a brief description of the model type and version, or whether it uses unaltered open source code. Badges, tags, and columns, described below, provide quick model identifying and scoring information.

Tags and indicators

The following table describes the tags and indicators:

Display/name Description

BPxx
Blueprint ID
Displays a blueprint ID that represents an instance of a single model type (including version) and feature list. Models that share these characteristics, regardless of the sample size used to build them, have the same blueprint ID. Blender models indicate the blueprints used to create them (for example, BP6+17+20).

Mxx
Model ID
Displays a unique ID for each model on the Leaderboard. The model ID represents a single instance of a model type, feature list, and sample size. Use the model ID to differentiate models when the blueprint ID is the same.

RECOMMENDED FOR DEPLOYMENT
Indicates that this is the model DataRobot recommends for deployment, based on model accuracy and complexity.

PREPARED FOR DEPLOYMENT
Indicates that the model has been through the Autopilot recommendation stages and is ready for deployment.

MOST ACCURATE
Deprecated, applicable to projects created prior to v6.1. Indicates that, based on the validation or cross-validation results, this model is the most accurate model overall on the Leaderboard (in most cases, a blender).

FAST & ACCURATE
Deprecated, applicable to projects created prior to v6.1. Indicates that this is the most accurate individual model on the Leaderboard that passes a set prediction speed guideline. If no models meet the guideline, the badge is not applied. The badge is available for OTV but not time series projects.

BASELINE
Applicable to time series projects only. Indicates a baseline model built using the MASE metric.

Beta
Indicates a model from which you can export the coefficients and transformation parameters necessary to verify steps and make predictions outside of DataRobot. Blueprints that require complex preprocessing will not have the Beta tag because you can’t export their preprocessing in a simple form (ridit transform for numerics, for example). Also note that when a blueprint has coefficients but is not marked with the Beta tag, it indicates that the coefficients are not exact (e.g., they may be rounded).

Frozen Run
Indicates that the model was produced using the frozen run feature. The badge also indicates the sample percent of the original model, for example: .

Insights
Indicates that the model appears on the Insights page.

MONO
Indicates that the model either was built with, or supports but was not built with, monotonic constraints.

Rating tables
Indicates that the model has rating tables available for download.

REF
Indicates that the model is a reference model. A reference model uses no special preprocessing; it is a basic model that you can use to measure performance increase provided by an advanced model.

SCORING CODE
Indicates that the model has Scoring Code available for download.

TUNED
Indicates that the model has been tuned.

Upper Bound Running Time
Indicates that the model exceeded the Upper Bound Running Time.

See also information on the model recommendation calculations.

Model icons

In addition to the tags, DataRobot displays a badge (icon) to the left of the model name indicating the type:

  • : specially tuned DataRobot implementation of a model
  • : anomaly detection model
  • : blender model
  • : Eureqa model
  • : H2O model
  • : Python model
  • : R model
  • : Spark model
  • : TensorFlow model
  • : Vowpal Wabbit (VW) model
  • : XGBoost model
  • : custom model, built with Jupyter Notebooks (deprecated)

Text below the model provides a brief description of the model type and version, or whether it uses unaltered open source code.

Columns and tools

Leaderboard columns give you at-a-glance information about a model's "specs":

The following table describes the Leaderboard columns and tools:

Column Description
Model Name and Description Provides the model name (type) as well as identifiers and description.
Feature List Lists the name of the Feature List used to create the model. Click the Feature List label to get a count of the number of features in the list.
Sample Size Displays the sample size used to create the model. Click the Sample Size label to see the number of rows the sample size represents or to set the display to only selected sample sizes. By default, DataRobot displays all sample sizes run for a project.
Validation Displays the Validation score of the model. This is the score derived from the first cross-validation fold. Some scores may be marked with an asterisk, indicating in-sample predictions.
Cross-Validation Displays the Cross-Validation score, if run. If the dataset is greater than 50,000 rows, DataRobot does not automatically start a cross-validation run. You can click the Run link to run cross-validation manually. Some scores may be marked with an asterisk, indicating in-sample predictions. If the dataset is larger than 800MB, cross-validation is not allowed.
Holdout Displays a lock icon that indicates whether holdout is unlocked for the model. When unlocked, some scores may be marked with an asterisk, indicating use of in-sample predictions to derive the score.
Metric Sets (and displays the selection of) an accuracy metric for the Leaderboard. Models display in order of their scoring (best to worst) for the metric chosen before the model building process. Click the orange arrow to access a dropdown that allows you to change the optimization metric.
Menu Provides quick access to comparing models, adding and deleting models, and creating blender models.
Search Searches for a model, as described below.
Add New Model Adds a model based on specific criteria that you set from the dialog.
Filter Models Filters by starred models. Alternatively, click a Leaderboard tag to filter by the selected tag.
Export Allows you to download the Leaderboard's contents as a CSV file, as described below.

Tag models for quick reference

You can tag or "star" one or more models on the Leaderboard, making it easier to refer back to the model when navigating through the application. You can then filter your Leaderboard results so that they only include starred models. Note that you can combine the star tag with badge filtering to, for example, only display tagged Elastic Net Classifier models.

To star a model, hover over the model in the Leaderboard and click the empty star:

To unselect the model, click again on the star. To filter and display only starred models, click the Filter Models label and select Starred Models in the dropdown.

To further filter by model type, click on a model badge in your results. (You can also click the badge first and then Filter Models.) DataRobot reports the filtering criteria in the action bar.

Search the Leaderboard

The Leaderboard provides a model filtering capability that limits the display to only those models matching your search criteria. There are three methods for searching:

  1. Click a badge that a model is tagged with to redisplay the list showing only models matching that type. The filter type is displayed in the action bar.

  2. Click in the search box and begin typing. As you type, the list automatically narrows to those models meeting (containing) your search criteria. Leaderboard search, which is case insensitive, filters on:

    Enter in the Search box... Returns...
    <model-name> or <model-feature> All models with name or features matching the search term.
    BI or bi All models with export coefficients with preprocessing information available.
    Reference All reference models.
    Insights All models that appear on the Insights page.
    Tuned All models created with the Advanced Tuning link.
    <tuned-subtitle> or <tuned-description> All tuned models with a subtitle or description matching the search term.
    BPxx All models matching the supplied blueprint number; search accepts an entry with our without a space between "BP" and the number.
    Mxx All models matching the supplied model number; search accepts an entry with or without a space between "M" and the number.
  3. You can display only starred models using the Filter Models dropdown.

Export the Leaderboard

The Leaderboard allows you to download its contents as a CSV file. To do so, click the Export button on the action bar:

Doing so prompts a preview screen:

This screen displays the Leaderboard contents (1), which you can copy, and lets you rename the .csv file (2). Note that:

  • .csv is the only available file type for exporting the Leaderboard.
  • Holdout scores are only included in the report if Holdout has been unlocked.

Click Download to export the contents.

Blender models

A blender model, sometimes referred to as an ensemble model, is a model that increases accuracy by combining the predictions of two or more models. DataRobot automatically creates blender models, after Autopilot runs, based on the top three regular Leaderboard models (for PLS, GLM, and average blenders) and the top eight models for advanced blenders (advanced average, advanced GLM, and ENET). Note that you can also manually create blender models from the Leaderboard.

To improve response times for blender models, DataRobot stores predictions for all models trained at the highest sample size used by Autopilot (typically 64%) and creates blenders from those results. Storing only the largest sample size (and therefore predictions from the best performing models) has the advantage of limiting the required disk space.

Asterisked scores

Sometimes, the Leaderboard's Validation, Cross-Validation, or Holdout score displays an asterisk. Hover over the score for a tooltip explaining the reason for the asterisk:

Note

Asterisked partitions do not apply to time series or multiseries projects.

By default, DataRobot uses up to 64% of the data for the training set. This is the largest sample size that does not include any data from the validation or holdout sets (16% of the data is reserved for the validation set and 20% for the holdout set). When model building finishes, you can manually train at larger sample sizes (for example, 80% or 100%). If you train above 64% but under 80%, the model trains on data from the validation set. If you train above 80%, the model trains on data from the holdout set.

As a result, if you train above 64%, DataRobot marks the Validation score with an asterisk to indicate that some in-sample predictions were used for that score. If you train above 80%, the Holdout score is also asterisked to indicate use of in-sample predictions to derive the score.

N/A scores

Sometimes, the Leaderboard's Validation, Cross-Validation, or Holdout score displays “N/A” instead of a score. This occurs if your project was trained into the validation or holdout set and meets any of the following criteria:

  • The dataset is larger than 750MB —resulting in a slim run project, which can have models that do not have stacked predictions.
  • It is a date/time partitioned project (both OTV and time series) and all of its models do not have stacked predictions.
  • It is a multiclass project with greater than 10 classes.
  • It is a Eureqa project (Eureqa models do not have stacked predictions).

Updated November 30, 2021
Back to top