The Leaderboard provides a wealth of summary information for each model built in a project. When models complete, DataRobot lists them in the Leaderboard with scoring and build information. Text below the model provides a brief description of the model type and version, or whether it uses unaltered open source code. Badges, tags, and columns, described below, provide quick model identifying and scoring information.
Tags and indicators¶
The following table describes the tags and indicators:
|Displays a blueprint ID that represents an instance of a single model type (including version) and feature list. Models that share these characteristics, regardless of the sample size used to build them, have the same blueprint ID. Blender models indicate the blueprints used to create them (for example, BP6+17+20).|
|Displays a unique ID for each model on the Leaderboard. The model ID represents a single instance of a model type, feature list, and sample size. Use the model ID to differentiate models when the blueprint ID is the same.|
RECOMMENDED FOR DEPLOYMENT
|Indicates that this is the model DataRobot recommends for deployment, based on model accuracy and complexity.|
PREPARED FOR DEPLOYMENT
|Indicates that the model has been through the Autopilot recommendation stages and is ready for deployment.|
|Deprecated, applicable to projects created prior to v6.1. Indicates that, based on the validation or cross-validation results, this model is the most accurate model overall on the Leaderboard (in most cases, a blender).|
FAST & ACCURATE
|Deprecated, applicable to projects created prior to v6.1. Indicates that this is the most accurate individual model on the Leaderboard that passes a set prediction speed guideline. If no models meet the guideline, the badge is not applied. The badge is available for OTV but not time series projects.|
|Applicable to time series projects only. Indicates a baseline model built using the MASE metric.|
|Indicates a model from which you can export the coefficients and transformation parameters necessary to verify steps and make predictions outside of DataRobot. Blueprints that require complex preprocessing will not have the Beta tag because you can’t export their preprocessing in a simple form (ridit transform for numerics, for example). Also note that when a blueprint has coefficients but is not marked with the Beta tag, it indicates that the coefficients are not exact (e.g., they may be rounded).|
|Indicates that the model was produced using the frozen run feature. The badge also indicates the sample percent of the original model, for example: .|
|Indicates that the model appears on the Insights page.|
|Indicates that the model either was built with, or supports but was not built with, monotonic constraints.|
|Indicates that the model has rating tables available for download.|
|Indicates that the model is a reference model. A reference model uses no special preprocessing; it is a basic model that you can use to measure performance increase provided by an advanced model.|
|Indicates that the model has Scoring Code available for download.|
|Indicates that the model has been tuned.|
Upper Bound Running Time
|Indicates that the model exceeded the Upper Bound Running Time.|
See also information on the model recommendation calculations.
In addition to the tags, DataRobot displays a badge (icon) to the left of the model name indicating the type:
- : specially tuned DataRobot implementation of a model
- : anomaly detection model
- : blender model
- : Eureqa model
- : H2O model
- : Python model
- : R model
- : Spark model
- : TensorFlow model
- : Vowpal Wabbit (VW) model
- : XGBoost model
- : custom model, built with Jupyter Notebooks (deprecated)
Text below the model provides a brief description of the model type and version, or whether it uses unaltered open source code.
Columns and tools¶
Leaderboard columns give you at-a-glance information about a model's "specs":
The following table describes the Leaderboard columns and tools:
|Model Name and Description||Provides the model name (type) as well as identifiers and description.|
|Feature List||Lists the name of the Feature List used to create the model. Click the Feature List label to get a count of the number of features in the list.|
|Sample Size||Displays the sample size used to create the model. Click the Sample Size label to see the number of rows the sample size represents or to set the display to only selected sample sizes. By default, DataRobot displays all sample sizes run for a project.|
|Validation||Displays the Validation score of the model. This is the score derived from the first cross-validation fold. Some scores may be marked with an asterisk, indicating in-sample predictions.|
|Cross-Validation||Displays the Cross-Validation score, if run. If the dataset is greater than 50,000 rows, DataRobot does not automatically start a cross-validation run. You can click the Run link to run cross-validation manually. Some scores may be marked with an asterisk, indicating in-sample predictions. If the dataset is larger than 800MB, cross-validation is not allowed.|
|Holdout||Displays a lock icon that indicates whether holdout is unlocked for the model. When unlocked, some scores may be marked with an asterisk, indicating use of in-sample predictions to derive the score.|
|Metric||Sets (and displays the selection of) an accuracy metric for the Leaderboard. Models display in order of their scoring (best to worst) for the metric chosen before the model building process. Click the orange arrow to access a dropdown that allows you to change the optimization metric.|
|Menu||Provides quick access to comparing models, adding and deleting models, and creating blender models.|
|Search||Searches for a model, as described below.|
|Add New Model||Adds a model based on specific criteria that you set from the dialog.|
|Filter Models||Filters by starred models. Alternatively, click a Leaderboard tag to filter by the selected tag.|
|Export||Allows you to download the Leaderboard's contents as a CSV file, as described below.|
Tag models for quick reference¶
You can tag or "star" one or more models on the Leaderboard, making it easier to refer back to the model when navigating through the application. You can then filter your Leaderboard results so that they only include starred models. Note that you can combine the star tag with badge filtering to, for example, only display tagged Elastic Net Classifier models.
To star a model, hover over the model in the Leaderboard and click the empty star:
To unselect the model, click again on the star. To filter and display only starred models, click the Filter Models label and select Starred Models in the dropdown.
To further filter by model type, click on a model badge in your results. (You can also click the badge first and then Filter Models.) DataRobot reports the filtering criteria in the action bar.
Search the Leaderboard¶
The Leaderboard provides a model filtering capability that limits the display to only those models matching your search criteria. There are three methods for searching:
Click a badge that a model is tagged with to redisplay the list showing only models matching that type. The filter type is displayed in the action bar.
Click in the search box and begin typing. As you type, the list automatically narrows to those models meeting (containing) your search criteria. Leaderboard search, which is case insensitive, filters on:
Enter in the Search box... Returns... <model-name> or <model-feature> All models with name or features matching the search term. BI or bi All models with export coefficients with preprocessing information available. Reference All reference models. Insights All models that appear on the Insights page. Tuned All models created with the Advanced Tuning link. <tuned-subtitle> or <tuned-description> All tuned models with a subtitle or description matching the search term. BPxx All models matching the supplied blueprint number; search accepts an entry with our without a space between "BP" and the number. Mxx All models matching the supplied model number; search accepts an entry with or without a space between "M" and the number.
You can display only starred models using the Filter Models dropdown.
Export the Leaderboard¶
The Leaderboard allows you to download its contents as a CSV file. To do so, click the Export button on the action bar:
Doing so prompts a preview screen:
This screen displays the Leaderboard contents (1), which you can copy, and lets you rename the .csv file (2). Note that:
- .csv is the only available file type for exporting the Leaderboard.
- Holdout scores are only included in the report if Holdout has been unlocked.
Click Download to export the contents.
A blender model, sometimes referred to as an ensemble model, is a model that increases accuracy by combining the predictions of two or more models. DataRobot automatically creates blender models, after Autopilot runs, based on the top three regular Leaderboard models (for PLS, GLM, and average blenders) and the top eight models for advanced blenders (advanced average, advanced GLM, and ENET). Note that you can also manually create blender models from the Leaderboard.
To improve response times for blender models, DataRobot stores predictions for all models trained at the highest sample size used by Autopilot (typically 64%) and creates blenders from those results. Storing only the largest sample size (and therefore predictions from the best performing models) has the advantage of limiting the required disk space.
Sometimes, the Leaderboard's Validation, Cross-Validation, or Holdout score displays an asterisk. Hover over the score for a tooltip explaining the reason for the asterisk:
Asterisked partitions do not apply to time series or multiseries projects.
By default, DataRobot uses up to 64% of the data for the training set. This is the largest sample size that does not include any data from the validation or holdout sets (16% of the data is reserved for the validation set and 20% for the holdout set). When model building finishes, you can manually train at larger sample sizes (for example, 80% or 100%). If you train above 64% but under 80%, the model trains on data from the validation set. If you train above 80%, the model trains on data from the holdout set.
As a result, if you train above 64%, DataRobot marks the Validation score with an asterisk to indicate that some in-sample predictions were used for that score. If you train above 80%, the Holdout score is also asterisked to indicate use of in-sample predictions to derive the score.
Sometimes, the Leaderboard's Validation, Cross-Validation, or Holdout score displays “N/A” instead of a score. This occurs if your project was trained into the validation or holdout set and meets any of the following criteria:
- The dataset is larger than 750MB —resulting in a slim run project, which can have models that do not have stacked predictions.
- It is a date/time partitioned project (both OTV and time series) and all of its models do not have stacked predictions.
- It is a multiclass project with greater than 10 classes.
- It is a Eureqa project (Eureqa models do not have stacked predictions).