Rating Tables¶

Rating tables helps provide a transparent view of a model by breaking down how an individual feature contributes to that model's predictions. You can export and then download the validated parameters used by the model as a CSV file, modify the coefficients, and apply the new CSV to the original (parent) model. This creates a new child model on the Leaderboard, which you can compare to the parent or other child models to see the impact of your changes.

The downloaded table helps you understand:

• Which features were chosen to form the prediction.
• The importance of each of these features.
• Whether a feature has a positive or negative impact on the outcome.

The Rating Tables insight provide information and actions for model tuning:     

The following provides an outline of the steps for using rating tables to iterate on, and manually tune, a model's logic. Each is detailed below:

1. From the Leaderboard, identify a supported model type and download the rating table.
2. Modify the coefficients outside of DataRobot.
3. Upload the modified table to the parent model.
4. Score the new model, adding it to the Leaderboard.
5. Open the child model to compare child scores with previous versions.
6. To iterate, download the child's rating table, modify as necessary, and create and score a new child model.

Create a child model¶

Create a child model by downloading, modifying, and re-uploading coefficients:

1. From the Leaderboard, select a GAM or GAM2 model (see model type availabliity)and open the Details > Rating Tables tab.

   

2. Click Download table and save the CSV file locally.

   Note

   See this additional information for help interpreting the rating table output.

3. Modify the coefficients in the rating table CSV file. Before making changes, review the considerations for editing guidance and calculation explanations.

   

4. With the parent open in the Rating Tables insight, upload the modified CSV via drag-and-drop or browsing.

   

5. DataRobot validates the new rating table and, after validation, provides an option to train the new model. Click to train, and when training completes, DataRobot adds the child model to the Leaderboard. The model name is in the format Modified Rating Table: .

   

Parent model view¶

Once one or more child models have been created from a parent, the Rating Tables insight changes to reflect those changes. All child models associated with the parent are listed.

From here you can:

Child model view¶

You can access child models from either the parent's Rating Tables insight or from the Leaderboard listing.

Actions available for working with the child include:

If you modify a child's rating table to iterate on coefficient changes, return to the parent to upload the new table and create a new child. You can then compare scores to evaluate changes. You cannot upload a new rating table to the child model.

Rating table validation¶

Validation assures that the downloaded parameters are correct and that you can reproduce the model's performance outside of DataRobot. When DataRobot builds a model that produces rating tables, it runs validation on the model before making it available from the Leaderboard. For validation, DataRobot compares predictions made by Java rating table Scoring Code (the same predictions to produce that specific rating table) against predictions made by a Python code model in the DataRobot application that is independent from the rating table CSV file. If the predictions are different, the rating table fails to validate, and DataRobot marks the model as errored.

Feature considerations¶

Review the availability, general, and editing considerations.

Availability¶

Rating tables are available for:

• GAM and GAM2 models.
• Frequency‑Cost GA2M and Frequency/Severity GAM and GA2M.
• They are not available for non-GAM Frequency/Severity or Frequency/Cost.
• They are not available for Eureqa GAM models.

General¶

The following are general considerations to keep in mind:

• Because rating models (GAM, GA2M, and Frequency/Severity) depend on DataRobot's specialized internal code generation for validation, they are limited to 8GB in-RAM datasets. Over this amount, the project may potentially fail due to memory issues. If you get an OOM, decrease the sample size and try again.

• Rating tables are not available for time-aware experiments.

• Rating tables are not created for models with Japanese text columns (they do not support the MeCab tokenizer).

• Models with custom rating tables cannot be retrained.

• Models with custom rating tables cannot be blended.

• Modified CSVs used to build child models are not stored in the Data Registry.

Editing¶

The following considerations are specific to editing the rating table output:

• Rating table modification does not support changing the header row of the dataset or data type of the columns. Some editors process data in a way that unintentionally makes these changes, for example, by truncating “000” to “0" or quoting every field so that coefficients are changed from numeric to string. This affects the table that is ultimately re-uploaded. Therefore, DataRobot strongly suggests using a text editor that does not change the data.

• If you are using a spreadsheet application, be careful that you do not convert column types (e.g., Num to Date). When you modify a rating table and upload it to the original parent model (and then run the model), DataRobot creates a child model with the modified version of the original parent model's rating table. Available from the Leaderboard, the new model has access to the same features as the parent (with these exceptions).

• In the first section of the table (which defines model parameters and pairwise interactions), you can only modify the values of Intercept and Base.

  

• In the first line of the second section (which defines how each variable is used to derive the coefficient that contributes to the prediction), you can edit the value of any column except: Feature Name, Type, Transform1, Value1, Transform2, Value2, and weight.

• You can add extra columns to the table (for example, to add comments).

• The Coefficient, Relativity, Intercept, and Base values must be numeric.

• Base is the exponential of Intercept and is computed from the Intercept value.

• Relativity is the exponential of Coefficient for each row and is computed from the Coefficient value in the row.

• Feature Strength is computed from the modified Coefficient values.

• CSV encoding must be UTF-8.

Additionally, for Frequency/Severity models:

• The Coefficient value for each row is the sum of the Frequency_Coefficient and Severity_Coefficient values for the row, and is computed from them. Relativity is computed from Coefficient as described above.

• Frequency_Relativity is the exponential of Frequency_Coefficient for each row, and is computed from the Frequency_Coefficient value in the row.

• Severity_Relativity is the exponential of Severity_Coefficient for each row and is computed from the Severity_Coefficient value in the row.

• Frequency_Coefficient, Severity_Coefficient, Frequency_Relativity, and Severity_Relativity values must be numeric.

Child model considerations¶

When DataRobot creates a child model with the modified version of the original parent model’s rating table, the new model has access to the same features as the parent, with these exceptions:

• In the Make Predictions tab, the child model is unable to make predictions on data that was used to train the original. That is, making predictions on the Validation and Holdout partitions of the training data is only possible if those partitions were not used for training. Predictions on those partitions are available when using a newly uploaded dataset.

• You cannot retrain a child model (for example, with a different feature list or sample size).

• You cannot change row order when modifying a rating table; any changes will result in error.

• You cannot upload a new rating table to the child model. You can only upload rating tables to the parent model.

• Models with a custom rating tables (child models) cannot be blended.