Skip to content

Click in-app to access the full platform documentation for your version of DataRobot.

Rating Tables

When a model displays the rating table icon on the Leaderboard, you can export the model's complete, validated parameters. Validation assures that the downloaded parameters are correct and that you can reproduce the model's performance outside of DataRobot. For organizations that have the capability enabled, you can modify the table coefficients and apply the new table to the original (parent) model, resulting in a new "child" model available on the Leaderboard.

Note that, for GA2M models, you can specify the pairwise interactions included in the model's output. Before working with rating tables, review the considerations regarding file size and model availability.

Download rating tables

To export rating table coefficients:

  1. From the Leaderboard, identify a model with this icon, indicating that it produced a rating table.

  2. Expand the model and click the Rating Table tab. (The screen may appear different, depending on your permissions.)

  3. Click the Download Table link to save the CSV file. See this additional information for help interpreting the rating table output.

  4. Modify your rating table in a text editor or spreadsheet application. If applicable, you can next upload the modified table to the parent and create a new child model with the table.

Modify rating tables

When you modify a rating table and upload it to the original parent model (and then run the model), DataRobot creates a child model with the modified version of the original parent model's rating table. Available from the Leaderboard, the new model has access to the same features as the parent (with these exceptions).

The following, briefly and then in detail, describes the workflow for creating a new child model.

Workflow overview

The following outlines the steps to iterate on building models with modified rating tables:

  1. Download the rating table from the parent.
  2. Modify the rating table outside of DataRobot using an appropriate editor.
  3. Upload the modified table to the parent model.
  4. Score the new model, adding it to the Leaderboard.
  5. Click Open Child Model to view the new model.
  6. To iterate on rating table changes, download the child's rating table.
  7. Modify the child's rating table outside of DataRobot.
  8. Upload the newly modified table to the parent model.
  9. Return to step 4 and repeat as necessary.

Detailed workflow

The following describes, in more detail, the steps for working with rating tables:

  1. Select a model from the Leaderboard that displays the rating table icon. This is the parent model.

  2. Download the parent model's rating table.

  3. Edit the coefficients in the rating table CSV file using an appropriate editor or spreadsheet.

  4. Once you have completed modifications to the exported rating table, drag-and-drop or browse to upload the new rating table:

    All available (newly and previously uploaded) ratings tables are listed under Uploaded Tables.

  5. If desired, and only before you run the model, you can click the pencil icon to rename the uploaded table, up to 50 characters. Note that the child model's name is based on the name of the rating table it was created from. You can also rename the table outside of the application. If you specify an existing name, DataRobot appends a numeric to the table name.

  6. Click the Add to Leaderboard link to create and score the new model. DataRobot first validates the new rating table and, after building completes, the new child model is available on the Leaderboard. A green check indicates a successfully validated and uploaded table; otherwise, DataRobot displays an error message indicating the issue. (You can monitor build status in the Worker Queue.)

  7. Once the build completes, click the Open Child Model link corresponding to the child model/rating table pair you would like to view. DataRobot opens (and places you in) the Rating Tables tab of the child model. The child model name is Modified Rating Table: <rating_table_name>.csv and is visible and accessible from the Leaderboard.

From the child model, you can do the following:

Link Action
Download Table Download the rating table of the child model. To iterate on coefficient changes in a table, download the child's rating table, upload the modified child rating table to the parent, compare scores, and continue the process as necessary.
Open Parent Model Move back to the Rating Tables tab of the parent (original) model. From there you can upload new tables, build new models, or open any built child models.

Note

You cannot upload a new rating table to the child model. You can only upload rating tables to the parent model.

Rating table validation

When DataRobot builds a model that produces rating tables (for example, GA2M), it runs validation on the model before making it available from the Leaderboard. For validation, DataRobot compares predictions made by Java rating table Scoring Code (the same predictions to produce that specific Rating Table) against predictions made by a Python code model in the DataRobot application that is independent from the rating table CSV file. If the predictions are different, the rating table fails to validate, and DataRobot marks the model as errored.

Feature considerations

Because rating models (GAM, GA2M, and Frequency/Severity) depend on DataRobot's specialized internal code generation for validation, they are limited to 8GB in-RAM datasets. Over this amount, the project may potentially fail due to memory issues. If you get an OOM, decrease the sample size and try again.

Editing considerations

While editing, keep the following in mind:

  • Rating table modification does not support changing the header row of the dataset or data type of the columns. Some editors process data in a way that unintentionally makes these changes, for example, by truncating “000” to “0" or quoting every field so that coefficients are changed from numeric to string. This affects the table that is ultimately re-uploaded. Therefore, DataRobot strongly suggests using a text editor that does not change the data such as Atom or Windows Notepad.

  • If you are using a spreadsheet application, be careful that you do not convert column types (e.g., Num to Date).

  • Rating tables are not created for models with Japanese text columns (they do not support the MeCab tokenizer).

  • In the first section of the table (which defines model parameters and pairwise interactions), you can only modify the values of Intercept and Base.

  • In the first line of the second section (which defines how each variable is used to derive the coefficient that contributes to the prediction), you can edit the value of any column except: Feature Name, Type, Transform1, Value1, Transform2, Value2, and weight.

  • You can add extra columns to the table (for example, to add comments).

  • The Coefficient, Relativity, Intercept, and Base values must be numeric.

  • Base is the exponential of Intercept and is computed from the Intercept value.

  • Relativity is the exponential of Coefficient for each row and is computed from the Coefficient value in the row.

  • Feature Strength is computed from the modified Coefficient values.

  • CSV encoding must be UTF-8.

Additionally, for Frequency/Severity models:

  • The Coefficient value for each row is the sum of the Frequency_Coefficient and Severity_Coefficient values for the row, and is computed from them. Relativity is computed from Coefficient as described above.

  • Frequency_Relativity is the exponential of Frequency_Coefficient for each row, and is computed from the Frequency_Coefficient value in the row.

  • Severity_Relativity is the exponential of Severity_Coefficient for each row and is computed from the Severity_Coefficient value in the row.

  • Frequency_Coefficient, Severity_Coefficient, Frequency_Relativity, and Severity_Relativity values must be numeric.

Child model considerations

When DataRobot creates a child model with the modified version of the original parent model’s rating table, the new model has access to the same features as the parent, with these exceptions:

  • The Advanced Tuning and DataRobot Prime tabs are not available.

  • In the Make Predictions tab, the child model is unable to make predictions on data that was used to train the original. That is, making predictions on the Validation and Holdout partitions of the training data is only possible if those partitions were not used for training. Predictions on those partitions are available when using a newly uploaded dataset.

  • You cannot rerun a child model (for example, with a different feature list or sample size).

  • You cannot change row order when modifying a rating table; any changes will result in error.

  • You cannot upload a new rating table to the child model. You can only upload rating tables to the parent model.


Updated November 10, 2022
Back to top