# Retraining
URL: https://docs.datarobot.com/en/docs/workbench/nxt-console/nxt-mitigation/nxt-retraining.html

> Set up automated retraining policies to maintain model performance after deployment.

# Retraining

To maintain model performance after deployment without extensive manual work, DataRobot provides an automatic retraining capability for deployments. On the Mitigation > Retraining tab, upon providing a retraining dataset registered in the Data Registry, you can define up to five retraining policies on each deployment. Each policy consists of a trigger, a modeling strategy, modeling settings, and a replacement action. When triggered, retraining will produce a new model based on these settings and notify you to consider promoting it.

> [!TIP] Configure retraining settings
> To configure an automated retraining policy, the deployment's [retraining settings](https://docs.datarobot.com/en/docs/workbench/nxt-console/nxt-settings/nxt-retraining-settings.html) must be configured.

## Create an automated retraining policy

To create and define a retraining policy for a deployment, navigate to the deployment's Mitigation > Retraining tab:

1. On theRetraining Summarypage, click+ Add Retraining Policy. If you haven't set up retraining, clickConfigure Retrainingand configure theretraining settings.
2. Enter a policy name and, optionally, a policy description.
3. Configure the following retraining policy settings:
4. ClickSave policy.

### Retraining trigger

Retraining policies can be triggered manually or in response to three types of conditions:

- Automatic schedule: Sets the time for the retraining policy to trigger. Choose from increments ranging from every three months to every day. Note that DataRobot uses your local time zone.
- Accuracy status: Initiates retraining when the deployment's accuracy status changes from a better status to the levels you select (green to yellow, yellow to red, etc.). After you configure the status change settings, you can also configure theStatus frequency options, to determine if DataRobot should continue to send notifications (on an ISO-string defined interval) while the accuracy status remains "At risk" or "Failing."
- Drift status: Initiates retraining when the deployment's data drift status declines to the level(s) you select (green to yellow, yellow to red, etc.). After you configure the status change settings, you can also configure theStatus frequency options, to determine if DataRobot should continue to send notifications (on an ISO-string defined interval) while the drift status remains "At risk" or "Failing."

> [!NOTE] Drift and accuracy trigger definitions
> Data drift and accuracy triggers are based on the definitions configured on the [Settings > Data drift](https://docs.datarobot.com/en/docs/workbench/nxt-console/nxt-settings/nxt-data-drift-settings.html) and [Settings > Accuracy](https://docs.datarobot.com/en/docs/workbench/nxt-console/nxt-settings/nxt-accuracy-settings.html) tabs.

Once initiated, a retraining policy cannot be triggered again until it completes. For example, if a retraining policy is set to run every hour but takes more than an hour to complete, it will complete the first run rather than start over or queue with the second scheduled trigger. Only one trigger condition can be chosen for each retraining policy.

### Use Case

To link a retraining policy to a Use Case in Workbench, select an existing Use Case or create a new Use Case. When a retraining policy is linked to a Use Case, the registered retraining models are listed [in the Use Case's assets](https://docs.datarobot.com/en/docs/workbench/nxt-workbench/usecases/build-usecase.html#manage), under More > Registered models.

To link a retraining policy to a Use Case, in the Use Case section, click + Select a Use Case:

In the Select Use Case modal, click Select Use Case or Create Use Case:

> [!NOTE] Default Use Case linking
> When a deployment is linked to a Use Case, that deployment's retraining policies and the resulting retrained models are automatically linked to that Use Case; however, you can override the default Use Case for each policy.

**Select Use Case:**
Select an existing Use Case from the Use Case name list, then click Select Use Case:

[https://docs.datarobot.com/en/docs/images/nxt-retrain-use-case-link-existing.png](https://docs.datarobot.com/en/docs/images/nxt-retrain-use-case-link-existing.png)

**Create Use Case:**
Enter a Use Case name and, optionally, a Description, then click Create and select:

[https://docs.datarobot.com/en/docs/images/nxt-retrain-use-case-create-new.png](https://docs.datarobot.com/en/docs/images/nxt-retrain-use-case-create-new.png)


### Model selection

Choose one of the following modeling strategies for the retraining policy; the strategy controls how DataRobot builds the new model on the updated data:

- Use same blueprint as champion at time of retraining: At the time of triggering, fits the champion model's blueprint on the new data snapshot. Select one of the following options:
- Use code-based retraining job: Run a retraining process defined in the selectedcustom retraining job's code. Code-based retraining job scheduling considerationsWhen configuring a retraining policy based on a code-based retraining job, consider the following:Updating a retraining policy schedule also updates the connected code-based retraining job's schedule.Defining a data drift or accuracy trigger in the retraining policy settings applies the selected trigger to scheduled code-based retraining jobsin addition tothe schedule.Deleting a retraining policy does not delete the retraining job, which can still run on a schedule defined by the code-based retraining job.Linking a retraining policy and job requires an existingmetadata.yamlcontaining theDEPLOYMENTandRETRAINING_POLICY_IDruntime parameters.Saving a retraining policy after linking it to a retraining job sets theDEPLOYMENTandRETRAINING_POLICY_IDruntime parameters for the retraining job; however, if a retraining job already has a specifiedDEPLOYMENTparameter and it differs from deployment for the current policy, the runtime parameters are not updated. A message appears prompting you to unlink the previous deployment to link the job to the current deployment.
- Use best Autopilot model(recommended): Run Autopilot on the new data snapshot and use the resulting recommended model. Choose from Quick or Autopilotmodeling modes. If selected, you can also toggle additional Autopilot options:

> [!NOTE] Note
> Comprehensive mode is not enabled when retraining models via the UI in Console. To set the retraining policy to Comprehensive mode, [use the API](https://docs.datarobot.com/en/docs/api/reference/public-api/mitigation_retraining.html#create-retraining-policies-by-id).

### Model action

The model action determines what happens to the model produced by a successful retraining policy run. In all scenarios, deployment owners are notified of the new model's creation and the new model is added as a model package to the [Registry](https://docs.datarobot.com/en/docs/workbench/nxt-registry/index.html). Apply one of three actions for each policy:

- Add new model as a challenger: If there is space in the deployment's five challenger models slots, this action—which is the default—adds the new model as a challenger model. It replaces any model that was previously added by this policy. If no slots are available, and no challenger was previously added by this policy, the model will only be saved to Registry. Additionally, the retraining policy run fails because the model could not be added as a challenger. Challengers added by a retraining policy are not re-scored on past data to prevent leaking training data into scoring data.
- Initiate model replacement with new model: Suitable for high-frequency (e.g., daily) replacement scenarios, this option automatically requests a model replacement as soon as the new model is created. This replacement is subject to definedapproval policiesand their applicability to the given deployment, based on its owners and importance level. Depending on that approval policy, reviewers may need to approve the replacement manually before it occurs.
- Save model: In this case, no action is taken with the model other than adding it to Registry.

> [!NOTE] Model actions with code-based retraining
> If, in the [model selection settings](https://docs.datarobot.com/en/docs/workbench/nxt-console/nxt-mitigation/nxt-retraining.html#model-selection), you select code-based retraining, it is possible to override the selected model action in the code for the [retraining job](https://docs.datarobot.com/en/docs/workbench/nxt-registry/nxt-jobs-workshop/nxt-create-jobs/nxt-create-custom-job.html#create-a-custom-job-for-retraining).

### Modeling strategy

The retraining modeling strategy defines how DataRobot sets up the new modeling run. Define the features, optimization metric, partitioning strategies, sampling strategies, weights, and [other advanced settings](https://docs.datarobot.com/en/docs/classic-ui/modeling/build-models/adv-opt/index.html) that instruct DataRobot on how to build models for a given problem.

You can either build using the same features as the champion model (when the trigger initiates) or allow DataRobot to identify [informative features](https://docs.datarobot.com/en/docs/classic-ui/modeling/build-models/build-basic/feature-lists.html#automatically-created-feature-lists) from the new data.

By default, DataRobot reuses the same settings as the champion model (at the time of the trigger initiating). Alternatively, you can define new partitioning settings, choosing from a subset of options that are available when building a new experiment.

## Manage retraining policies

After creating a retraining policy, you can start it manually, cancel it, or update it, as explained in the table below.

|  | Element | Definition |
| --- | --- | --- |
| (1) | Policy | Click the retraining policy row to view information about all previous runs of a training policy, successful or failed, including the start time, end time, duration, and—if the run succeeded—links to the resulting project and model package. |
| (2) | Edit | Click the edit button to edit the policy definition. |
| (3) | Run | Click the run button to start a policy manually. Alternatively, edit the policy by clicking the policy row and scheduling a run using the retraining trigger. |
| (4) | Remove | Click the remove button to delete a policy. Click Remove in the confirmation window. Policies cannot be removed while they are running. |
| (5) | Cancel | Click the cancel button to cancel a policy that is in progress or scheduled to run. You can't cancel a policy if it has finished successfully, reached the "Creating challenger" or "Replacing model" step, failed, or has already been canceled. |

## View retraining history

You can view all previous runs of a training policy, successful or failed. Each run includes a start time, end time, duration, and—if the run succeeded—links to the resulting Project and Model package:

> [!NOTE] Running policies
> Policies cannot be deleted or edited while they are running; they must be cancelled first. If the retraining worker and organization have sufficient workers, multiple policies on the same deployment can be running at once.

## Retraining strategies

The Challengers and Retraining tabs allow for simple performance comparison, meaning retraining strategies can be evaluated empirically and customized for different use cases. You may benefit from initial experimentation, using various time frames for the "same-blueprint" and Autopilot strategies. For example, consider running "same-blueprint" retraining strategies using both a nightly and a weekly pattern and comparing the results.

Typical strategies for implementing automatic retraining policies in a deployment include:

- High-frequency automatic schedule : Frequently (e.g., daily) retrain the currently deployed blueprint on the newest data to stabilize the deployed model selection.
- Low-frequency automatic schedule : Periodically (e.g., weekly, monthly) run Autopilot to explore alternative modeling techniques and potentially optimize performance. You can restrict this process to only Scoring Code-supported models if that is how you deploy. See the Include only blueprints with Scoring Code support advanced option for more information.
- Drift status trigger : Monitor data drift and trigger Autopilot to prepare an alternative model when the champion model has shown data drift due to changing situations.
- Accuracy status trigger : Monitor accuracy drift and trigger Autopilot to search for a better-performing model after the champion model has shown accuracy decay. This strategy is most effective for use cases with fast access to actuals.

## Retraining availability

Only binary, multiclass, and regression target types support retraining. The Retraining tab doesn't appear when a deployment's champion has a multilabel target type. See also the handling of [time series retraining](https://docs.datarobot.com/en/docs/workbench/nxt-console/nxt-mitigation/nxt-retraining.html#retraining-for-time-series).

### Unsupported models and projects

Retraining is not supported for the following DataRobot models and project types. In those cases, the Retraining tab doesn't appear when a deployment's champion uses any of the listed functionality:

**SaaS:**
Multilabel modeling projects
Feature Discovery
-based projects
Unsupervised learning projects
(including anomaly detection and clustering)
Unstructured custom inference models

**Self-Managed:**
Multilabel modeling projects
Feature Discovery
-based projects
Unsupervised learning projects
(including anomaly detection and clustering)
Unstructured custom inference models
Imported model packages


### Partially supported models

The following model types partially support retraining. For each partially supported model, only the supported (✔) options are available in retraining policies on the Retraining tab:

> [!NOTE] Note
> Only some retraining policy options are model-dependent. If the support matrix below doesn't include a model type, all options of a retraining policy are available for configuration.

| Model type | Same blueprint as champion | Champion model's feature list | Project options from champion model | Custom project options |
| --- | --- | --- | --- | --- |
| Custom inference |  |  |  | ✔ |
| External (agent) |  |  |  | ✔ |
| Blender* |  |  | ✔ | ✔ |
| Time series | ✔ | ✔ | ✔ |  |

* Blender models are not available in Workbench.

## Retraining for time series

Time series deployments support retraining, but there are limitations when configuring policies due to the time series [feature derivation process](https://docs.datarobot.com/en/docs/reference/pred-ai-ref/ts-reference/feature-eng.html#feature-reference). This process generates features such as lags and [moving averages](https://docs.datarobot.com/en/docs/reference/pred-ai-ref/ts-reference/ts-adv-opt.html#exponentially-weighted-moving-average) and creates a new modeling dataset.

### Time series model selection

Same blueprint as champion: The retraining policy uses the same engineered features as the champion model's blueprint. The search for newly derived features does not occur because it could potentially generate features that are not captured in the champion's blueprint.

Autopilot: When using Autopilot instead of the same blueprint, the time series feature derivation process does occur. However, Comprehensive Autopilot mode is not supported. Additionally, time series Autopilot does not support the option to only include Scoring Code blueprints or models with SHAP value support.

### Time series modeling strategy

Same blueprint as champion: When creating a "same-blueprint" retraining policy for a time series deployment, you must use the champion model's feature list and advanced modeling options. The only option that you can override is the [calendar used](https://docs.datarobot.com/en/docs/reference/pred-ai-ref/ts-reference/ts-adv-opt.html#calendar-files) because, for example, a new holiday or event may be included in an updated calendar that you want to account for during retraining.

Autopilot: When creating an Autopilot retraining policy for a time series deployment, you must use the informative features modeling strategy. This strategy allows Autopilot to derive a new set of feature lists based on the informative features generated by new or different data. You cannot use the model's original feature list because time series Autopilot uses a feature extraction and reduction process by default. You can, however, override additional modeling options from the champion's project:

| Option | Description |
| --- | --- |
| Treat as exponential trend | Apply a log-transformation to the target feature. |
| Exponentially weighted moving average (EWMA) | Set a smoothing factor for EWMA. |
| Apply differencing | Set DataRobot to apply differencing to make the target stationary prior to modeling. |
| Add calendar | Upload, add from the catalog, or generate an event file that specifies dates or events that require additional attention. |

## Time-aware retraining

For time-aware retraining, if you choose to reuse options from the champion model or override the champion model's project options, consider the following:

- If the champion's project used the holdout start date and end date, the retraining project does not use these settings but instead uses holdout duration, the difference between these two dates.
- If the champion project used the holdout duration with either the holdout start date or end date, the holdout start/end date is dropped, and holdout duration is used in the retraining project. A new holdout start date is computed (the end of the retraining dataset minus the holdout duration).

Your [customizations to backtests](https://docs.datarobot.com/en/docs/classic-ui/modeling/time/ts-adv-modeling/ts-date-time.html#edit-individual-backtests) are not retained; however, the number of backtests is retained. At retraining time, the training start and end dates will likely differ from the champion's start and end dates. The data used for retraining might have shifted so that it no longer contains all of the data from a specific backtest on the champion model.

---
