Replace deployed models¶
Because model predictions tend to degrade in accuracy over time, DataRobot provides an easy way to switch models and model packages for deployments. This ensures that models are up-to-date and accurate. Using the model management capability to switch model packages for deployments allows model creators to keep models current without disrupting downstream consumers. It helps model validators and data science teams to track model history. And, it provides model consumers with confidence in their predictions without needing to know the details of the changeover.
Replace a model package¶
You are redirected to the Overview tab of the deployment. Click Import from to choose your method of model replacement.
Local File: Upload a model package exported from DataRobot AutoML to replace an existing model package (standalone MLOps users only).
Model Registry: Select a model package from the Model Registry to replace an existing model package.
Paste AutoML URL: Copy the URL of the model from the Leaderboard and paste it into the Replacement Model field.
When you have confirmed the model package for replacement, select the replacement reason and click Accept and replace.
Model replacement considerations¶
When replacing a deployed model, note the following:
Model replacement is available for all deployments. Each deployment's model is provided as a model package, which can be replaced with another model package, provided it is compatible.
The new model package cannot be the same leaderboad model as an existing champion or challenger; each challenger must be a unique model. If you create multiple model packages from the same leaderboard model, you can't use those models as challengers in the same deployment.
While only the most current model is deployed, model history is maintained and can be used as a baseline for data drift.
Model replacement validation¶
DataRobot validates whether the new model is an appropriate replacement for the existing model and provides warning messages if issues are found. DataRobot compares the models to ensure that:
- The target names and types match. For classification targets, the class names must match.
- The feature types match.
- There are no new features. If the new model has more features, the warning identifies the additional features. This is intended to help prevent prediction errors if the new model requires features not available in the old model. If the new model has fewer or the same number of features, there is no warning.
- The replacement model supports all humility rules.
- If the existing model is a time series model, the replacement model must also be a time series model and the series types must match (single series/multiseries).
- If the model is a custom inference model, it must pass custom model tests.
- Prediction intervals must be compatible if enabled for the deployment.
- Segments must be compatible if segment analysis is enabled for the deployment.
DataRobot is only able to validate your model’s input features if you have assigned training data to both model packages (the existing model package for your deployment, and the one you selected to replace it with). Otherwise, DataRobot is unable to validate that the two model packages have the same target type and target name. A warning message informs you that model replacement is not allowed if the model, target type, and target name are not the same:
Model package replacement compatibility¶
Consider the compatibility of each model package type (external and DataRobot) before proceeding with model package replacement for a deployment:
Custom model packages are DataRobot model packages. DataRobot model packages can only replace other DataRobot model packages. They cannot be replaced by external model packages.
Custom model packages and imported .mlpkg files are both DataRobot model package types. DataRobot model packages can only replace other DataRobot model packages. They cannot be replaced by external model packages.